Bug summary
This is a follow-up to #8270 / #8691 and the worker port in #9530.
Currently, when a ProcessWorker running in Kubernetes receives SIGTERM during a rollout or scale-down, the worker forwards it as SIGINT. That interrupts the worker's local process lifecycle, causing active local flow subprocesses to be terminated rather than allowed to finish within the pod's terminationGracePeriodSeconds.
This differs from the intended behaviour described in #8270: stop accepting new work after SIGTERM, while allowing already-started flow runs to complete before the worker exits.
I'd like to restore that behaviour for ProcessWorker: on SIGTERM, stop listening for new work, allow running flows to continue until they complete.
I've got a downstream patch (with tests, etc) that seems to work: GrowthLoop#1
Before cleaning this up and preparing an upstream PR, could y'all let me know if this patch would be considered?
Version info
Additional context
Related, but distinct: #16746 - marking flows as "Crashed" when a process worker goes down.
Bug summary
This is a follow-up to #8270 / #8691 and the worker port in #9530.
Currently, when a
ProcessWorkerrunning in Kubernetes receivesSIGTERMduring a rollout or scale-down, the worker forwards it asSIGINT. That interrupts the worker's local process lifecycle, causing active local flow subprocesses to be terminated rather than allowed to finish within the pod'sterminationGracePeriodSeconds.This differs from the intended behaviour described in #8270: stop accepting new work after
SIGTERM, while allowing already-started flow runs to complete before the worker exits.I'd like to restore that behaviour for
ProcessWorker: onSIGTERM, stop listening for new work, allow running flows to continue until they complete.I've got a downstream patch (with tests, etc) that seems to work: GrowthLoop#1
Before cleaning this up and preparing an upstream PR, could y'all let me know if this patch would be considered?
Version info
Additional context
Related, but distinct: #16746 - marking flows as "Crashed" when a process worker goes down.