Skip to content

ProcessWorker should drain active local flow runs on SIGTERM #22528

Description

@wolever-gl

Bug summary

This is a follow-up to #8270 / #8691 and the worker port in #9530.

Currently, when a ProcessWorker running in Kubernetes receives SIGTERM during a rollout or scale-down, the worker forwards it as SIGINT. That interrupts the worker's local process lifecycle, causing active local flow subprocesses to be terminated rather than allowed to finish within the pod's terminationGracePeriodSeconds.

This differs from the intended behaviour described in #8270: stop accepting new work after SIGTERM, while allowing already-started flow runs to complete before the worker exits.

I'd like to restore that behaviour for ProcessWorker: on SIGTERM, stop listening for new work, allow running flows to continue until they complete.

I've got a downstream patch (with tests, etc) that seems to work: GrowthLoop#1

Before cleaning this up and preparing an upstream PR, could y'all let me know if this patch would be considered?

Version info

3.6.27

Additional context

Related, but distinct: #16746 - marking flows as "Crashed" when a process worker goes down.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions