Skip to content

Ensure the second shutdown signal can interrupt graceful shutdown dispatch #3833

Description

@cijothomas

Description

The Dataflow Engine's double-signal shutdown convention handles the first OS signal by calling control_plane.shutdown_all() and only waits for the second signal after that synchronous call returns.

If graceful shutdown dispatch blocks, for example while waiting for an active runtime-recovery worker to stop, the signal listener cannot process the second signal immediately. This weakens the intended escape hatch where a second SIGINT/SIGTERM forces process::exit(1).

Proposed direction

Keep the signal listener responsive while graceful shutdown dispatch runs separately. The listener should concurrently observe:

  • completion of graceful shutdown dispatch;
  • listener cancellation during controller teardown;
  • a second OS signal, which forces immediate process exit.

Acceptance criteria

  • A second signal can force exit while shutdown_all() is still blocked.
  • Normal first-signal graceful shutdown behavior remains unchanged.
  • Listener cancellation and controller teardown remain bounded and do not leak a worker thread.
  • Add a regression test using a deliberately blocked control plane or shutdown worker.

Related to #2325.

Metadata

Metadata

Assignees

Type

No type

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions