Skip to content

Gracefully drain workers on shutdown instead of terminating them - #387

Open
artem-sitd wants to merge 1 commit into
pgjones:mainfrom
artem-sitd:fix/graceful-worker-drain-on-shutdown
Open

artem-sitd wants to merge 1 commit into
pgjones:mainfrom
artem-sitd:fix/graceful-worker-drain-on-shutdown

Conversation

@artem-sitd

Copy link
Copy Markdown

Fix #386.

Problem

With --workers > 1, SIGTERM/SIGINT torn down in-flight requests:

  • idle workers exit immediately once shutdown_event is set;
  • the master is blocked in wait(process.sentinel ...), which returns as soon as any worker exits;
  • the loop then ends (because active is False) and the master calls process.terminate() on the remaining, busy workers.

The result is that requests still being served are cut off ("Empty reply from server"), even when -w 1 serves them fine.

Change

Replace the terminate() loop with _drain(processes), which joins the remaining workers so they can finish their own graceful shutdown (bounded on the worker side by config.graceful_timeout). This matches the single-worker behaviour and mirrors the existing unbounded process.join() in the reload handler.

Tests

Added tests/test_run.py asserting the master joins workers and never terminates them. Existing test_utils.py and test_config.py pass.

Note: on Python 3.12+, worker-side draining can be effectively unbounded because of Server.wait_closed() (#308); that is a separate issue and its fix would let config.graceful_timeout (or a future hard cap in the master) be applied without cutting in-flight requests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

With multiple workers, SIGTERM kills in-flight requests because the master terminates busy workers once the first idle worker exits

1 participant