Skip to content

gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths - #154837

Open
calvinrp wants to merge 1 commit into
python:mainfrom
calvinrp:fix-subprocess-wait-timeout-overflow
Open

gh-154836: Fix Popen.wait() with very large timeouts on the pidfd/kqueue wait paths#154837
calvinrp wants to merge 1 commit into
python:mainfrom
calvinrp:fix-subprocess-wait-timeout-overflow

Conversation

@calvinrp

@calvinrp calvinrp commented Jul 28, 2026

Copy link
Copy Markdown

Body

Fixes the 3.15 regression where subprocess.Popen.wait(timeout=...) (and
subprocess.run(..., timeout=...)) fails for very large timeout values such
as float("inf"), sys.maxsize or 1e10 — all accepted on 3.14 and
earlier. The event-driven wait introduced by gh-83069 (#144047) passes the
caller's timeout unclamped to poll() / kqueue.control(); values that do
not fit the C timestamp conversion raise OverflowError on Linux and, on
macOS/BSD, a misleading TypeError: timeout must be a real number or None, not float.

Changes

  • Lib/subprocess.py — both _wait_pidfd() and _wait_kqueue() now
    clamp each OS-level wait to a new module constant
    _MAXIMUM_WAIT_TIMEOUT = 24 * 3600 (following asyncio's
    MAXIMUM_SELECT_TIMEOUT precedent, Lib/asyncio/base_events.py) and loop
    until the caller's real deadline, so TimeoutExpired is still raised at
    the right time and huge-but-legal timeouts never reach the C conversion.
    In _wait_kqueue() the kevent changelist is only submitted on the first
    control() call (the KQ_EV_ADD | KQ_EV_ONESHOT registration persists
    across timed-out waits).
  • Modules/selectmodule.c — the kqueue control() timeout-conversion
    failure is now only rewritten into the "timeout must be a real number or
    None" TypeError when the original exception is a TypeError, guarded
    by PyErr_ExceptionMatches(PyExc_TypeError) exactly like the sibling
    select(), poll(), devpoll() and epoll() sites already do. Huge
    timeouts now surface the true OverflowError instead of a false claim
    that a float is not a real number.

Tests

  • Lib/test/test_subprocess.py (FastWaitTestCase):
    test_wait_huge_timeout (10**10, sys.maxsize, float('inf')),
    test_run_huge_timeout (the run()/communicate() path), and
    test_wait_slices_do_not_expire_early (with the slice limit patched down
    to 10 ms, a process outliving many slices is still waited for — guards
    against a clamp that fires TimeoutExpired at the slice boundary). All
    POSIX-gated: the Windows _wait has a separate, pre-existing
    int(timeout * 1000) overflow for float("inf") that predates 3.15 and
    is deliberately out of scope here.
  • Lib/test/test_kqueue.py: test_control_overflowing_timeout — an
    out-of-range timeout raises OverflowError, not TypeError; non-numbers
    still raise TypeError.

How tested

On macOS (arm64), against a 3.15.0b4 build with the patched
subprocess.py overlaid: the three new test_subprocess tests pass, the
entire pre-existing FastWaitTestCase and the POSIX wait-related suite
stay green, and wait(timeout=float("inf")) / run(..., timeout=1e10)
succeed. The Python-level clamp alone already fixes the user-visible
wait()/run() breakage (the overflowing value never reaches
kq.control()). The test_kqueue addition and the OverflowError
surfacing require the rebuilt select module (verified expected-fail
against the unpatched 3.15.0b4 C module); the Linux _wait_pidfd loop is
exercised by the new tests on Linux CI.


NEWS entry (already in the commit, blurb format)

Misc/NEWS.d/next/Library/2026-07-19-20-00-00.gh-issue-154836.kqWait.rst:

Fix :meth:subprocess.Popen.wait raising :exc:TypeError (macOS and other
BSDs) or :exc:OverflowError (Linux) for very large timeout values such
as float('inf'), a 3.15 regression in the new event-driven wait. Also
fix :meth:select.kqueue.control masking :exc:OverflowError for
out-of-range timeouts as :exc:TypeError.

…fd/kqueue wait paths

The event-driven wait introduced by pythongh-83069 passes the caller's timeout
unclamped to poll() / kqueue.control(), so values that do not fit the C
timestamp conversion (float('inf'), sys.maxsize, 1e10, ...) raise
OverflowError on Linux and, on macOS/BSD, a misleading
"TypeError: timeout must be a real number or None" -- all of which
worked on 3.14 and earlier.

- Lib/subprocess.py: clamp each wait to _MAXIMUM_WAIT_TIMEOUT (24h,
  following asyncio's MAXIMUM_SELECT_TIMEOUT precedent) and loop until
  the real deadline in both _wait_pidfd() and _wait_kqueue().
- Modules/selectmodule.c: only rewrite the kqueue.control() timeout
  conversion failure into TypeError when the original exception IS a
  TypeError, exactly like the select()/poll()/devpoll()/epoll() sites,
  so OverflowError surfaces for out-of-range values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@calvinrp
calvinrp requested a review from gpshead as a code owner July 28, 2026 18:11
@python-cla-bot

python-cla-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@BHUVANSH855 BHUVANSH855 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I noticed that the only failing required check is the Contributor License Agreement (CLA). It looks like the implementation, and the rest of the CI checks are passing.

Could you please sign the CLA? Once that's done, the required check should pass, and the PR will be ready for code review.

@gpshead gpshead self-assigned this Jul 28, 2026
@calvinrp

Copy link
Copy Markdown
Author

CLA signed, thanks

@gpshead gpshead added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 29, 2026
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 9b7cdf2 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F154837%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Jul 29, 2026
@gpshead

gpshead commented Jul 29, 2026

Copy link
Copy Markdown
Member

the changes look good, running them through the bots now to get some additional kqueue coverage as well. there'll probably be some unrelated failure noise from a few of the less reliable buildbots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants