Skip to content

BUG: Release the Python GIL during pipeline updates - #7002

Open
sdiebolt wants to merge 3 commits into
InsightSoftwareConsortium:mainfrom
sdiebolt:fix/7000-python-update-gil
Open

sdiebolt wants to merge 3 commits into
InsightSoftwareConsortium:mainfrom
sdiebolt:fix/7000-python-update-gil

Conversation

@sdiebolt

@sdiebolt sdiebolt commented Oct 23, 2026 •

Copy link
Copy Markdown

Let Python threads run during ITK pipeline updates, and make the regression test check filtering rather than setup. Closes #7000.

Why the GIL stayed locked

The 2019 GIL-release change enabled SWIG threading but also set nothreadallow, citing this SWIG discussion. That advice served a different goal: protecting director callbacks while deliberately avoiding GIL release around native calls. The combination therefore did not provide the concurrency described by the commit.

The later -threads addition in #5776/#6022 left that override in place. Its regression test accepted overlapping setup/filtering intervals as evidence of GIL release. Measured intervals were about 0.81 seconds, but only 0.002–0.003 seconds were inside Update().

What this fixes
  • Release the GIL for Update() and UpdateLargestPossibleRegion(), retaining it for unrelated wrappers and respecting ITK_PYTHON_RELEASE_GIL.
  • Reacquire the GIL in PyImageFilter callbacks and before PyCommand validates observer callables.
  • Replace the overlap test with a waiting-thread probe around preconstructed filters, suppressing periodic Python thread switching.
  • Exercise native Update() exceptions instead of errors that could arise in Python-side file-reading setup.
  • Document the scope and select an available wrapped type for the Python-filter test.
Validation

Linux x86_64, CPython 3.13.9, NumPy 2.3.4, Release build:

  • Unpatched wrappers: original regression passed; replacement failed.
  • Patched wrappers: replacement passed 20 consecutive runs; all three targeted tests passed.
  • Issue Python Update() appears to hold the GIL despite GIL-release support #7000's ticker workload: 913–927 ticks during 0.97–0.98 seconds of filtering.
  • Compiled SWIG probes confirmed enabled/disabled threading behavior; local hooks passed.
  • An instrumented observer probe detected PyCallable_Check() without the GIL before the PyCommand fix; the probe and full safety script pass after it.

The initial Linux CI run also passed all three targeted tests. Its separate median type-hint failure is described in the PR comments.

Full local suite, Python 3.14 wheel, and full GIL-disabled native build validation were not run. CI must validate the latest revision.

PR Checklist
  • No API changes were made (or the changes have been approved)
  • No major design changes were made (or the changes have been approved)
  • Added test (or behavior not changed)
  • Updated API documentation (or API not changed)
  • Added license to new files (if any) — no new files
  • Added Python wrapping to new files (if any) as described in ITK Software Guide Section 9.5 — no new classes
  • Added ITK examples for all new major features (if any) — bug fix, no new major feature
AI assistance

AI assistance supported diagnosis, implementation, and local testing.

The global SWIG nothreadallow feature overrides threading support.
Limit GIL release to pipeline updates and reacquire it for callbacks.
Replace setup-time overlap detection with a native execution check.

Fixes InsightSoftwareConsortium#7000
@github-actions github-actions Bot added type:Bug Inconsistencies or issues which will cause an incorrect result under some or all circumstances type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots area:Python wrapping Python bindings for a class type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct area:Core Issues affecting the Core module area:Documentation Issues affecting the Documentation module labels Oct 23, 2026
@sdiebolt
sdiebolt marked this pull request as ready for review October 24, 2026 00:51
@greptile-apps

greptile-apps Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The change affects pipeline updates across ITK’s Python wrappers.

Fix the unchecked float requirement in PyImageFilterTest before merging.

Findings

  1. P1 Float-disabled builds fail this test ▶

Summary

The PR enables GIL release for Update() and UpdateLargestPossibleRegion() and adds GIL guards around Python filter callbacks.

  • ITK pipeline updates let other Python threads run.
  • Python filter callbacks run with the GIL held.

Diagram

sequenceDiagram
  participant P as Python thread
  participant W as Update wrapper
  participant F as Native pipeline
  participant C as Python callback
  P->>W: Update()
  W->>W: Release GIL when enabled
  W->>F: Run pipeline
  F->>C: Acquire GIL and call Python
  C-->>F: Return and release acquired GIL
  F-->>W: Finish update
  W->>W: Reacquire GIL
  W-->>P: Return or raise
Loading

Reviews (1) · Last reviewed commit: "BUG: Release the Python GIL during pipel..." · Reviewed by Greptile

Comment thread Wrapping/Generators/Python/Tests/PyImageFilterTest.py Outdated
Custom wrapping builds may disable float while retaining other scalar
types. Exercise an available two-dimensional specialization instead.
@sdiebolt

sdiebolt commented Oct 24, 2026 •

Copy link
Copy Markdown
Author

The Linux failure looks separate from this fix: filtering succeeds, then the median test’s type-hint check raises NameError: name 'xr' is not defined. The unchanged parent-commit type alias reproduces it when xarray is available.

All three GIL-related tests passed in that job. The float-wrapping review finding is fixed in 2d8272e; those tests also pass locally.

Observers can execute while pipeline updates have released the GIL.
Protect callable validation as well as invocation and error handling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Core Issues affecting the Core module area:Documentation Issues affecting the Documentation module area:Python wrapping Python bindings for a class type:Bug Inconsistencies or issues which will cause an incorrect result under some or all circumstances type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python Update() appears to hold the GIL despite GIL-release support

1 participant