Bound shutdown waits and retain Python cleanup

.agents/skills/duroxide-python-orchestrations/SKILL.md

@@ -273,7 +273,7 @@ pytest tests/scenarios/ -v           # scenario tests (6 tests)
 - Use `SqliteProvider.in_memory()` for fast isolated tests (SQLite smoketest only)
 - All PG tests need `DATABASE_URL` in `.env` (loaded by `python-dotenv`)
 - Each test file uses a separate PG schema for isolation
-- Use short `runtime.shutdown(100)` timeout — it waits the full duration
+- `runtime.shutdown(100)` permits 100 ms of grace plus five seconds of cleanup headroom. It can finish early; timeout raises while cleanup remains owned and requires application/supervisor termination.
 - Set `RUST_LOG=info` and use `pytest -s` to see traces in test output
 - Use `worker_lock_timeout_ms=2000` in tests needing fast activity cancellation detection
 

.github/copilot-instructions.md

@@ -210,7 +210,7 @@ Orchestration code must be deterministic — same input + history = same sequenc
 - PostgreSQL tests use schema isolation (one schema per test file)
 - `python-dotenv` loads `DATABASE_URL` from `.env`
 - `PyRuntimeOptions(dispatcher_poll_interval_ms=50)` for fast test dispatch
-- `runtime.shutdown(100)` — short timeout, it waits the full duration
+- `runtime.shutdown(100)` — 100 ms grace plus five seconds total headroom; idle work can finish early, and failures raise while retaining cleanup
 - Use `SqliteProvider.in_memory()` only for SQLite smoketest
 - `worker_lock_timeout_ms=2000` for tests needing fast cancellation detection
 

CHANGELOG.md

@@ -7,6 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Add string-valued `RuntimeStopStatus` and owned, metadata-only
+  `RuntimeStopObserver` through `Runtime.stop_observer()`. Synchronous change
+  waits release the GIL, support finite timeout and Python signals, and allow
+  terminal cross-thread close. Final state arrives before normal source end.
+  Observation does not retain worker callbacks or change shutdown ownership,
+  deadlines, and application policy.
+
+### Changed
+
+- Use fallible core startup and bounded shutdown waits while preserving successful `None` returns.
+  Total waiting is the supplied grace plus five seconds.
+  Omitted grace is one second. Idle workers can finish early.
+  Invalid durations fail before ownership changes, including for no-work and repeated calls.
+  Repeated calls retain the original operation, errors, and deadlines.
+  They do not report success merely because the wrapper transferred its core.
+  Registry and preparation failures become ordinary retained startup errors.
+  Registration policy does not change.
+- Pre-start shutdown is terminal.
+  Start and registration calls fail after activation or stop.
+  Post-stop metrics remain unavailable.
+- Shutdown timeout raises `TimeoutError`. Startup and operational failures raise `RuntimeError`.
+  Incomplete cleanup requires application- or supervisor-owned process termination, even after actual late cleanup.
+  Finite grace does not guarantee bounded forced cleanup.
+  This adapter does not add .NET's foreign-continuation barrier or own detached application work.
+  Release requires a matching published core dependency.
+  Do not distribute local development overrides.
+
+### Fixed
+
+- Release obsolete native activity-callback ownership after proven cleanup,
+  including quiescent failures and late completion without another shutdown call.
+  This breaks bound-method and self-capturing callback cycles that Python cannot otherwise collect.
+  The first budgets and terminal outcomes remain available.
+  Incomplete cleanup or a lost cleanup owner still retains ownership.
+  Independent Clients, providers, and the global orchestration-handler map do not change.
+
+### Tests
+
+- Add SQLite-only lifecycle coverage through the real wrapper.
+  Cases include feature-gated provider and fault controls, repeated ordered races,
+  isolated held-provider probes, and production-export checks.
+- Activity-only collection controls, failed-startup metrics, rejected-registration
+  atomicity, and low-level driver preconditions.
+  Subprocess failures retain bounded diagnostics.
+  Transport watchdogs are separate from the unchanged shutdown timing limits.
+  Windows tests measure those limits with high-resolution clocks.
+
 ## [0.2.2] - 2026-10-21
 
 ### Changed

CONTRIBUTING.md

@@ -29,4 +29,27 @@ cargo clippy --all-targets
 pytest -v
 ```
 
-After Rust source changes (`src/*.rs`), re-run `maturin develop` before running Python tests.
+After Rust source changes (`src/*.rs`), re-run `maturin develop` before running Python tests.
+
+### Lifecycle checks without PostgreSQL
+
+Use an isolated virtual environment with the declared development dependencies.
+Rebuild and install the extension for each mode.
+Importing an old extension is not lifecycle evidence.
+For example, in PowerShell with that environment activated:
+
+```powershell
+maturin develop --features test-hooks
+$env:DUROXIDE_LIFECYCLE_TEST_HOOKS = '1'
+python -m pytest -v tests\test_lifecycle.py tests\test_e2e.py::test_sqlite_smoketest
+Remove-Item Env:DUROXIDE_LIFECYCLE_TEST_HOOKS
+maturin develop
+python -m pytest -v tests\test_lifecycle.py tests\test_e2e.py::test_sqlite_smoketest
+```
+
+Instrumented cases exercise real provider waits and contained faults in owned tasks.
+The suite checks native import provenance and runs 100 repetitions per ordered race.
+It also checks bounded child-process cleanup and confirms that production builds omit test exports.
+Production mode skips only hook-dependent cases.
+Do not publish instrumented assets or temporary local core overrides.
+Release requires a matching published core minimum.

Cargo.toml

@@ -7,6 +7,9 @@ edition = "2021"
 name = "duroxide_python"
 crate-type = ["cdylib"]
 
+[features]
+test-hooks = ["duroxide/test-hooks"]
+
 [dependencies]
 duroxide = { version = "0.1.32", features = ["sqlite"] }
 duroxide-pg = "0.1.37"

README.md

@@ -24,6 +24,62 @@ Write durable workflows as Python generators. The Rust runtime handles replay, p
 - **Activity client access** — `ctx.get_client()` lets activities start new orchestrations
 - **Runtime metrics** — `metrics_snapshot()` for orchestration/activity counters
 
+## Runtime lifecycle (unreleased)
+
+`runtime.start()` returns after startup. Core startup errors raise `RuntimeError`.
+`runtime.shutdown(timeout_ms)` still returns `None` on success.
+The argument is a nonnegative integer grace period in milliseconds.
+It must fit the native monotonic deadline range.
+Omitted grace is 1000 ms. Total waiting is grace plus 5000 ms.
+Idle work can finish early.
+
+The first stop fixes both deadlines. Repeated calls observe the same operation.
+They can report timeout, operational errors, or actual late completion.
+Shutdown before start is terminal. Later start or registration calls fail.
+Metrics remain unavailable after stop.
+Serialize lifecycle and registration calls on each runtime.
+
+Timeout raises `TimeoutError`. Startup and operational failures raise `RuntimeError`.
+A timeout means cleanup is still owned and incomplete.
+Do not restart the worker.
+The application or supervisor must terminate the process, even if a later observation reports completion.
+The SDK does not terminate the process.
+
+This adapter does not track arbitrary Python work.
+It does not add the .NET SDK's additional managed foreign-retirement barrier.
+Independent provider and Client resources have their own lifetime. See the
+[lifecycle guide](docs/user-guide.md#runtime-lifecycle) for validation and an
+executable early-drain/repeated-stop example.
+**Quiescence** means that tracked core execution and cleanup have ended.
+Proven cleanup releases obsolete native activity-callback ownership, including after quiescent failures and late completion.
+Terminal outcomes and the original budgets remain available.
+Unproven cleanup never permits callback release.
+
+These unreleased changes require a compatible core lifecycle release before distribution.
+A temporary local Cargo override provides development evidence, not a release dependency pin.
+
+### Passive stop observation
+
+`runtime.stop_observer()` creates an owned, metadata-only `RuntimeStopObserver`.
+Read its `state` before calling `wait_for_change(last_seen, timeout_ms=None)`.
+`RuntimeStopStatus` has string-valued `NOT_REQUESTED`, `REQUESTED`, and sticky
+`FAULTED` members. They describe recorded stop/fault state, not readiness or
+completed cleanup. A late observer can already see a fault.
+
+Use one pending wait per observer. A finite timeout raises `TimeoutError`;
+`KeyboardInterrupt` propagates. Either permits retry without stopping the worker.
+`close()` can run from another thread and is terminal and idempotent.
+Pending and later operations fail after close. Normal source end instead
+returns an unseen final state before `None`.
+
+The synchronous wait releases the GIL and checks Python signals.
+It still blocks its calling thread. Do not run it on the asyncio event-loop
+thread or assume that canceling `asyncio.to_thread` cancels the underlying wait.
+Use a finite timeout or explicitly close the observer.
+Observation retains no worker callback ownership and changes no shutdown
+deadlines or termination requirements. See the
+[observer example](docs/user-guide.md#passive-stop-observation).
+
 ## Installation
 
 ```bash

ci/smoke/smoke.py

@@ -18,15 +18,26 @@
 """
 
 import os
+import hashlib
+import json
 import platform
 import sys
 import tempfile
 import time
+from pathlib import Path
 
 from duroxide import SqliteProvider, Client, Runtime, RuntimeOptions
+from duroxide import _duroxide as native
 
 
 def main() -> int:
+    extension = Path(native.__file__).resolve()
+    assert extension.is_relative_to(Path(sys.prefix).resolve()), "smoke must load the installed wheel"
+    assert not hasattr(native, "_lifecycle_test_hooks")
+    assert not hasattr(native, "LifecycleTestHooks")
+    print("[smoke] native=" + json.dumps({
+        "path": str(extension), "sha256": hashlib.sha256(extension.read_bytes()).hexdigest(),
+    }))
     tmp = tempfile.mkdtemp(prefix="duroxide-smoke-")
     db_path = os.path.join(tmp, "smoke.db")
     print(
@@ -62,6 +73,16 @@ def hello_world(ctx, inp):
     finally:
         runtime.shutdown(200)
 
+    assert runtime.shutdown(0) is None
+    assert runtime.metrics_snapshot() is None
+    unstarted = Runtime(provider)
+    unstarted.shutdown(0)
+    try:
+        unstarted.start()
+        raise AssertionError("pre-start stop was not terminal")
+    except RuntimeError as error:
+        assert "lifecycle_terminal" in str(error)
+
     return 0
 
 

docs/architecture.md

@@ -297,7 +297,82 @@ File-based SQLite can hit "database is locked" errors under concurrent orchestra
 
 ### Runtime Shutdown
 
-`Runtime.shutdown(timeout_ms)` returns as soon as all runtime work has stopped. `timeout_ms` (default 1000) is the longest it waits for in-flight work before cancelling the rest; it is not a fixed delay.
+In the unreleased lifecycle, `Runtime.shutdown(timeout_ms)` still returns `None` on success.
+It waits for actual core retirement within supplied grace plus 5000 ms.
+Omitted grace is 1000 ms. Idle execution can finish early.
+Zero requests immediate force. Force does not prove that cleanup ended.
+The first valid stop fixes both deadlines.
+Repeated calls cannot obtain a new budget.
+
+The wrapper retains the prepared core Arc before it awaits fallible startup.
+The first valid stop transfers that core and its callbacks to one independently owned native cleanup future.
+The wrapper keeps the first deadline and completion record, not an obsolete callback registry.
+
+```mermaid
+flowchart TD
+    Stop["First valid stop: fix grace and total"] --> Cleanup["Keep one native cleanup operation"]
+    Stop --> Wait["Wait within the original total"]
+    Cleanup -->|Cleanup proven| Release["Release obsolete core and activity-callback ownership"]
+    Cleanup -->|Cleanup unproven| Retain["Retain ownership"]
+    Wait -->|Completion published| Result["Return None or raise an operational error"]
+    Wait -->|Total expires first| Timeout["Raise TimeoutError; keep cleanup owned"]
+    Timeout --> Terminate["Application or supervisor must terminate the process"]
+```
+
+The caller's wait and cleanup have separate lifetimes.
+Proven cleanup releases the core and its activity callbacks.
+This includes quiescent failures and late completion without another shutdown call.
+PyO3 defers off-GIL reference-count decrements until the next GIL entry.
+A timeout does not cancel the cleanup owner.
+Losing that owner or receiving an unproven result retains the core instead of releasing it.
+
+Upstream tracking includes native task destruction.
+Provider futures can be canceled under the provider's atomic acknowledgment and recovery contract.
+Repeated stop observes the retained completion or error.
+It does not report success merely because the wrapper transferred its core.
+Release does not clear the global orchestration-handler map or dispose independent Clients or providers.
+
+A valid pre-start stop is terminal.
+Later registration and start calls fail.
+Metrics remain unavailable after stop or accepted startup failure.
+Serialize lifecycle and registration calls on the same instance.
+This includes calls from other threads while the GIL is released.
+
+`TimeoutError` means that ordinary waiting ended while cleanup remains owned.
+The application or supervisor must terminate the process, even after actual late completion.
+Operational and startup failures raise `RuntimeError`; they do not report successful retirement.
+The SDK never kills the process.
+This adapter does not add the .NET foreign-continuation barrier or track arbitrary Python work.
+Independent Clients, providers, and provider token-refresh work keep their separate lifetimes.
+See the [lifecycle guide](user-guide.md#runtime-lifecycle) for validation and
+release prerequisites.
+
+### Passive Stop Observation
+
+`Runtime` caches a receiver-only native source when it creates its wrapper.
+`stop_observer()` uses this source without borrowing the mutable native Runtime
+during startup or shutdown. Each observer owns metadata and a pending-wait slot,
+not the Runtime, provider, or Python callback registry.
+
+Publisher retirement freezes the final state and detaches its live core receiver.
+Core retirement alone is not SDK source end. A wait returns an unseen final
+state before `None`; local `close()` instead makes pending and later use fail.
+The final snapshot remains readable until local close.
+
+Foreground waiting releases the GIL. Native waits have bounded signal-check
+opportunities, and Python signal handlers run outside metadata locks.
+One absolute optional timeout is kept across those checks.
+The pending-slot guard cancels/clears each return or error, including
+`KeyboardInterrupt` and reentrant signal exceptions.
+Native notification tasks own no Python or worker objects and cannot clear a
+later wait's slot. Accepted oneshot-send counters are not Python-settlement counts.
+
+Notification final access is separate from public wait settlement and worker
+cleanup. An observer can remain alive while eligible worker callbacks collect.
+An offloaded Python wait still belongs to its calling thread: asyncio task
+cancellation does not cancel it. A foreground/non-daemon waiting thread can
+keep the process running. See the
+[observer contract and example](user-guide.md#passive-stop-observation).
 
 ### Platform-Specific Binary