Track runtime shutdown and retain foreign cleanup

.gitattributes

@@ -0,0 +1,3 @@
+# Replay fixture provenance hashes these sources byte-for-byte.
+tests/queue_race_upgrade.rs text eol=lf
+tests/common/cancellation_history.rs text eol=lf

CHANGELOG.md

@@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Foreign-runtime lifecycle integration** - Add fallible preparation and startup,
+  prompt stop requests, and typed cleanup results to the existing `TaskTracker` model.
+  Registered foreign cleanup remains owned after the native turn ends.
+  Finalization closes new durable effects before the snapshot.
+  Native shutdown waits through grace and cleanup. Language SDKs separately limit
+  caller waits and keep cleanup running.
+- **Pending foreign-retirement count** - One gauge reports retained foreign
+  completions per recorder, without Runtime labels or additional series.
+  New runtimes do not reset pending counts from other runtimes.
+  Structured events provide each Runtime's identity and count.
+  Broken completion observers remain counted. A zero count does not prove cleanup.
+- **Native stop/fault observation** - Add `Runtime::stop_status()` and an owned,
+  read-only `RuntimeStopObserver` that does not retain worker resources.
+  Keep terminal state for late observers and preserve fault escalation during rollback.
+  Report known turn faults before later provider waits, with protected diagnostics.
+  Status and source closure do not prove cleanup. SDK observation and host policy
+  remain separate from native cleanup.
 - Opt-in activity resource admission through `ActivityAdmission`, the default
   `ActivityHandler::try_admit` method, and the `register_with_admission` and
   `register_typed_with_admission` registration helpers.
@@ -26,8 +43,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   the handler. Orchestration-level retry helpers still retry these errors
   according to their policy. (#74)
 
+### Changed
+
+- **Internal lifecycle fault shutdown** - Faults caught at protected internal,
+  destruction, finalization, and registered-cleanup boundaries stop new work and
+  request zero-grace shutdown of that Runtime.
+  This can interrupt unrelated admitted work. Ordinary application failures still
+  affect only their operation. The failed worker cannot restart and must still
+  finish cleanup. Existing recovery and retry rules can repeat external effects.
+  Shutdown does not terminate the process, undo committed history, or guarantee exactly-once execution.
+- **Finalization failure before the snapshot** - A detected failure rejects the
+  new durable batch from that attempt. It does not publish an ordinary failed turn.
+  Foreign cleanup remains owned. Later driver or cleanup faults do not rewrite a
+  valid decision or undo a committed acknowledgement. Earlier force can cancel
+  an acknowledgement that has not committed.
+  Finalizer-admitted durable work must be reproduced during normal replay;
+  finalization is not another replay stage.
+
 ### Fixed
 
+- Preserve an observed `Drained` terminal outcome against a stale concurrent force
+  decision, while retaining native grace shortening before completion.
+- Preserve actual Tokio-generated activity-panic history text on the same task
+  after guarded destruction, instead of replacing it with payload-only text.
+- SQLite queue fetches, orchestration acknowledgements, instance deletion and
+  execution pruning reserve writer access before reading transaction state.
+  This avoids read-to-write upgrade failures and repeated fetch backoff during
+  concurrent activity deferrals.
+  Pool return rolls back transactions left by cancellation during acquisition,
+  so a cancelled fetch cannot retain writer access.
+  Existing lock timeouts, retry classification, attempt accounting and stored
+  formats remain unchanged.
 - SQLite worker selection now orders eligible items by visibility time, then
   queue ID, so a deferred backlog cannot repeatedly overtake already-ready work.
   Migration `20240113000000` adds the supporting `visible_at, id` index to new

CONTRIBUTING.md

@@ -55,6 +55,36 @@ For broader runtime changes, also run the comprehensive two-pass suite:
 ./run-tests.sh
 ```
 
+### Hook-free lifecycle validation on Windows
+
+The integration-test dev-dependency enables `test-hooks`, even with
+`cargo test --no-default-features`. Those binaries are not production binaries
+without test hooks. To test the public lifecycle API without that feature, run:
+
+```powershell
+.\tests\Run-LifecycleProductionTests.ps1
+```
+
+The runner builds the SQLite production library in `target\lifecycle-production`.
+It uses locked dependencies and offline mode, then checks Cargo's feature and profile evidence.
+It uses `rustc` to link `tests\lifecycle_production_tests.rs` and
+`tests\foreign_retirement_metrics_tests.rs` directly to that library.
+
+A generated recorder fixture builds `metrics-util` without core's dev-dependencies.
+Its lock starts from the repository lock.
+The runner rejects registry dependencies with different versions or checksums, then builds with `--locked`.
+It checks that the recorder and core use the same metrics facade artifact.
+The recorder tests cover 100 successive Runtime lifetimes, overlapping populations,
+and concurrent updates without lifecycle test hooks.
+
+Each suite must run at least one test, with no failures or skips.
+The output's `evidence` directory records test totals and the paths and hashes of each artifact.
+Use `-TargetDirectory` and `-EvidenceDirectory` to select other isolated paths inside this repository.
+The runner does not copy test assets into SDK, package, or production native-asset directories.
+
+Keep the instrumented lifecycle, replay/CAN/KV, polling, formatting, and Clippy checks.
+This runner does not replace PostgreSQL, other-platform, or release-dependency validation.
+
 ## Filing pull requests
 
 Use the pull request template in [.github/pull_request_template.md](.github/pull_request_template.md) and fill in the docs checkboxes.

Cargo.toml

@@ -30,6 +30,7 @@ uuid = { version = "1", features = ["v4"] }
 tracing = { version = "0.1", features = ["std"] }
 tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter", "json"] }
 semver = "1"
+futures-util = { version = "0.3", default-features = false, features = ["std"] }
 
 # used by provider-tests
 futures = { version = "0.3", optional = true }

README.md

@@ -56,6 +56,9 @@ duroxide = "0.1"  # Core only — bring your own Provider
 
 ### Hello world
 
+The fallible lifecycle calls below are **unreleased**. Use the source build
+containing these changes; the published 0.1.32 crate does not have them.
+
 ```rust
 use std::sync::Arc;
 use duroxide::{ActivityContext, Client, OrchestrationContext, OrchestrationRegistry, OrchestrationStatus};
@@ -82,7 +85,10 @@ let orchestrations = OrchestrationRegistry::builder()
     })
     .build();
 
-let rt = runtime::Runtime::start_with_store(store.clone(), activities, orchestrations).await;
+let rt = runtime::Runtime::prepare(
+    store.clone(), activities, orchestrations, runtime::RuntimeOptions::default(),
+)?;
+Arc::clone(&rt).start_execution().await?;
 let client = Client::new(store);
 
 client.start_orchestration("inst-1", "HelloWorld", "Rust").await?;
@@ -91,11 +97,47 @@ match client.wait_for_orchestration("inst-1", std::time::Duration::from_secs(5))
     other => panic!("unexpected: {other:?}"),
 }
 
-rt.shutdown(None).await;
+rt.shutdown_with_grace(std::time::Duration::from_secs(1)).await?;
 # Ok(())
 # }
 ```
 
+### Worker lifetime (unreleased)
+
+Use `Runtime::prepare` followed by `Arc::clone(&runtime).start_execution().await`
+to handle startup errors without a panic. Preparation does not start work or
+call the provider. The legacy `start*` factories still panic on startup failure.
+The runtime retains partially started work until rollback finishes.
+
+Use `shutdown_with_grace(grace)` to observe actual cleanup with a typed result.
+It returns `Result<ShutdownOutcome, RuntimeShutdownError>`. Both `Drained` and
+`Forced` confirm that tracked native work and registered foreign cleanup have ended.
+Grace limits the wait before force, not the total wait.
+Code that does not yield, or foreign work that does not finish, can prevent cleanup from completing.
+
+Concurrent native callers can shorten grace. Dropping a shutdown future does not
+release its tasks. Another call may be necessary to request force and finish cleanup.
+
+`request_shutdown()` promptly closes admission; `shutdown_until(deadline)` lets
+an adapter use an existing grace deadline. Each language SDK retains one cleanup
+future and applies a separate limit to the caller's wait. A stopped runtime cannot
+restart. An operational failure remains an error. Check `is_quiescent()` to find
+out whether all owned work and cleanup have ended.
+
+An internal lifecycle fault stops new work and requests **zero-grace shutdown of
+the affected Runtime**. This can interrupt other work in the same runtime.
+It does not terminate the process. An ordinary activity or orchestration error
+still affects only that operation. Interrupted work follows the existing recovery
+and retry rules. A retry can repeat an external effect. Read
+[fault shutdown and recovery](docs/ORCHESTRATION-GUIDE.md#fault-shutdown-and-recovery)
+before replacing a failed worker.
+
+Legacy `shutdown(None)` still uses one second of grace and returns `()` after cleanup.
+It has no total deadline and logs operational failures. SDK nanosecond transport
+limits do not restrict native grace. See the
+[lifecycle guide](docs/ORCHESTRATION-GUIDE.md#worker-lifecycle-unreleased) for
+validation, provider I/O, and language-adapter boundaries.
+
 ### Surviving crashes
 
 Each completed step is durably recorded, so a restart replays history and