Clean up old generations that are building up in active long-running workflows

CHANGELOG.md

@@ -6,6 +6,33 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc
 
 ## [Unreleased]
 
+### Changed
+
+- **Historical execution retention (#415):** reconciliation now removes completed,
+  non-current execution generations and their history after
+  `pg_durable.retention_days`, including generations of still-running workflows
+  and nested loops. Current and Running executions remain protected. Cleanup
+  advances through bounded pages under server-side deadlines, reducing the page
+  size after query cancellation so otherwise-prunable generations do not remain
+  stuck in an oversized batch. `pg_durable.reconcile_interval = 0` disables it.
+- **Execution inspection (#415):** `df.instance_executions()` reads execution
+  details and event counts in one snapshot so concurrent pruning does not fail
+  the listing. Pruned generations no longer appear in subsequent inspections.
+
+> **Upgrade:** Stop and drain all provider callers, not just workers, before
+> upgrading. A full PostgreSQL stop/start drains pg_durable's old server
+> processes; coordinate any external clients and deletion jobs sharing the
+> provider store too. The new worker backfills missing completion timestamps
+> before publishing readiness. Large stores can extend this window and generate
+> substantial WAL, while submissions and engine operations must wait. Review
+> retention cutoffs before restarting because recovered timestamps can make old
+> generations eligible for cleanup immediately. The `statement_timeout` in effect for
+> `pg_durable.worker_role` in the control database must allow the backfill to
+> complete. Missing terminal history leaves the timestamp NULL with a warning
+> rather than guessing its age; malformed JSONB aborts the migration with the
+> affected history key in the error. No `ALTER EXTENSION UPDATE` is needed for
+> this provider-managed change. See [upgrade and repair guidance](docs/upgrade-testing.md#historical-execution-pruning).
+
 ### Security
 
 - **Start admission quotas (#139):** `df.max_concurrent_per_user` (default

Cargo.toml

@@ -36,8 +36,9 @@ uuid = { version = "1.0", features = ["v4", "serde"] }
 percent-encoding = "2.3"
 
 # duroxide integration
-duroxide = "=0.1.30"
-duroxide-pg = "=0.1.34"
+duroxide = "=0.1.32"
+# DONOTLAND: migrations 0027/0028 are not released; replace with an exact compatible crates.io pin.
+duroxide-pg = { git = "https://github.com/microsoft/duroxide-pg", rev = "79870890d1a4a14eb840ec1d31d0b6dff0716c0a" }
 tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] }
 
 # For ExecuteSQL activity (connecting back to PostgreSQL)

USER_GUIDE.md

@@ -2011,8 +2011,9 @@ runtime. Distinguish three roles:
     `database` argument selects another database on the same cluster; that target
     does not need the extension unless the SQL itself calls `df` APIs.
 
-Satellite installation requires a compatible control worker with readiness schema
-version `3` (including its origin registry and protected submission accounting).
+Satellite installation requires a compatible control worker with the current
+readiness schema version, including its provider migrations, origin registry,
+and protected submission accounting.
 Provider migrations run only in control through the worker's `ApplyAll`;
 satellite creation never creates control
 automatically. The provider namespace is extension-owned, its contents are created
@@ -2533,19 +2534,25 @@ satellite `df._worker_epoch` is normal; satellites do not have separate workers.
 pg_durable tracks each workflow in its own `df` tables, while the durable engine
 keeps workflow state in a separate schema. To keep those two stores consistent —
 and to keep `df.instances`/`df.nodes` from growing without bound — the background
-worker runs a **best-effort reconciliation pass** that does two things:
+worker runs a **best-effort reconciliation pass** that:
 
 - **Removes expired terminal instances.** Old **terminal** instances (status
   `completed`, `failed`, or `cancelled`) and their `df.nodes` rows are deleted,
-    along with their engine records. Retention does not remove running or pending
-    instances from an existing installation, regardless of age.
+  along with their engine records. Retention does not remove running or pending
+  instances from an existing installation, regardless of age.
 - **Reclaims orphaned engine records.** `df.start()` writes the `df` rows in the
   caller's transaction but hands the workflow to the engine over a separate
   connection; if that transaction **rolls back**, the `df` rows vanish while the
   engine keeps an inert record (it can never load its rolled-back graph, so it
   ends up failed). Reconciliation deletes such df-less engine records once they
-  age past `retention_days`. Anything the engine is still tracking with a live
-  `df` row is left untouched.
+  age past `retention_days`. Instances with a live `df` row are not orphans.
+- **Prunes old execution generations.** Loops start a new execution generation
+  on each iteration. Reconciliation removes historical generations and their
+  history events once their completion is older than `retention_days`, including
+  those belonging to still-running workflows and nested loops in any origin.
+  The current execution and any running execution are always preserved. This
+  does not remove the workflow's `df` rows or its current results, but old
+  generations will no longer appear in execution-history inspection.
 
 After the corresponding engine work, history, and origin metadata are gone,
 reconciliation removes the protected submission record and frees its retained
@@ -2570,6 +2577,29 @@ for at most one 1,000-candidate page, then other origins get a turn before a bus
 origin resumes. Only age-expired or excess-over-10,000 rows consume the retention
 candidate budget. Legacy orphan candidates exclude satellite roots and children
 in SQL before their bounded page is loaded into memory.
+Execution pruning scans at most 1,000 execution keys per slice, including young
+and protected generations. Each batch deletes at most those 1,000 generations;
+this is a work limit, not a generation-count retention policy. Failed pages are
+logged. Query cancellations, including statement timeouts, retry the same position
+with smaller pages, down to one execution. The smaller limit is retained for the
+rest of that scan; the next scan starts at 1,000 again. Lock timeouts and other
+failures, or cancellation of a single-execution page, skip that page until the
+next scan so they cannot block later pages. Pruning uses the maintenance
+connection, not the workflow runtime's connection pool, with server-side
+five-second statement and
+1.5-second lock deadlines. A timeout rolls back the batch. The number of history
+events in one generation is not capped; an individual generation that cannot
+finish within the deadline can remain unpruned despite aging past the retention
+window. Smaller pages can lengthen the scan, but other pending maintenance work
+still gets a turn between pruning slices.
+
+Pruning old generations holds instance-row locks until the batch transaction
+ends. Acknowledgements for those same workflows can wait even though their
+current generations are preserved and pruning uses a separate connection.
+Expect brief turn-latency spikes during scans; an expensive batch can hold those
+locks for close to the five-second statement deadline. The first scans after
+upgrade can cause more contention because backfill makes accumulated history
+eligible at once.
 
 Metadata queries have server-side deadlines and satellite operations have a
 15-second overall deadline. The candidate limit bounds materialized rows, not
@@ -2587,12 +2617,13 @@ Two Postmaster-context GUCs govern it (set in `postgresql.conf`, restart to appl
 pg_durable.reconcile_interval = 3600
 
 # Days a terminal instance is retained before reconciliation removes it (and its
-# engine record); also the age bound for reclaiming orphaned engine records.
-# 0 removes terminal instances as soon as the next pass runs.
+# engine record); also the age bound for orphaned engine records and completed
+# historical execution generations. The current execution is never pruned.
+# 0 makes cleanup eligible on the next pass.
 pg_durable.retention_days = 30
 ```
 
-Retention combines that window with a fixed hard cap:
+Terminal-instance retention combines that window with a fixed hard cap:
 
 - **Hard cap per origin — at most 10,000 terminal instances are retained, regardless of
   age.** The newest 10,000 terminal instances are kept; any beyond that are
@@ -2604,9 +2635,44 @@ Equivalently, a terminal instance is retained only while it is **both** among th
 newest 10,000 terminal instances **and** younger than `retention_days`; otherwise
 it is eligible for removal.
 
+Historical execution generations use only the age window, not a count cap.
+Pruning bounds how long completed history is retained, not its byte size: a
+busy workflow can still retain many generations within the window. An execution
+that has not continued into a new generation retains its current history.
+Completed child instances created by earlier loop iterations retain their current
+execution until the owning workflow is removed.
+Setting `reconcile_interval = 0` also disables execution pruning.
+
+On binary upgrade, the worker recovers missing completion timestamps for older
+continued generations from their recorded terminal history events. If an event
+is missing, it logs a migration warning and leaves that generation ineligible for
+age-based pruning rather than guessing its age. Malformed history that cannot be
+decoded as JSONB instead aborts the migration and identifies the affected schema,
+instance, execution, and event in the initialization error. The worker retries
+initialization; it does not skip the malformed history.
+
+This upgrade requires all old provider calls and transactions to finish before
+migrations run, not a rolling upgrade. Pause deletion/pruning jobs and stop and
+drain every provider caller, including clients and direct SQL callers as well as
+workers. A full PostgreSQL stop/start drains pg_durable's old server processes;
+also coordinate any external callers sharing the provider store. The new
+background worker applies migrations automatically. Wait for its readiness
+before resuming provider traffic, and use only updated external callers.
+
+The one-time backfill can delay readiness and generate substantial WAL on a large
+history store. New submissions and engine operations such as signals and
+cancellation must wait. Review retention settings before restarting: recovered
+timestamps may make many old generations eligible for deletion on the next scan.
+Allow sufficient time for the backfill: the `statement_timeout` in effect for
+`pg_durable.worker_role` in the control database must permit it to finish.
+Adjust the applicable role or database timeout defaults for the upgrade if
+necessary; otherwise startup retries the backfill instead of completing.
+See [upgrade and repair guidance](docs/upgrade-testing.md#historical-execution-pruning),
+including the special handling of databases that used earlier pre-release pins.
+
 Notes:
 
-- "Age" is measured from `completed_at` when set (instances that reached
+- Terminal-instance "age" is measured from `completed_at` when set (instances that reached
   `completed`), otherwise from `created_at`. `updated_at` is intentionally **not**
   used, because it is user-writable and would let a low-privilege user influence
   what is removed.

docs/upgrade-testing.md

@@ -358,6 +358,95 @@ what the upgrade script handles, and any backward compatibility considerations.
 
 ### v0.2.9 → v0.2.10 (unreleased)
 
+#### Historical execution pruning
+
+- **B1:** background reconciliation prunes completed non-current execution
+  generations using `pg_durable.retention_days`, including running workflows and
+  nested loops in every origin. The query uses existing provider tables, so no
+  extension upgrade DDL or new `df` schema detection is needed.
+- **Provider migrations:** the worker's startup `ApplyAll` applies migration `0027`
+  before starting the runtime. It timestamps `ContinuedAsNew` executions and
+  backfills older NULL timestamps from terminal history events. Unrecoverable
+  timestamps remain NULL with a warning; those generations are not age-pruned.
+  Malformed history that cannot be decoded as JSONB aborts migration `0027`
+  atomically and reports the schema, instance, execution, and event key. The
+  worker logs the initialization error and retries; malformed history is not
+  silently skipped or treated as a missing terminal event.
+  This dependency pair also includes `0023` (partial indexes), `0025` (fetch
+  leases), `0026` (renewal and acknowledgement lease locking), and `0028`
+  (cursor-based pruning and consistent instance-lock ordering). Readiness schema
+  version `4` supersedes main's version `3` submission ledger and prevents
+  backends from treating a pre-migration worker marker as ready.
+- **Drained upgrade:** pause delete/prune jobs and stop and drain every provider
+  caller, including workers, clients, maintenance jobs, and direct SQL callers.
+  Wait for in-flight calls and transactions to finish; stopping only workers
+  does not prevent an old client's deletion logic from removing a pending
+  continuation after backfill. A full PostgreSQL stop/start drains pg_durable's
+  old backend and worker processes. Also coordinate any external callers sharing
+  the provider schema. Restart PostgreSQL with the new extension library and
+  let its worker apply migrations before resuming provider traffic;
+  wait for readiness and restart only updated external callers. No separate
+  migration process is required for pg_durable.
+- **Backfill operations:** allow for initialization time and WAL growth on large
+  stores. Any worker-role/database statement timeout must permit the backfill to
+  finish, or startup retries it. Review retention cutoffs before restarting:
+  recovered timestamps can make many old generations eligible on the next scan.
+  Expect brief workflow turn-latency spikes during those first scans: pruning
+  holds candidate instance-row locks until each batch transaction ends, and
+  acknowledgements for the same running workflows wait on those locks. An
+  expensive batch can hold them for close to the five-second statement deadline;
+  using a maintenance connection does not remove this row-lock contention.
+  If an improperly drained upgrade left NULL timestamps behind, drain all
+  callers first and use the provider's
+  [documented data-only repair](https://github.com/microsoft/duroxide-pg/blob/79870890d1a4a14eb840ec1d31d0b6dff0716c0a/migrations/README.md#repairing-missing-continued-execution-timestamps).
+  Do not delete migration-ledger rows or re-run whole migrations to repair data.
+- **Bounded maintenance:** each slice scans at most 1,000 execution keys, including
+  protected rows. The provider owns deletion predicates and atomically prunes
+  that page on the maintenance connection under server-side deadlines. It first
+  checks eligibility without locking, so protected rows advance the cursor
+  without locking their instances. Deletion candidates still lock instances
+  before history and recheck eligibility after acquiring the locks, protecting
+  replacement instances and coordinating with acknowledgement and deletion.
+  Query cancellation retries the same cursor with a smaller page only after
+  rollback succeeds; the reduced limit lasts until that scan ends.
+  Lock failures and other errors, or a cancelled single-execution page, retain
+  the skip-and-retry-next-scan behavior. Runtime provider connections are not
+  borrowed.
+- **Inspection:** `df.instance_executions()` reads execution details and event
+  counts in one MVCC snapshot, so concurrent pruning does not produce spurious
+  per-generation not-found errors. Genuine query failures still raise errors.
+  The snapshot and pruning pre-scan depend on provider column names and types.
+  `execution_snapshot_matches_provider_api` checks snapshot rows against the
+  provider's public metadata API using a store created by its actual migrations
+  and populated through provider operations. The existing pruning retention and
+  failure-progress test exercises pre-scan key columns against the same migrated
+  schema contract. Run these regressions when changing the provider dependency.
+- **Replay:** no orchestration code, durable-operation inputs, or continuation
+  payloads change. Pruning preserves the current and running executions.
+- **Dependency pair:** the provider accepts duroxide `^0.1.32`; pg_durable chooses
+  the exact `=0.1.32` pin so replay-sensitive dependency updates remain deliberate.
+  The combined release with #429 needs later duroxide changes, so select exact,
+  compatible released runtime/provider pins for both PRs rather than treating
+  `0.1.32` as an upstream requirement. In the currently pinned runtime, shutdown
+  cancels and joins owned tasks; zero means immediate cancellation rather than
+  a drain grace period. Validate the real previous-binary lifecycle and worker
+  shutdown when updating this pair.
+- **Release blocker:** the provider is pinned to immutable commit
+  `79870890d1a4a14eb840ec1d31d0b6dff0716c0a` of
+  [microsoft/duroxide-pg#32](https://github.com/microsoft/duroxide-pg/pull/32).
+  Released provider `0.1.38` does not contain migrations `0027`/`0028`.
+  Replace the `DONOTLAND` git pin with an exact compatible released version and
+  refresh both dependency resolutions together before landing. No local Cargo
+  override is needed to build or test the current pinned commit.
+- **Pre-release databases:** revisions of microsoft/duroxide-pg#32 changed migration
+  `0027`/`0028` bodies without changing their numbers. A store that already
+  recorded those migrations from an earlier pin will not install the revised
+  functions just by loading this binary. Use fresh disposable provider schemas
+  for development validation. Preserving such a store requires a deliberate
+  forward migration; changing the readiness marker or deleting ledger entries
+  is not a substitute. The data-only repair above does not replace functions.
+  Check migration identity again when replacing the pre-release pin.
+
 #### `df.start()` admission quotas (#139)
 
 - **DDL/B1:** The upgrade adds `idx_instances_submitted_by` on

scripts/test-e2e-docker.sh

@@ -52,6 +52,7 @@ SKIP_TESTS=(
     "95_start_limits_rollback_cleanup"
     "96_start_limits_failed_metadata_cleanup"
     "88_multi_database_maintenance_backlog"
+    "89_execution_pruning"
     # Require the force-drop phase (one execution slot, reconciliation disabled).
     "78_multi_database_force_drop"
     "79_multi_database_remote_origin"

scripts/test-e2e-local.sh

@@ -221,7 +221,7 @@ phase_for_test() {
         46_connection_limit_startup_validation)
             echo "connlimit-startup"
             ;;
-        54_reconcile_orphans|76_multi_database_reconcile|95_start_limits_rollback_cleanup|96_start_limits_failed_metadata_cleanup)
+        54_reconcile_orphans|76_multi_database_reconcile|89_execution_pruning|95_start_limits_rollback_cleanup|96_start_limits_failed_metadata_cleanup)
             echo "reconcile"
             ;;
         88_multi_database_maintenance_backlog)