HTTP policy control (concurrency, etc)

.agents/skills/pg-durable-sql/SKILL.md

@@ -73,6 +73,15 @@ df.wait_for_signal(
 ### Control Flow
 
 ```sql
+-- HTTP concurrency constraint on a graph or subtree (0.2.10+).
+-- All enclosing limits apply; inner policies cannot raise outer/server limits.
+-- Scopes are shared across parallel branches and loop iterations.
+df.with_policy(fut TEXT, policy JSONB) → TEXT
+df.with_policy(
+    df.http('https://api.example.com/a', 'GET') & df.http('https://api.example.com/b', 'GET'),
+    '{"http":{"max_concurrency":1}}'::jsonb
+)
+
 -- Sequence (function variant of ~>)
 df.seq(a TEXT, b TEXT) → TEXT
 

CHANGELOG.md

@@ -6,8 +6,49 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc
 
 ## [Unreleased]
 
+> **Pre-release databases:** The temporary duroxide-pg pin includes migration
+> `0029` without the pending `0027`/`0028` migrations. Its recorded migration
+> ledger is not a prefix of the combined provider stack, so those stores cannot
+> upgrade to that stack in place. Use this branch only with disposable databases,
+> including when testing a binary-only upgrade. Before retaining a database,
+> replace the pin with a compatible combined/released provider. See
+> [pre-release provider guidance](docs/upgrade-testing.md#pre-release-provider-databases).
+
+### Added
+
+- **HTTP execution limits:** `pg_durable.max_http_concurrency` bounds ordinary
+  and multipart HTTP executions together across all origin databases, while
+  `pg_durable.max_activity_concurrency` controls total activity capacity.
+  Both default to two and require a server restart. Capacity-blocked HTTP work
+  stays durably queued without retaining execution or origin-connection slots
+  between admission attempts. Large deferred backlogs can still delay other
+  work; these limits do not guarantee scheduling fairness.
+- **Workflow HTTP policies:** `df.with_policy()` adds lexical concurrency
+  constraints that compose with enclosing scopes and the global ceiling.
+  Policies are shared by branches and loop generations and survive restart.
+
+### Fixed
+
+- HTTP permit cleanup no longer panics on inconsistent counters or a poisoned
+  lock. Recovery is logged, existing reservations are retained, and one activity
+  failure does not permanently poison HTTP admission across all databases.
+- PostgreSQL shutdown interrupts runtime startup and readiness-publication I/O
+  rather than waiting on connections to the stopping server.
+- The merged activity-admission runtime normally resolves activity metadata
+  without rereading execution history on each deferral. Metadata-read failures
+  are handled as infrastructure failures before invoking the activity.
+- The pinned PostgreSQL provider orders worker items by visibility time and then
+  queue ID, so a saturated deferred HTTP backlog yields to already-ready SQL,
+  graph-loading, and status work. Worker readiness version `4` gates the bundled
+  provider migrations, including the queue-ordering index.
+
 ### Security
 
+- `df.start()` now checks the submitter's HTTP function privileges before
+  persisting or enqueueing any workflow, including raw JSON and untaken branches.
+  URL/endpoint and ordinary/multipart overloads retain their separate grants.
+  Both transaction modes reject missing privileges with SQLSTATE `42501`.
+  Execution-time checks remain in place for later revocation and older workflows.
 - **Start admission quotas (#139):** `df.max_concurrent_per_user` (default
   `100`) and `df.max_instances_per_user` (default `10000`) limit active and
   retained instances per submitting role and origin database. Both are

Cargo.toml

@@ -36,8 +36,8 @@ uuid = { version = "1.0", features = ["v4", "serde"] }
 percent-encoding = "2.3"
 
 # duroxide integration
-duroxide = "=0.1.30"
-duroxide-pg = "=0.1.34"
+duroxide = "=0.1.32"
+duroxide-pg = "=0.1.38"
 tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] }
 
 # For ExecuteSQL activity (connecting back to PostgreSQL)
@@ -67,12 +67,19 @@ url = "2.5"
 # For df.http_multipart() — base64-decodes part payloads carried in the config JSON.
 base64 = "0.23"
 
-# For duroxide tracing integration
+# For duroxide tracing integration and background worker diagnostics
+tracing = "0.1"
 tracing-subscriber = { version = "0.3", features = ["env-filter"] }
 
 [dev-dependencies]
 pgrx-tests = "=0.16.1"
 
+# Temporary integration pins: merged admission support and its pending PostgreSQL companion.
+# DONOTLAND: this provider pin requires disposable databases; see docs/upgrade-testing.md.
+[patch.crates-io]
+duroxide = { git = "https://github.com/microsoft/duroxide", rev = "36b789d802f98963241a59ee7706e3ce029d583f" }
+duroxide-pg = { git = "https://github.com/microsoft/duroxide-pg", rev = "9ae067b34abc4c1f1f1436329973fe37731edb2a" }
+
 [profile.dev]
 panic = "unwind"
 

USER_GUIDE.md

@@ -1074,6 +1074,13 @@ URLs and endpoints. After upgrading an existing installation, run the helper for
 roles that need the newly added endpoint support. Existing URL calls retain their
 permissions.
 
+`df.start()` checks the submitting `current_user` for the required HTTP function
+privileges in the origin database before creating or enqueueing an instance.
+This includes raw JSON nodes, multipart requests, and HTTP nodes inside branches
+that may never run. URL and endpoint overloads require their respective grants.
+Both transaction modes reject missing privileges with SQLSTATE `42501`.
+Execution-time checks remain in place for later revocation and older workflows.
+
 The path starts with one `/` and is appended to the server's base path prefix:
 `https://host/api/` plus `/items` becomes `https://host/api/items`. Absolute URLs,
 protocol-relative paths (`//host`), backslashes, fragments, whitespace, dot
@@ -3109,6 +3116,112 @@ enqueues are recorded even while limits are disabled.
 pg_durable uses separate limits for control management, provider, origin metadata,
 SQL execution, and independent-start connections.
 
+### Activity and HTTP Concurrency
+
+Connection budgets are separate from execution capacity:
+
+| Setting | Default | Scope |
+|---------|---------|-------|
+| `pg_durable.max_activity_concurrency` | 2 | All concurrently executing activities, including SQL, HTTP, and metadata work |
+| `pg_durable.max_http_concurrency` | 2 | HTTP and multipart HTTP activities combined |
+
+Both settings accept integers from 1 to 1000 and require a PostgreSQL restart.
+The HTTP ceiling is shared across users, workflows, and databases served by the
+cluster's single runtime. It counts an HTTP execution from request preparation
+through response-body processing and optional sink storage, not idle pooled
+connections. It is not a requests-per-second limit.
+
+Requests waiting for HTTP capacity stay in the durable queue. Between admission
+attempts they retain neither an execution slot nor an origin-database connection.
+They do not fail solely because of capacity contention or consume their request
+timeout while waiting. Cancellation still applies. Actual request failures
+retain their usual behavior. A committed deferral restores its fetch attempt;
+interrupted deferrals or metadata/storage failures can still consume the retry
+budget and eventually fail an activity before its request is sent.
+
+To reserve activity slots from admitted HTTP work, set the HTTP ceiling below
+total activity capacity:
+
+```ini
+pg_durable.max_activity_concurrency = 8
+pg_durable.max_http_concurrency = 4
+```
+
+Increasing activity capacity does not increase database connection budgets.
+If both settings are equal, active HTTP requests can occupy all activity slots;
+the unchanged defaults preserve the previous two-activity execution capacity.
+There is no strict fairness guarantee between workflows.
+
+Deferred HTTP work yields its queue position to work that was already ready,
+so repeated capacity deferrals cannot indefinitely overtake that work. A large
+backlog can still delay SQL and status updates: each activity turn must wait
+its turn, and admission polls reread request inputs. Activity metadata normally
+uses a small instance lookup; it falls back to reading the requested execution's
+history when that lookup is unavailable or describes a different execution.
+Keep fan-out and intermediate payloads bounded; these limits are not
+tenant-isolation or scheduling-fairness guarantees.
+
+Waiting for capacity is not idle: each admission retry fetches and updates a
+queue row, reads activity metadata, and commits a deferral. Repeated updates
+generate index writes and WAL for as long as saturation lasts, even though no
+HTTP request is sent and successful deferrals do not consume execution attempts.
+Monitor queue churn and WAL volume as well as request concurrency. See
+[measured admission polling costs](docs/http-security.md#admission-polling-cost)
+for one saturated-backlog measurement. Retry-delay tuning is separate work;
+longer delays reduce polling at the cost of slower admission after capacity frees.
+
+HTTP capacity is released when the admitted handler returns or its task is
+dropped. Background cleanup of origin metadata connections can finish later;
+it retains the separate origin-connection reservation, not HTTP capacity.
+
+Pending requests use the configured limits after restart; recorded results
+replay unchanged. A crash or cancellation cannot guarantee that a remote server
+stops processing an already-sent request. Limits bound local executions, not
+remote effects, and do not provide exactly-once HTTP delivery.
+
+### Workflow HTTP Policies
+
+Use `df.with_policy(fut TEXT, policy JSONB) RETURNS TEXT` to add a local HTTP
+concurrency constraint to a whole workflow or any subtree. Like other DSL
+functions, it builds a graph; only `df.start()` executes it.
+
+```sql
+SELECT df.start(
+    df.with_policy(
+        df.join3(
+            df.http('https://api.github.com/repos/microsoft/pg_durable', 'GET'),
+            df.http('https://api.github.com/repos/microsoft/duroxide', 'GET'),
+            df.http('https://api.github.com/repos/microsoft/duroxide-pg', 'GET')
+        ),
+        '{"http": {"max_concurrency": 2}}'::jsonb
+    ),
+    'repository-metadata'
+);
+```
+
+All enclosing limits apply together. If an outer policy allows two HTTP
+executions and each of three inner branches allows one, at most two run across
+the outer scope and at most one runs in each branch. Inner policies cannot
+raise an outer limit or the server-wide ceiling. Applying another policy to the
+same subtree root keeps the smaller limit.
+
+Ordinary and multipart requests share the scope. Parallel branches, internal
+child workflows, and loop iterations use the same enclosing allowance, including
+after restart. Separately started workflows have independent local budgets, and
+reusing a graph fragment in two positions creates separate scopes. A workflow
+started by another workflow's SQL does not implicitly inherit its caller's policy.
+
+The policy must be a JSON object. The supported option is
+`http.max_concurrency`, an integer from 1 to 4294967295; omission of `http` adds
+no constraint. Unknown keys, JSON `null`, arrays, zero, and negative or noninteger
+limits are errors. SQL `NULL` for either argument returns SQL `NULL`. Limits are
+captured with the graph, not read from session settings at execution time.
+`df.explain()` shows the enclosing scope IDs and limits on each HTTP node.
+
+Available after `ALTER EXTENSION pg_durable UPDATE TO '0.2.10'`.
+HTTP privileges and destination restrictions are unchanged. Request timeout caps,
+admission timeouts, workflow deadlines, and rate limits are not policy options.
+
 ### Connection Architecture
 
 The main connection categories are:

docs/E2E_TESTING.md

@@ -89,6 +89,11 @@ restart with the phase's configuration, not a rebuild.
 | `48_http_allow_all.sql` | `http-allow-all` | Unrestricted mode admits plaintext HTTP, unlisted domains, and private destinations for both HTTP activities |
 | `69_http_allowed_domains.sql` | `http-custom-domains` | Custom allow-list enforcement, malformed startup rejection, protected mode settings, and restart-only policy changes |
 | `70_http_allowed_domains_empty.sql` | `http-empty-domains` | Empty restricted-mode allow-list denies all domains |
+| `90_http_concurrency.sql` | `http-concurrency` | Measures HTTP ceilings of one and two; covers both HTTP functions, satellite origins, SQL headroom, queued and active cancellation, queue-independent request timeouts, blocked sink writes, and permit release after failure |
+| `91_http_policies.sql` | `http-concurrency` | Measures nested and sibling policy limits, independent workflow budgets, nonblocking admission, and shared scopes across loop generations |
+| `92_http_restart.sql` | `http-concurrency` | Two-stage test restarts PostgreSQL with active and queued HTTP work; drains old fixture requests, re-arms the gate and resets peak accounting, then measures a lexical cap of one independently of a global ceiling of three |
+| `93_http_admission_progress.sql` | `http-concurrency` | Holds the sole HTTP slot while 500 requests are submitted with permission, then revokes HTTP grants; verifies unrelated SQL progress before release and execution-time denial afterward |
+| `94_http_submission_privileges.sql` | `standard` | Rejects raw/nested HTTP across all four function variants and both transaction modes before persistence/enqueue, including under hostile search paths; verifies inherited grants and SECURITY DEFINER identity |
 
 ## Test Structure
 

docs/api-reference.md

@@ -433,6 +433,31 @@ df.secret('partner_api', 'token') || '{"prefix":"Bearer "}'::jsonb
 
 See [Explicit Secret Bindings](../USER_GUIDE.md#explicit-secret-bindings).
 
+### df.with_policy(fut, policy)
+
+**Added in 0.2.10.**
+
+Returns a graph with a lexical HTTP concurrency constraint; does not execute it.
+
+| Parameter | Type | Auto-wrap | Description |
+|-----------|------|-----------|-------------|
+| `fut` | TEXT | Yes | Any workflow graph or SQL node |
+| `policy` | JSONB | No | An object such as `{"http":{"max_concurrency":2}}` |
+
+`http.max_concurrency` must be a positive 32-bit unsigned integer. `{}` adds no
+constraint. Unknown keys, JSON `null`, malformed objects, arrays, and invalid
+limits raise an error; SQL `NULL` for either argument returns SQL `NULL`.
+
+Every HTTP or multipart execution reserves capacity against all enclosing
+scopes and the server-wide HTTP ceiling. Nested policies only constrain, never
+override. Reapplying a policy to the same root keeps the smaller limit. Scopes
+are shared by that subtree's parallel branches and loop generations, but not by
+separately started workflows or different occurrences of a reused fragment.
+Policies survive restart and are shown by `df.explain()`.
+
+See [Workflow HTTP Policies](../USER_GUIDE.md#workflow-http-policies) for examples
+and the distinction from administrator limits and request timeouts.
+
 ### df.with_http_options(fut, options)
 
 **Added in 0.2.9.**
@@ -527,6 +552,13 @@ database privileges. Install in `pg_durable.database` first and wait for control
 readiness before creating satellites; see
 [Multi-Database Support](../USER_GUIDE.md#multi-database-support).
 
+Every HTTP or multipart node, including nodes in untaken branches, requires the
+submitting `current_user` to hold `EXECUTE` on its matching URL/endpoint function
+in the origin database. Missing privileges raise SQLSTATE `42501` before
+instance/node persistence or enqueueing, including for raw graph JSON and
+`transaction_mode => 'new'`. Inherited grants and `SECURITY DEFINER` identities
+follow PostgreSQL rules. Activities still recheck privileges at execution time.
+
 Non-superuser starts are subject to `df.max_concurrent_per_user` (default `100`)
 and `df.max_instances_per_user` (default `10000`) in their origin database.
 Rejection raises SQLSTATE `53400` with a tuning hint before any insert or enqueue.
@@ -924,6 +956,31 @@ inherit the caller's session `SET` overrides. See
 
 ---
 
+### pg_durable.max_activity_concurrency / pg_durable.max_http_concurrency
+
+| Setting | Meaning | Default | Range |
+|---------|---------|---------|-------|
+| `pg_durable.max_activity_concurrency` | Concurrent activities in the shared runtime, including SQL, HTTP, and metadata | `2` | `1` to `1000` |
+| `pg_durable.max_http_concurrency` | Concurrent HTTP and multipart HTTP activities combined, across all origin databases | `2` | `1` to `1000` |
+
+Both are `POSTMASTER` settings and require a PostgreSQL restart. An HTTP slot is
+held through preparation, response-body handling, and optional sink storage.
+Requests awaiting capacity remain queued without retaining activity slots or
+origin connections between admission attempts; admission waiting does not consume
+the request timeout.
+Contention does not fail the request or count as an execution retry.
+
+HTTP concurrency cannot exceed total activity capacity. Set the HTTP ceiling
+lower than activity capacity to prevent admitted HTTP from consuming every
+activity slot. Committed deferrals move behind already-ready work, allowing
+non-HTTP activities to progress under saturation. Large backlogs still add
+latency; spare slots are not a per-workflow fairness or throughput guarantee.
+These settings do not raise connection budgets or impose request-rate limits. No extension SQL
+update is required for existing installations to use these settings with the
+new binary. See [Activity and HTTP Concurrency](../USER_GUIDE.md#activity-and-http-concurrency).
+
+---
+
 ### pg_durable.http_security
 
 Server-wide outbound HTTP policy for `df.http()` and `df.http_multipart()`.