[Thom Chiovoloni (thomcc-work)]

Summary

Add administrator-enforced HTTP concurrency limits and workflow-local policies, with nonblocking admission so requests waiting for capacity remain durably queued rather than occupying activity slots.

Server-wide limits

Add two restart-only GUCs, both defaulting to 2:

• pg_durable.max_activity_concurrency: total concurrent activities, including SQL, HTTP, and metadata work.
• pg_durable.max_http_concurrency: combined HTTP and multipart HTTP capacity across all users, workflows, and origin databases.

Defaults preserve the previous two-activity execution capacity. Administrators can set the HTTP ceiling below total activity capacity to leave room for non-HTTP work.

Workflow-local policies

Add df.with_policy(fut, policy) to constrain HTTP concurrency within an entire workflow or a subtree:

SELECT df.start(
    df.with_policy(
        df.join(
            df.http('https://api.github.com/repos/microsoft/pg_durable', 'GET'),
            df.http('https://api.github.com/repos/microsoft/duroxide', 'GET')
        ),
        '{"http": {"max_concurrency": 1}}'::jsonb
    ),
    'repository-metadata'
);

• All enclosing scopes and the global ceiling apply together; admission reserves all required capacity atomically or reserves none.
• Inner policies cannot relax outer limits. Reapplying a policy to the same subtree root keeps the smaller limit.
• Parallel branches and loop generations share their enclosing scopes, including after restart. Separate starts and separate occurrences of reused fragments have independent local scopes.
• Policy input is strictly validated, and df.explain() displays applicable scope IDs and limits.

Execution behavior

• Admission occurs before origin routing and credential lookup. Permits cover request preparation, response-body processing, and optional sink storage.
• Capacity waiting does not consume the request timeout. Committed deferrals do not record activity results or spend the fetch-attempt budget; infrastructure failures can still consume attempts.
• The provider orders deferred work behind already-ready items, preventing a saturated HTTP backlog from indefinitely overtaking unrelated SQL and metadata activities.
• HTTP authorization, destination restrictions, and credential isolation remain unchanged.
• Also make runtime startup/publication I/O interruptible to fix a shutdown race found during validation.

This does not add rate limits, timeout/deadline policies, per-tenant fairness, or exactly-once/remote-quiescence guarantees. Large backlogs still add latency.

Dependencies

Uses immutable Git pins for:

• Add nonblocking activity admission control duroxide#74 (on main now) — merged admission support.
• Prevent deferred worker items from starving ready work duroxide-pg#51 — pending PostgreSQL queue-ordering fix.

These pins allow combined testing and review before upstream releases; they are temporary integration dependencies.

Upgrade and compatibility

• Global limits apply after a binary update, including on supported older extension schemas.
• df.with_policy() requires ALTER EXTENSION pg_durable UPDATE TO '0.2.10'.
• Policies use existing HTTP configuration storage: no new policy tables, executable node types, or orchestration permit operations.
• Unconfigured HTTP payloads retain their existing bytes for replay compatibility.
• Worker readiness advances to 4, published after the bundled provider migrations complete.
• Migration guidance covers index-build locking, concurrent prebuilding, and older binaries rejecting newer provider migration state on startup.

[Copilot]

🔵 Needs a closer look

The runtime pin relies on an unmerged dependency and changes admission, shutdown, and replay behavior, requiring PostgreSQL shutdown, epoch-replacement, HTTP restart, and N-1 replay integration results beyond source inspection.

No open issues 🎉

What changed in this PR

Adds server-wide and workflow-scoped HTTP concurrency controls to pg_durable’s shared runtime, with durable queue deferral when capacity is unavailable. The PR depends on the admission support in microsoft/duroxide#74.

Changes:

• Adds configurable activity and HTTP limits, plus composable subtree policies through df.with_policy().
• Integrates admission checks and shutdown-interruptible runtime initialization.
• Adds concurrency, cancellation, and restart tests, alongside API and operational documentation.

File	Description
USER_GUIDE.md	Explains concurrency settings, workflow policies, and limitations.
tests/​e2e/​sql/​92_http_restart.sql	Tests interrupted HTTP recovery and persisted policy enforcement.
tests/​e2e/​sql/​91_http_policies.sql	Tests nested scopes, independent budgets, loops, and races.
tests/​e2e/​sql/​90_http_concurrency.sql	Tests global limits, cancellation, SQL headroom, and permit lifetimes.
tests/​e2e/​http_origin_server.py	Adds gated responses and concurrency counters.
src/​worker.rs	Configures runtime capacity and interruptible startup.
src/​types.rs	Carries policies and materializes HTTP scope metadata.
src/​registry.rs	Registers admission hooks for both HTTP activities.
src/​policies.rs	Implements policy validation, scope accounting, and admission guards.
src/​lib.rs	Registers concurrency settings and adds integration tests.
src/​explain.rs	Displays HTTP scope limits in graph explanations.
src/​dsl.rs	Adds the df.with_policy() graph helper.
sql/​pg_durable--0.2.9--0.2.10.sql	Exposes the policy helper during extension upgrades.
scripts/​test-e2e-local.sh	Runs HTTP concurrency scenarios and restart orchestration.
scripts/​test-e2e-docker.sh	Skips tests requiring the local TLS fixture.
docs/​upgrade-testing.md	Documents schema compatibility, replay behavior, and validation.
docs/​multi-database-installation.md	Updates runtime shutdown guarantees and remaining limitations.
docs/​http-security.md	Explains capacity enforcement and its security boundaries.
docs/​grammar.md	Documents policy annotations in the DSL grammar.
docs/​E2E_TESTING.md	Lists the new HTTP test scenarios.
docs/​api-reference.md	Documents the policy API and concurrency settings.
CHANGELOG.md	Records HTTP limits, policies, and startup shutdown handling.
Cargo.toml	Updates Duroxide and pins the temporary admission revision.
Cargo.lock	Locks the updated runtime dependency graph.
.agents/​skills/​pg-durable-sql/​SKILL.md	Adds workflow policy syntax and usage guidance.

_{Review effort: Balanced}