Add nonblocking activity admission control

CHANGELOG.md

@@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Opt-in activity resource admission through `ActivityAdmission`, the default
+  `ActivityHandler::try_admit` method, and the `register_with_admission` and
+  `register_typed_with_admission` registration helpers.
+  Admission can retain a guard for the handler's lifetime or durably defer
+  execution without occupying a worker slot or recording an activity result.
+  Positive retry delays are clamped to 1 ms–24 hours. Only committed deferrals
+  restore their fetch attempt; failed or interrupted deferrals may still use
+  the poison-message budget, even when the handler never starts. Uncommitted
+  deferral delays may be lost on lock expiry. Successfully observed deferrals
+  have a separate `duroxide_activity_admission_deferrals_total` metric and the
+  `MetricsProvider::activity_admission_deferrals()` accessor for in-process
+  counts without changing the public `MetricsSnapshot` shape. (#74)
+- Admission errors, callback panics, and zero retry delays produce terminal
+  `ActivityFailed` application results with `retryable: false`, without invoking
+  the handler. Orchestration-level retry helpers still retry these errors
+  according to their policy. (#74)
+
+### Fixed
+
+- SQLite worker selection now orders eligible items by visibility time, then
+  queue ID, so a deferred backlog cannot repeatedly overtake already-ready work.
+  Migration `20240113000000` adds the supporting `visible_at, id` index to new
+  and existing file-backed databases.
+  Other providers must implement equivalent queue progress semantics; existing
+  stored histories and work-item formats are unchanged. SQLite workers must all
+  be upgraded to provide this behavior in a mixed-version deployment. (#74)
+- Activity metadata uses the lightweight instance lookup for the current
+  execution instead of re-reading its full history on every attempt. Activities
+  from earlier executions read their own execution's metadata. Both lookup paths
+  record provider-operation metrics. If the execution-history fallback fails,
+  processing now abandons the fetch with its attempt counted and reports an
+  infrastructure error instead of invoking the activity with `unknown` metadata.
+  This applies to ordinary and admission-controlled activities; repeated read
+  failures can exhaust the fetch budget before either handler runs. Metadata
+  failures are logged separately from acknowledgment and deferral failures. (#74)
+- Failed deferral retries preserve the requested delay and attempt-restoration
+  flag. (#74)
+- Relative SQLite queue delays saturate instead of overflowing their timestamp
+  representation. (#74)
+
 ## [0.1.32] - 2026-10-11
 
 **Release:** <https://crates.io/crates/duroxide/0.1.32>

docs/ORCHESTRATION-GUIDE.md

@@ -242,6 +242,106 @@ activities.register("CallExternalAPI", |ctx: ActivityContext, request: String| a
 | `ctx.is_cancelled()` | Returns `true` if cancellation was requested |
 | `ctx.cancelled()` | Returns a `Future` that completes when cancelled (use in `select!`) |
 
+### Activity Admission
+
+Use `ActivityRegistry::builder().register_with_admission(name, admission, handler)`
+to impose resource limits without occupying worker slots while waiting.
+The synchronous admission callback receives `&ActivityContext` and the activity's
+input as `&str`. It must not block or perform I/O.
+
+```rust
+use duroxide::runtime::{ActivityAdmission, registry::ActivityRegistry};
+use std::{sync::Arc, time::Duration};
+use tokio::sync::{Semaphore, TryAcquireError};
+
+let capacity = Arc::new(Semaphore::new(4));
+let activities = ActivityRegistry::builder()
+    .register_with_admission(
+        "Limited",
+        move |_ctx, _input| match capacity.clone().try_acquire_owned() {
+            Ok(permit) => Ok(ActivityAdmission::admit(permit)),
+            Err(TryAcquireError::NoPermits) => {
+                Ok(ActivityAdmission::Defer(Duration::from_millis(100)))
+            }
+            Err(error) => Err(error.to_string()),
+        },
+        |_ctx, input| async move { Ok(input) },
+    )
+    .build();
+```
+
+`admit(guard)` retains a resource reservation until the handler returns or its
+task is dropped, including error, panic, and cancellation. If several limits
+apply, reserve all of them atomically or defer holding none. Detached work is
+not protected after the handler finishes.
+
+For typed input and output, use
+`register_typed_with_admission(name, admission, handler)`. The admission callback
+receives `&In`; the handler receives an owned `In` and returns `Result<Out, String>`.
+It uses the same codec as `register_typed`, including raw-string compatibility.
+Invalid input fails before either callback runs. Input is decoded for each
+admission check and again when invoking an admitted handler, so deserialization
+must be side-effect-free.
+
+Using the imports above, the same semaphore limit can protect a typed activity:
+
+```rust
+let capacity = Arc::new(Semaphore::new(4));
+let activities = ActivityRegistry::builder().register_typed_with_admission(
+    "Limited",
+    move |_ctx, _input: &u32| match capacity.clone().try_acquire_owned() {
+        Ok(permit) => Ok(ActivityAdmission::admit(permit)),
+        Err(TryAcquireError::NoPermits) => Ok(ActivityAdmission::Defer(Duration::from_millis(100))),
+        Err(error) => Err(error.to_string()),
+    },
+    |_ctx, input: u32| async move { Ok(input) },
+).build();
+```
+
+`Defer(delay)` requires a positive delay. The runtime clamps it to 1 ms through
+24 hours (`ActivityAdmission::MAX_DEFER_DELAY`) before passing it to any provider.
+Computed delays such as `deadline.saturating_duration_since(now)` can become zero
+and fail that activity attempt; use `.max(Duration::from_millis(1))` when zero
+should mean retry soon.
+Deferral leaves the original activity pending in durable storage, frees the
+worker to fetch other work, and does not record an activity result.
+A **committed** deferral restores its fetch attempt. Crashes, shutdown aborts,
+and failed abandons can still consume the poison-message attempt budget, so
+leave room for recovery attempts. This is a fetch-attempt budget, not a count of
+handler starts: an activity can become poison without ever invoking its handler.
+Failures loading activity metadata before admission also consume attempts.
+Deferral is not a retry of an executed activity.
+
+An admission error, panic, or zero delay fails the activity rather than invoking
+the handler. These deliberately use the same durable `ActivityFailed` result as
+handler errors, with application classification and `retryable: false`.
+`Defer` is the retry signal for temporary capacity shortages; `Err` means a
+permanent rejection, while panic and zero delay indicate faulty admission logic.
+Automatically retrying those failures as infrastructure backpressure could
+repeat invalid input or broken configuration indefinitely. The orchestration
+can catch the activity failure and choose its own recovery.
+`schedule_activity_with_retry` receives only an error message, so it retries
+admission failures regardless of `retryable: false`, scheduling fresh attempts
+and rerunning admission up to the policy's attempt limit.
+
+Storage errors during deferral are infrastructure errors; an immediate fallback
+abandon preserves the same delay and attempt-restoration flag. If that also
+fails, recovery waits for the work lock to expire, which can happen before the
+requested delay. A requested delay is not a durable "not before" guarantee until
+its abandon commits.
+Deferrals whose abandon returns success increment
+`duroxide_activity_admission_deferrals_total`, not the activity execution counter;
+a commit whose response is lost can restore the attempt without being counted.
+The "Activity started" log is emitted only after admission succeeds.
+
+Admission runs outside orchestration replay; completed results are unaffected.
+Each recovered execution attempt is admitted again. Resource counters belong
+to the application and are not automatically coordinated across runtimes.
+Keep total worker capacity above the admitted activity ceiling if other work
+needs headroom. Providers must allow already-ready work to progress past deferred
+items, including after a deferral's delay expires. This is not a guarantee of
+equal throughput or admission fairness between workflows.
+
 ### The Replay Model
 
 
@@ -2693,4 +2793,3 @@ async fn fast_updates(ctx: OrchestrationContext, items_json: String) -> Result<(
 ---
 
 **You now have everything needed to build production-grade durable workflows!** 🎉
-

docs/duroxide-telemetry-spec.md

@@ -184,6 +184,31 @@ by (activity_name)
 
 ---
 
+#### `duroxide_activity_admission_deferrals_total` (Counter)
+
+**Description:** Admission deferrals whose abandon returned success, including
+a successful fallback after an abandon error.
+
+**Labels:**
+- `activity_name` - Registered activity name
+- `tag` - Routing tag, or `"default"` for untagged work
+
+**Semantics:** This counts observed successful deferrals, not handler executions.
+Deferrals do not contribute activity execution counts or duration samples.
+A commit with a lost response may not be counted. Abandon failures count as
+`infra_error`; admission `Err`, panic, and zero delay produce terminal
+`ActivityFailed` application results and count as `app_error`.
+
+The in-process accessor is `MetricsProvider::activity_admission_deferrals()`,
+without adding fields to the existing public `MetricsSnapshot`.
+
+**Example:**
+```promql
+sum by (activity_name, tag) (rate(duroxide_activity_admission_deferrals_total[5m]))
+```
+
+---
+
 ### 1.3 Provider (Storage) Metrics
 
 #### `duroxide_provider_operation_duration_seconds` (Histogram)
@@ -196,6 +221,11 @@ by (activity_name)
 
 **Use Case:** Database performance monitoring, identify slow queries
 
+Activity metadata is included with `operation="get_instance_info"` for the
+lightweight lookup and `operation="read_with_execution"` for the history
+fallback, each with `status="success"` or `"error"`. Errors also increment
+`duroxide_provider_errors_total` with the same operation label.
+
 **Example:**
 ```promql
 # p95 database operation latency

docs/metrics-specification.md

@@ -28,6 +28,7 @@ This document provides a complete reference of all metrics emitted by duroxide v
 | `duroxide_worker_queue_depth` | Gauge | Runtime | _(none)_ | ✅ `test_queue_depth_gauges_tracking` | ✅ Code Review | Observable gauge with callback, initialized via `initialize_gauges()` |
 | **Activity Execution** |
 | `duroxide_activity_executions_total` | Counter | Activity | `activity_name`, `outcome`, `retry_attempt`, `tag` | ✅ `metrics_capture_activity_and_orchestration_outcomes` | ✅ Code Review | Tracks all outcomes including app_error |
+| `duroxide_activity_admission_deferrals_total` | Counter | Activity | `activity_name`, `tag` | ✅ `admission_deferral_metric_has_activity_and_tag_labels` | ✅ Recorder test | Counts observed successful deferral abandons, not executions |
 | `duroxide_activity_duration_seconds` | Histogram | Activity | `activity_name`, `outcome`, `tag` | ✅ `test_activity_duration_tracking` | ✅ Code Review | Buckets: 0.01-300s |
 | `duroxide_activity_infrastructure_errors_total` | Counter | Activity | `activity_name` | ✅ `metrics_capture_activity_and_orchestration_outcomes` | ✅ Code Review | Infra-specific errors |
 | `duroxide_activity_configuration_errors_total` | Counter | Activity | `activity_name` | ✅ `test_error_classification_metrics` | ✅ Code Review | Config-specific errors |
@@ -461,6 +462,29 @@ topk(5,
 
 ---
 
+### 2.5 `duroxide_activity_admission_deferrals_total` (Counter)
+
+**Description:** Admission deferrals whose abandon operation returned success,
+including a successful fallback after an abandon error.
+
+**Labels:**
+- `activity_name` (string) - Activity name
+- `tag` (string) - Activity routing tag, or `"default"` for untagged work
+
+**Semantics:** Deferrals do not count as activity executions or produce activity
+duration samples. Failed abandons count as infrastructure errors; a storage
+commit whose response is lost may not increment this counter. Admission `Err`,
+panic, and zero delay instead produce terminal application failures (`app_error`).
+
+For in-process diagnostics, use `MetricsProvider::activity_admission_deferrals()`;
+the public `MetricsSnapshot` shape is unchanged.
+
+```promql
+sum by (activity_name, tag) (rate(duroxide_activity_admission_deferrals_total[5m]))
+```
+
+---
+
 ## 3. Sub-Orchestration Metrics
 
 > **Session Metrics (Future):** Session-specific metrics (session claims, migrations,
@@ -531,6 +555,8 @@ histogram_quantile(0.95,
   - `"save_event"` - Save event to history
   - `"create_instance"` - Create new orchestration instance
   - `"query"` - Query operation
+  - `"get_instance_info"` - Lightweight activity metadata lookup
+  - `"read_with_execution"` - Execution-history metadata fallback
 - `status` (string) - Operation status:
   - `"success"` - Operation succeeded
   - `"error"` - Operation failed
@@ -566,6 +592,7 @@ histogram_quantile(0.99,
   - `"deadlock"` - Database deadlock
   - `"corruption"` - Data corruption
   - `"constraint_violation"` - Database constraint violation
+  - `"other"` - Errors not classified as timeout, connection, or deadlock
 
 **Purpose:** Database health monitoring, alerting on infrastructure issues
 

docs/observability-guide.md

@@ -195,6 +195,7 @@ All metrics follow Prometheus naming conventions with `_total` for counters and
 | `duroxide_active_orchestrations` | Gauge | Orchestration | _(none)_ | **Current** active count |
 | **Activity Execution** |
 | `duroxide_activity_executions_total` | Counter | Activity | activity_name, outcome, retry_attempt | Track success/failure rates |
+| `duroxide_activity_admission_deferrals_total` | Counter | Activity | activity_name, tag | Track admission deferrals whose abandon returned success |
 | `duroxide_activity_duration_seconds` | Histogram | Activity | activity_name, outcome | Activity latency p95/p99 |
 | `duroxide_activity_errors_total` | Counter | Activity | activity_name, error_type, retryable | Detailed error tracking |
 | **Sub-Orchestration** |
@@ -265,6 +266,23 @@ All metrics follow Prometheus naming conventions with `_total` for counters and
 
 #### Activity Execution Metrics
 
+**`duroxide_activity_admission_deferrals_total`** (Counter)
+- **Description**: Admission deferrals whose abandon returned success, including a successful fallback after an abandon error
+- **Labels**: `activity_name`, `tag`
+- **Use**: Identify resource-constrained activities without counting deferrals as executions
+
+For an in-process count, use `MetricsProvider::activity_admission_deferrals()`.
+The existing public `MetricsSnapshot` fields remain unchanged.
+
+The "Activity started" log appears only after admission succeeds. Admission
+rejections, panics, and zero retry delays count as `app_error`; storage failures
+while deferring count as `infra_error`, even if the fallback abandon succeeds.
+Uncommitted deferrals do not increment the deferral counter and may consume a
+fetch attempt on recovery. A deferral that commits before shutdown or a lost
+response can restore its attempt without incrementing this counter. The counter
+is not a durable audit of all storage commits. Activity metadata-read failures
+before admission count as infrastructure errors, not deferrals.
+
 **`duroxide_activity_executions_total`** (Counter)
 - **Description**: Activity execution attempts
 - **Labels**: `activity_name`, `outcome` (success|app_error|infra_error|config_error), `retry_attempt` (0|1|2|3+)
@@ -645,4 +663,3 @@ Import this dashboard into Grafana to get:
 - [Library Observability Guide](library-observability.md) - For library developers
 - [examples/with_observability.rs](../examples/with_observability.rs) - Working example
 - [examples/grafana-dashboard.json](../examples/grafana-dashboard.json) - Production dashboard
-

docs/provider-implementation-guide.md

@@ -748,7 +748,7 @@ async fn fetch_work_item(
          WHERE id = (
              SELECT id FROM worker_queue
              WHERE visible_at <= ? AND (lock_token IS NULL OR locked_until <= ?)
-             ORDER BY id LIMIT 1
+             ORDER BY visible_at, id LIMIT 1
          )
          RETURNING work_item, attempt_count, instance_id",
         lock_token, locked_until, now, now
@@ -1393,7 +1393,7 @@ async fn fetch_work_item(
             SELECT id FROM worker_queue
             WHERE visible_at <= ?3 
               AND (lock_token IS NULL OR locked_until <= ?3)
-            ORDER BY id 
+            ORDER BY visible_at, id
             LIMIT 1
         )
         RETURNING id, work_item, attempt_count, instance_id, execution_id
@@ -1442,6 +1442,28 @@ async fn get_execution_state(&self, instance_id: &str, execution_id: Option<i64>
   - `attempts`: How many times this item has been fetched (for poison detection)
   - `state`: Execution state (Running or Terminated for cancellation)
 
+**Worker queue progress:** Repeatedly deferred activities must not overtake work
+that was already ready, even after their delay expires. Order eligible rows by
+`visible_at, id` (with a matching index), or use an equivalent requeue policy.
+Apply the same ordering to session-aware and tag-filtered fetches. Admission
+deferral restores an attempt only when `abandon_work_item(..., true)` commits;
+failed or interrupted abandons can still consume attempts.
+If no abandon commits, lock expiry can make the item eligible before its requested
+retry delay. Conversely, if the commit succeeds but its response is lost, the
+stored delay and restored attempt survive even though the runtime did not observe
+success. Neither fetch counts nor a durable admission/start-intent write can prove
+that a handler actually began executing across a process crash.
+
+The worker uses `ProviderAdmin::get_instance_info` for lightweight activity
+metadata when it describes the activity's execution. Providers without that
+capability, and activities from an earlier execution, fall back to
+`read_with_execution`, loading the full execution history on every attempt,
+including repeated admission deferrals. There is no descriptor cache.
+Both lookup paths record provider-operation latency and errors. Metadata must
+describe the requested execution, not a newer continue-as-new execution. If the
+execution-history read fails, the worker abandons the fetch without restoring
+its attempt and does not invoke admission or the handler.
+
 ### ack_work_item() — Complete Implementation
 
 Delete from worker queue and optionally enqueue completion: