[Thom Chiovoloni (thomcc-work)]

Summary

Fixes #41. Taking over for #42.

The worker dispatcher previously abandoned a work item on every acknowledgement error, including retryable provider failures. Abandonment discards the completed activity's result and redelivers the item, potentially running the handler again. That is permitted by at-least-once execution, but is avoidable for transient acknowledgement failures.

This change retries acknowledgements in place for activity success, application errors, and cancellation. There are at most five total attempts, with waits of 100, 200, 400, and 800 milliseconds between attempts. The waits total 1.5 seconds of cumulative backoff, excluding provider-call duration; this is not a wall-clock deadline. Permanent errors and exhausted retries use the existing abandonment path.

The rebased worker retains lock renewal while a normal completion's acknowledgement is pending. Activities still require at-least-once-safe behavior: crashes, lost locks, and exhausted retries can cause re-execution. Poison-message handling, provider APIs, persisted schemas, and SQLite pool configuration are unchanged.

The tests now check exact acknowledgement/abandonment sequences, lock tokens, and completion payloads:

• Success on the fifth and final attempt without re-executing the handler.
• Five failed attempts, then abandonment and successful redelivery.
• A permanent error followed immediately by abandonment, not an in-place retry.
• Retried application-error completions without re-executing the handler.
• Retried cancellation acknowledgements with no completion payload.
• An acknowledgement that commits but reports a retryable error: the retry encounters the missing token, stops, and does not duplicate the activity or its completion.

These fault-injection tests use file-backed SQLite so shared-cache deadlocks do not interfere with their exact call-count assertions. The existing in-memory, multi-worker session test remains a separate regression check.

Checklist

• Tests added/updated
• Regular tests pass locally (cargo nt)
• Doc tests pass locally (cargo test --doc --all-features)
• Docs updated
  • Existing docs updated where behavior changed
  • New doc added under docs/ if introducing a new area
  • Linked from docs/README.md

Links / Design notes

SQLite diagnosis

The original issue and PR description attributed the shared-cache failure to SQLx not using SQLite unlock notifications. That explanation is incorrect for SQLx 0.8.6, used in this checkout: its statement execution loop already calls sqlite3_unlock_notify, and the dependency enables unlock-notification support.

SQLite emits the reported database is deadlocked error when unlock-notification registration detects a wait cycle. Waiting longer cannot resolve that cycle. SQLite documents rollback and later retry of the transaction as the recovery mechanism. The failed acknowledgement transaction is rolled back,
and the next provider call opens a fresh transaction.

This supports retrying the complete acknowledgement operation rather than rerunning the activity. The same policy also applies to other providers' retryable failures. A connection failure may leave a commit outcome unknown; atomic deletion plus lock-token validation prevents a repeated acknowledgement from enqueueing the completion again if the first call already committed.

Historical measurements

With the strict, exactly-once assertion retained as a measurement probe, Pino reported 3 failures in 14 runs before the runtime fix and 0 in 20 after it. The latter runs recorded 19 acknowledgement retries and 741 SQLITE_LOCKED events. Those measurements were not repeated during the takeover; the current validation uses deterministic fault injection and the regression tests listed above.

[Copilot]

🟢 Approval recommended

The implementation matches the documented retry policy and has comprehensive deterministic regression coverage.

0 open findings

What changed in this PR

Adds bounded retries for transient activity acknowledgement failures, reducing unnecessary activity re-execution while preserving at-least-once semantics.

Changes:

• Retries acknowledgements up to five times with exponential backoff.
• Adds fault-injection coverage for success, failure, cancellation, exhaustion, and uncertain commits.
• Documents acknowledgement retry and atomicity requirements.

File	Description
src/​runtime/​dispatchers/​worker.rs	Implements acknowledgement retries.
tests/​common/​fault_injection.rs	Adds retryable failure injection and call tracking.
tests/​worker_reliability_test.rs	Adds acknowledgement retry scenarios.
tests/​session_e2e_tests.rs	Aligns assertion with at-least-once execution.
docs/​provider-implementation-guide.md	Documents retry and commit semantics.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.