Tracking: lock loss, replay and commit edge cases (core, duroxide-pg and the SDKs) #55

Why this list exists

A process stall that lasted longer than the orchestration lock (5 s by default) showed a bug in the Node.js and Python SDKs. Two replays of one instance were alive in one process, and they shared one context. The change for that bug is in microsoft/duroxide-node#17 and microsoft/duroxide-python#17.

While I checked the code around it, I found the items below. Each item has its own issue with the code references, the way it fails, and a suggested fix.

• Run means: reproduced by a run.
• Read means: found by reading the code, not reproduced.

Core (this repo)

• A failed activity lock renewal can lose the activity result; the orchestration waits forever #56 A failed activity lock renewal can lose the activity result. Run
• prune_kv_values_updated_before emits actions in HashMap order; replay fails with kv clear mismatch #57 prune_kv_values_updated_before emits its actions in HashMap order. Run
• A failed commit becomes a permanent OrchestrationFailed without parent notification; the fallback abandon never runs #58 A failed commit becomes a permanent OrchestrationFailed; the fallback abandon never runs. Read
• A turn that lost its orchestration lock keeps running, and the same runtime can start a second replay of the instance #59 A turn that lost its lock keeps running, and a second replay can start in the same runtime. Run
• Orchestration completion and failure are logged and counted before the commit #60 Completion and failure are logged and counted before the commit. Read
• One failed session lock renewal loses the session: no retry, no log, no signal to running activities #61 One failed session lock renewal loses the session. Read
• Runtime::shutdown cannot stop the child tasks its dispatchers spawn #44 (fix in Make Runtime::shutdown actually stop the tasks its dispatchers spawn #45) Shutdown stops the lock renewal while activities still run, so an activity can run twice. Already tracked.

PostgreSQL provider

• ack_orchestration_item validates the lock without a row lock; a concurrent fetch can leave consumed messages in the queue duroxide-pg#29 ack_orchestration_item validates the lock without a row lock. Read
• Most SQLSTATEs are classified as permanent, including statement timeout and failover errors duroxide-pg#30 Most SQLSTATEs are classified as permanent. Read
• A work item or history over ~256 MB can never be fetched: the fetch builds one jsonb value, and ack has no size guard duroxide-pg#31 A work item or history over ~256 MB can never be fetched; no size guard at ack. Seen in a deployment

Node.js SDK

• Fix replay cross-talk in the orchestration context lookup; add orchestrator lock options duroxide-node#17 (PR, merged) Replays of one instance shared a context, and isCancelled() went back to false. Run
• Orchestration functions live in one process-wide map; two Runtime objects that register the same name share one function duroxide-node#18 Orchestration functions live in one process-wide map. Run
• A first yield that is not a task leaks the generator duroxide-node#19 A first yield that is not a task leaks the generator. Read

Python SDK

• Fix replay cross-talk in the orchestration context lookup; add orchestrator lock options duroxide-python#17 (PR, merged) Replays of one instance shared a context. Run
• Orchestration functions live in one process-wide dict; two Runtime objects that register the same name share one function duroxide-python#18 Orchestration functions live in one process-wide dict. Run
• A first yield that is not a task leaks the generator duroxide-python#19 A first yield that is not a task leaks the generator. Run
• finally blocks in an orchestration run at the end of every turn duroxide-python#20 finally blocks run at the end of every turn. Run
• The GIL is taken on a tokio worker thread outside block_in_place duroxide-python#21 The GIL is taken outside block_in_place. Read

.NET SDK

• microsoft/duroxide-net#15 (PR, merged) Replays of one instance shared a context. Run
• microsoft/duroxide-net#16 Read
• microsoft/duroxide-net#17 Read
• microsoft/duroxide-net#18 Read
• microsoft/duroxide-net#19 Read

Java SDK

• microsoft/duroxide-java#8 (PR, merged) Replays of one instance shared a context. Run
• microsoft/duroxide-java#9 Read
• microsoft/duroxide-java#10 Read
• microsoft/duroxide-java#11 Read
• microsoft/duroxide-java#12 Read

The .NET and Java repositories are not public. Their links work for people who have access.

Suggested order

1. A failed activity lock renewal can lose the activity result; the orchestration waits forever #56 and prune_kv_values_updated_before emits actions in HashMap order; replay fails with kv clear mismatch #57. Each one ends an orchestration, or leaves it waiting forever, although the orchestration code is correct.
2. A turn that lost its orchestration lock keeps running, and the same runtime can start a second replay of the instance #59. It removes the condition that the SDK bug needed.
3. A failed commit becomes a permanent OrchestrationFailed without parent notification; the fallback abandon never runs #58 together with Most SQLSTATEs are classified as permanent, including statement timeout and failover errors duroxide-pg#30. Together they turn a database error that goes away into a failed instance.