worker_reliability_test: crash tests fail at random and never test a crash before the completion is saved #69

Summary

Two tests in tests/worker_reliability_test.rs have the same three problems:

• activity_reliability_after_crash_before_completion_enqueue
• multiple_activities_reliability_after_crash

1. They fail at random.
2. They never test the case their names describe.
3. They set an environment variable that nothing reads.

The first test failed once in a local ./run-tests.sh run while preparing the 0.1.32 release (#68). The bug is in the tests. The runtime behaves correctly.

1. They fail at random

tests/worker_reliability_test.rs:89: Activity should not have completed yet

What the test does:

1. Start the orchestration
2. Poll history every 5 ms until ActivityScheduled appears
3. Assert ActivityCompleted is not in history yet   ← nothing guarantees this
4. Shut down the runtime (the "crash"), start a new one, wait for completion

The activity returns at once. The runtime picks up the activity, and then its completion, on its next dispatcher polls (by default every 100 ms, with 2 orchestration slots and 2 worker slots). When both pickups happen right away, ActivityCompleted is saved before the test's first poll sees ActivityScheduled. In the failed run, the orchestration turn that scheduled the activity and the turn that received its result were 4.7 ms apart.

A scratch copy of the test's first phase shows the result depends only on how fast the runtime picks up work:

multiple_activities_reliability_after_crash makes the same kind of assertion ("No activities should have completed yet") and has the same race.

2. They never test a crash before the completion is saved

The "crash" is rt1.shutdown(None). That is a graceful shutdown: it sets a flag and waits 1 s. Worker slots stop fetching new work, but an activity that is already running finishes and saves its result. So at the "crash", the activity has either:

• not been picked up yet, so phase 2 runs it for the first time, or
• already finished and saved its result, so phase 2 only processes that result.

The case in the first test's name, "the activity ran but its completion was never saved", never happens.

3. They set an environment variable that nothing reads

Both tests set DUROXIDE_SQLITE_LOCK_TIMEOUT_MS=1000 so that redelivery after the crash is fast. Nothing in src/ has read that variable since a261621 (October 2025). The real worker lock timeout is the 30 s default (RuntimeOptions::worker_lock_timeout).

Proposed fix

Hold the activity in phase 1, so the crash really happens while it runs:

Phase 1 (first runtime, worker_lock_timeout = 2 s):
1. Activity: record "started", then never return
2. Test waits for "started" instead of polling history
3. Assert: ActivityScheduled is in history, ActivityCompleted is not
   → always true now: the activity is still running
4. Shut down the runtime (the "crash")
   → the lock-renewal task stops at shutdown; the activity never returns

Phase 2 (second runtime, same store):
5. Activity: record "started", return at once
6. The phase-1 lock expires within a few seconds → the work item is delivered again
7. Assert: the orchestration completes, and the activity started twice

• Set the lock timeout through RuntimeOptions (worker_lock_timeout and worker_lock_renewal_buffer), and remove the environment variable.
• multiple_activities_reliability_after_crash: same change, with worker_concurrency: 3 so all three activities start in phase 1. Each activity should start twice.
• Don't use FailingProvider to fail every ack in phase 1. Each failed ack counts as an attempt, so with the default max_attempts of 10 the activity becomes a poison message within about a second.

Relevant code

• tests/worker_reliability_test.rs: the two tests
• tests/common/mod.rs: wait_for_history, which polls every 5 ms
• src/runtime/dispatchers/worker.rs: the worker loop, run_activity_with_cancellation, and the lock-renewal task, which stops at shutdown
• src/runtime/mod.rs: Runtime::shutdown and the RuntimeOptions defaults