Repository navigation
[BUG][regression] Global _index.json full rewrite is unsafe under concurrent tasks (real corruption under JetBrains multi-agent) #1231
Description
Summary
Task history persistence uses a global _index.json updated via
read → mutate entire entries array → atomic full-file rewrite (safeWriteJson).
That is safe for a single writer, not for concurrent agents.
We are already seeing production corruption with concurrent agents on the
JetBrains plugin (not observed on last Roo Code CE with serial usage).
This blocks trustworthy parallelizable tasks :
runtime pieces (TaskRegistry, TaskScheduler, TaskSemaphore, status machine)
increase concurrent writers; storage still assumes one writer.
Failure mode
- Task A reads index (version V).
- Task B reads index (version V).
- A writes V+A (atomic rename — OK in isolation).
- B writes V+B and drops A’s updates.
safeWriteJson prevents half-written JSON; it does not prevent lost updates.
(Real-life situation is probably more complex, as ui_message.json and history_item.json also get corrupted if one opens a task in the UI!)
Observed corruption
- Placeholder task names / wiped fields
size=0while task dirs have real UI/ACH data- Broken delegation graph (parent/child status, delegatedToId, childIds)
- Invalid transitions when one writer restores stale status
Environment
- IDE: PhpStorm 2026.2.1 / WebStorm 2026.2.1 (build WS-262.9437.145)
- Java: 25.0.3
- JCEF Support: Yes
- Zoo Code JetBrains plugin: 3.76.0 (5b0bcd0)
- OS: Windows 11 10.0 (amd64)
- Multiple agents: yes (One session in Phpstorm, another session in Webstorm, they do NOT both need to actually be active)
What changed vs Roo CE (hypothesis)
Zoo added richer lifecycle persistence and parallel-task infrastructure:
- TaskStatus transitions (active/delegated/completed/interrupted)
- More parent/child fields written on delegate/return/cancel
- TaskRegistry / scheduler / semaphore → more concurrent activity
Storage model remained a single global mutable JSON document.
Proposal (phased)
P0 – stop the bleeding
- Optimistic concurrency (occ): record file identity on read (size/mtime/inode);
refuse write if changed; retry read-patch-write for single-entry updates. - Process-wide mutex around index updates (same extension host).
- Prefer updating
tasks/<id>/history_item.jsonas source of truth; treat
_index.jsonas a rebuildable cache where possible.
Possible implementation: saveFile
P1 – reduce contention
- Workspace-scoped indexes (no single file for all projects).
- Narrow writes: patch one entry, not conceptual “rewrite the world” without merge.
P2 – durable design for #355
- Append-only event log (JSONL or line protocol) or SQLite (WAL) task catalog.
- Repository interface shared by GUI, CLI, and repair tools.
Evidence / tooling
- Repair CLI that rebuilds index from per-task files and detects corruption.
- Snapshot-checked writer prototype (mtime/ctime/size/inode) that turns
silent clobber intoConcurrent modification detected.
Ask
Treat concurrent history writes as a release blocker for parallel tasks,
not a rare edge case. JetBrains multi-agent already proves the race in the wild.
Activity
[-][BUG][regression] Concurrent writes corrupt global _index.json[/-][+][BUG][regression] Global _index.json full rewrite is unsafe under concurrent tasks (real corruption under JetBrains multi-agent)[/+]morgan-coded commentedon Aug 16, 2026
Claiming this.
I landed the in-process
withLockwork in #912; this is the same lost-update problem at process scope, and the harness above confirms it — the_index.jsonread-modify-write is guarded only bysafeWriteJson's per-file lockfile. I'd take the P0 shape building on #1210: a cross-process lock covering read/merge/write (or OCC with retry), the stale-snapshot case as a regression test in the deterministic-promise style, and index reconciliation fromhistory_item.jsonas a follow-up if you want it.Draft #926 is narrower and idle since July — happy to fold it in or supersede it, whichever you prefer.
martin-rueegg commentedon Aug 16, 2026
Thank you guys for looking into this. I am not sure if the race on writing on _index.json is the only issue and everything else I observed is just a consequence of this.
Or maybe the issue is broader, or - as @morgan-coded pointed out - the same issue on the
history_item.json(HI).I just want to describe a bit more on the symtoms:
I found out about the corruption by using zoo-code plugin in Jetbrains IDEs. I had migrated from the Roo Code CE plugin in two instances (Webstorm and Phpstorm). I hence had it running in both IDEs at the same time. Although working on different workspaces altogether. So - apart from
_index.json(IDX) - there was no overlap in tasks being updated.Once I opened the plugin in the second IDE i could see the task history as usual. I am not sure if I first opened a root task from the offered list, or if I already opened the subtask list and jumped straight to the subtask of the root process. However, once I opened the subtask, nothing was shown in the task history. Just a blank page. So I wanted to go back to the parent. Turned out that the subtask - after opening - was shown with
"Task #0 (Incomplete)"label once I got back to the history index. Opening a second subtask of the same parent produced the same issue. I think I end it up havingTask #x (No messages)Task #0 (Incomplete)So I stopped opening any other task as the history data was crucial to me. Then I found that in at least one subtask directory, the
ui_messages.json(UIM) was gone. In at least one other, it was set to[].I've been able to rebuild everything from the
api_conversation_history.json(ACH) using the zoo-code-history-repair tool I created (apart from the MCP Server communication which is only stored in UIM). However, I can't really explain why the"Task #0 (Incomplete)". Analysing the code, the only failure-path would be a reading error on the UIM file. Which I find weird, as it was just one single instance that was trying to open and display the task.In the wake of all this, the following values of HI got reset to
0:tokensIntokensOuttotalCostsizecacheWritescacheReadswhich obviously also ended up in the global IDX (including the truncated
taskfield)martin-rueegg commentedon Aug 16, 2026
I'm with you to first trying to stop this from happening with the current setup. I'm not sure if the approach I use in saveFile with the FileSnapshot would be a feasible approach.
But I think it is also important to think of ways to improve the way the logging is happening. In my opinion, there are several design issues:
fix(task-history): atomic per-task merge and drop shared index file (Z…