Repository navigation
Retry an unconditional append after a concurrent change on every provider (one-way door: the Common API) - #74
Merged
Merged
Conversation
… a Physical folder delete no link can steer The two follow-ups 2027.1.0 left open, as a patch: an append with no condition retries after a concurrent change on every provider, as Azure Blob does, and Physical deletes a folder through handles, so no link swapped in can steer it outside the root. Two CI jobs run Physical's tests on macOS and on Linux on Arm first. Reviewed in six rounds and approved by the maintainer. Refs #65, #73, #70
…ider An append with no condition ended one way on Azure Blob, which read the file again and appended after another writer's change or create, and another on every other provider, which threw at the first conflict. AppendByRewriteAsync and Physical now do what Azure does: up to five attempts, then PreconditionFailedException with the last conflict inside. A caller's IfMatch or IfNotExists still fails at once. The caller's stream is read once: the rewrite keeps its bytes in the latest attempt's temp file, and Physical copies them to a hidden temp file beside the target, after the refusals that need none, and retries only on its own signals of a change. Refs #65
On Physical, where links are refused, a new name is taken with an empty file that the winner's content is renamed onto afterwards. An append that lost the name and read past that empty file appended to it, and the winner's rename then replaced the append, which had reported success. A lost create is now read past only once the file has content; while it is empty the append throws FileAlreadyExistsException, as before. Also: SharePoint's remarks on appends, what a caller's IfMatch gets for a file deleted before its read, and two comments. Refs #65
…here it can't Windows takes a new name with its content already in it, so an append that lost the name has no empty file to fear there: it reads past it as the other providers do. On Linux and macOS a lost create is read past unless the file there is still empty (a move's destination, a file system without hard links, or a file another writer left empty), and a name gone by then is read past again. The remarks, README, guide, errors.md, the exception's docs and the changelog say so. Refs #65
…ne reason A test now has another writer leave an empty file: where names are taken without replacing (Windows) the append reads past it, elsewhere it throws FileAlreadyExistsException. The docs give the reason that holds for all three cases (the provider can't tell an empty file from one a rename is about to fill), the interface's exception doc says what the others say, and the remarks' paragraph is reflowed. Refs #65
serionist
marked this pull request as ready for review
October 13, 2026 09:36
serionist
added a commit
that referenced
this pull request
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Step 1 of the 2027.1.1 plan. An append with no condition that met another writer's change or create ended one way on Azure Blob (it read the file again and appended after it, up to five attempts) and another everywhere else (it threw at the first conflict:
PreconditionFailedException, orFileAlreadyExistsExceptionfor a file created meanwhile). Now it ends the Azure way on every provider (#65; the maintainer chose this direction, D1).AppendByRewriteAsync(Common; S3, MinIO, Google Cloud Storage, Box, SharePoint, the in-memory provider and any third-party provider that calls it): a loop of at most five attempts. After a concurrent change or create, or a file deleted since its info was read (its read throwsFileNotFoundException, or whatIsNotFoundrecognises), it reads the file again and appends after what is there now; thenPreconditionFailedException, the base type, with the last conflict inside. A caller's ownIfMatchorIfNotExistsstill fails at once. The caller's stream is read once: a retry copies the caller's bytes from the previous attempt's temp file, so nothing costs more until a conflict happens. A failure of the caller's stream ends the append, whatever its type.IsReplaceableand the new file's mode included). The caller's stream is copied once, after the refusals that need none and before the file is opened, to a hidden.wfp-<guid>.tmpbeside the file: created0600on Linux and macOS, deleted when the append ends, hidden from listings, and swept byDeleteTempFilesAsyncif a crash leaves it. It retries only on its own signals (the version check before the rename, a name taken meanwhile, a file deleted since the look), which it records as it throws them. A caller's stream that throwsPreconditionFailedExceptionis not mistaken for one.UnauthorizedAccessException, since Windows won't rename over a file another append is reading. The guide says both.Docs:
AppendByRewriteAsync's andUpdateFileAsync's XML docs,FileAlreadyExistsException's, Physical's class remarks;reading-and-writing.md(Append),providers.md,errors.md,writing-a-provider.md,migration-2027.md(the racing-append rows); the S3, MinIO, Google Cloud Storage, Box, SharePoint and Physical READMEs; a2027.1.1section inCHANGELOG.md.Plan
docs/plans/2026-10-13-2027-1-1.md(approved by @serionist, reviewed in six rounds), committed first on this branch. This PR is its step 1; its progress log records where the code departs from the plan's wording.Tested by
Write_AppendRaceToAMissingFileKeepsBothnow expects both racing appends to succeed, the file holding both parts in either order;Write_ConcurrentAppendIsNeverLostexpects the append to succeed after the staged write, astheirs+mine.BaseClassTests, on a new scripted store): a lost create appends after the other writer's file; a change meanwhile is read past with the caller's stream read once; five conflicts in a row throw exactlyPreconditionFailedExceptionwith the last conflict (a lost create) inside; a file deleted before its read is created, for the store's own not-found and forFileNotFoundException; a conflict, then a deleted file, writes the caller's bytes once; a stream that throwsPreconditionFailedExceptionends the append with no write; the caller'sIfMatchandIfNotExistsfail at the first conflict.TempCreatedand expect the append after the other content (over the three ways new names are taken); five changes in a row; the caller's version, at the first change and between the look and the open; a failing stream; a refused append leaves no folder; a file deleted as it is looked at; after a lost create, the file found is checked and its mode taken; the spool is beside the file, named as the sweep takes it, hidden and gone afterwards; on Linux and macOS it is0600.Append_ToAFileWrittenMeanwhile_KeepsTheWriteAndAppendsAfterIt) and SharePoint (Append_RacedAfterItsCheck_IsRefusedByTheIfMatchItSends_ThenAppendsAfterTheRace) expect the retry where they expected the throw.Live, with Docker required (Azurite, S3Mock, MinIO, fake-gcs-server), on Windows: 2,789 tests, 0 failed; Physical again after its last four tests, 285, 0 failed. Physical and Common on Linux in a container, as a user who isn't root: Physical 288 (275 passed, 13 inconclusive for Windows-only cases), Common 340, 0 failed. Coverage, from CI's Linux gate: Physical 98.81% lines and 96.39% branches (floors 98 and 93), Common 97.31% and 94.96% (96 and 92); every assembly above its floor.Review
review-code, standard tier (Opus), four rounds: clean on e5d94cf, plus three wording nits from round 4, applied in c441a34.
UpdateFileAsync's docs, the changelog). G2: a name gone by the time the guard looks is read past again. G3: the live fixture says the race test can fail by design without hard links.If-Match, Box's internal retry, temp-file ownership on every path, signals matched by identity, the refactor changing overwrites or copies.One-way doors
The Common API:
AppendByRewriteAsync, a protected helper ofWriteableFileProvider<TOptions>, keeps its signature and its promise (a concurrent write is never lost) and now retries where it threw at the first conflict. The plan reads this as a fix toward one meaning across providers, shipped as a patch. The maintainers agreed on #65: #65 (comment)Closes #65