[kinyoklion]

Summary

Applications and editors rewrite data files in place: they truncate the file and then write the new content. A reload that reads an override file in between finds it empty. On this stack the override loader chose the parser by content, so the empty file did not start with {, was parsed as YAML, and loaded as a document with no entries: the override source briefly removed every override until the next reload. A rewrite with a 300 ms pause between truncating and writing made a client serve a -> launchdarkly-value -> b in watching mode on every run (8 of 8), and in polling mode whenever a poll landed in the pause (5 of 24 runs with varied timing).

The OVERRIDE spec's retry requirement exists to keep "a partially written or briefly invalid file from emptying the layer" (section 1.3). This change makes the file-based override source keep its overrides through such a rewrite. It is the Java counterpart of dotnet-core #394.

The later commits follow the js-core and .NET reviews: the second makes the delay exact with the retry armed for the time remaining, counts YAML with no document as empty, and reads the file's state through symbolic links; the third gives each held file its own bound and reads the clock after each file's read; the fourth starts a file's bound over only at a load that holds nothing back; the fifth includes the file whose read or parse fails in that, and ends a file's empty state when it is read with content, even content that fails to parse, or found missing; the sixth, after the general and adversarial reviews, reads every file before failing a load, merges only after every file is read, puts the paths in the skip-unchanged digest, and detects the text encoding. A last commit trims the public docs to what an application needs. The description below is the result of all of them.

The format comes from the extension, and an empty JSON file is invalid

The override loader now chooses the parser by the file name: .yaml and .yml, in any case, are YAML, and any other file, including one with no extension, is JSON. The OVERRIDE spec (requirement 1.3.3, sdk-specs #272) recommends the extension and lets an SDK whose file data source inspects the content keep doing so for override files; it says nothing about empty files. The override source is unreleased, so it now follows the extension, which lets the format decide whether an empty file is valid, like the rest of the content:

• An empty or white-space JSON file fails to parse with the file has no content, which is not a JSON document, and the last good overrides stay, as for any other content that cannot be parsed. To remove a file's overrides, write {} or delete the file.
• An empty YAML file is still a document with no entries.
• An empty JSON file, such as a placeholder created with touch, therefore fails the whole reload: the other files' changes are not applied until it holds a document such as {} or is deleted. An empty YAML placeholder is fine.
• A JSON document that is not an object, such as null or [], fails with the JSON document is not an object.

Before, a .json file with YAML content loaded as YAML, and a YAML file that happened to start with { loaded as JSON. Neither is released. The test harness names its YAML override file .yaml and its JSON files with no extension, so the contract tests are unaffected. Other compatibility cases with the content-based choice:

• A JSON file in UTF-16, or in UTF-8 with a byte order mark, did not start with { and loaded as YAML, which detects the encoding. The override source now detects the encoding from a byte order mark for both formats, with SnakeYAML's UnicodeReader, and reads UTF-8 otherwise, so those files still load, and JSON no longer depends on the platform's default encoding. The file data source, which shares the JSON parser, keeps reading JSON in the platform's default encoding.
• A JSON document indented with tabs, or with the escape \/, loaded as JSON and still does in a .json file. In a .yaml file SnakeYAML, which reads YAML 1.1, rejects it; the docs say so.

The file data source (FileData.dataSource(), FileData.initializer(), FileData.synchronizer()) is released and keeps choosing by content: it still calls selectForContent, and it never passes empty content to the JSON parser, because it parses a file as JSON only when it starts with {. The FileOverrides docs note the difference.

A file that is emptied is held back until it has stayed empty

A YAML file can legitimately be empty, so its parser cannot tell an emptied file from one in the middle of a rewrite. A file counts as empty when it holds only white space or a byte order mark, or when its parser reads it as no document: SnakeYAML returns null for a part of a file with only a comment or a --- marker, which a read can see while the file is being written. {} is a document and is applied right away. Content that is only white space is parsed only when the file is not held back, so an empty JSON file is reported only then.

The loader keeps, for each file:

• whether it contributed content to the overrides in effect, updated only when a load is applied or skipped as unchanged; the reloader calls the new Loader.accepted. The skip-unchanged digest covers each file's path, whether it exists, and its content: the format comes from the path, so the same bytes moved from x.yaml to x.json are different data and are applied. Moving a file between configured paths (mv a.yaml b.yaml) is therefore applied too, which moves the record to the new path, so a later in-place rewrite of it is held back;
• for a file whose content is in effect and that loads have found empty: since when it has been empty without being written (cleared when a load reads it with content, even content that fails to parse, or finds it missing, so that its next emptying starts a new delay even where a coarse modification time makes it look the same), its state then, the modification time and size read through any symbolic link (with the poller's FileDataPoller.observe, now package-private, whose Files.readAttributes follows links on every platform), compared only for equality, and when a load first held it back.

A load that finds a file empty parses it right away when the file's content is not in effect (a placeholder that never had content, or a file whose empty state was applied) or when retries are off (a zero retry delay gives no grace). Otherwise a new state restarts the file's delay. The file is held back while it has stayed empty for less than the retry delay (1 second), and for at most five retry delays from the first load that held it back. The bound is the file's own: one file reaching it does not cut short another file's rewrite. It starts over only at a load that holds nothing back, for every file that load read (with content, parsed empty, found missing, or failing to read or parse), whether the load is applied, skipped as unchanged, or fails. So loads that hold another file back keep a file's bound, which bounds files rewritten in turn, files kept empty and truncated again in staggered turns, and a file kept written while another file fails now and then. A file saved in place with a mistake and fixed later has a fresh bound when the fix is saved, a file read during a long failure of another file has a fresh bound when that failure ends, and a file that a tool rewrites in place now and then with the same content is held back again on a later rewrite, because the load skipped as unchanged in between held nothing back. The clock is read after each file's own read, so a slow read of another file does not shorten a file's delay.

A load reads and classifies every configured file, also after a read or parse failure, so files emptied together start their delays in that load and every file that is not held back is reached. It then fails with the first failure, before a hold; then it holds files back; and only then merges the entries, so a duplicate key or an entry that the data model rejects cannot keep a later file from being read. A holding load keeps the overrides in effect, including changes to the other files, because a reload applies all the files or none. It logs at debug level (File ... is empty, possibly while it is being rewritten; reading it again before applying it), reports no failure, and arms the retry for when the first held file's delay or bound is over, plus a tenth of the retry delay, so that a timer that fires a little early still finds the time over; the clock is read again for that. A failure still arms the normal retry delay. Elapsed times are differences of System.nanoTime() readings, which do not overflow; the loader tests start their clock just below Long.MAX_VALUE to check that.

So:

• A file that stays empty without being written is parsed by the retry, one retry delay after it was found empty: invalid JSON, reported once while the overrides stay, or a YAML document with no entries. A load within the delay, from a change signal or another file's change, does not apply it.
• A write and a new truncation that no load saw in between restart the delay. That covers two quick rewrites whose write and second truncation fall in one debounce window, a change to another file while one file is mid-rewrite, and a poll or retry that finds the second of two rewrites about a second apart.
• The bound limits a file that keeps being rewritten so that every load finds it empty, and files rewritten in turn so that every load finds one of them empty.
• A JSON file that stayed empty and failed to load is held back again when it is rewritten in place later, instead of reporting a second error, because the failing load held nothing back and so started its bound over.
• A placeholder file never delays the other files' changes. The override source always retries.

The reloader passes its retry delay (or zero) to Loader.load and handles a new EmptiedFileException, which names the held files and when to load again. A retry armed by a hold logs Reading flag data again after a file was found empty instead of the failure message. The reloader records the delay of each retry it is asked to arm, for tests.

Earlier versions of this change, and of dotnet-core #394 and js-core, used rules that reviews proved wrong: one keyed on whether the last read found content applied the empty file in the three rewrite sequences above; a per-file cap reset by reads with content, with a nine-tenths slack, held files rewritten in turn without end and let another load apply a gap shorter than the delay; a cap on the run of holding loads was restarted by a failing other file and cut short another file's rewrite, while one clock reading per load let a slow read of an earlier file shorten a later file's delay; a per-file bound that a file cleared itself when it had stayed empty for the delay held files emptied in staggered turns without end, while one kept through a long failure of another file applied a later rewrite empty; a reset that left out the file whose own read or parse failed applied the fix of a file saved in place with a mistake while it was empty; and a load that stopped at the first failure, or at a duplicate key while it merged, never read a later file, which kept a stale bound and had its next rewrite applied empty.

Accepted and documented in the reviews, not changed: with several files each kept written while empty and started 4.5 retry delays apart, the first application comes after about 4.5 retry delays per file, because the per-file bounds do not overlap; and a contrived failure of another file at every bound's end can hold the files back without end.

A rewrite is covered when the gap between truncating and writing is shorter than the retry delay. A writer that pauses longer gets the empty file reported (JSON) or applied (YAML), as before.

Documentation

The public docs say only what an application needs. The hold's delay, its bound, the other files waiting and the retries stay in the internal comments and in this description.

• FileOverrides.source(): a .yaml or .yml file is YAML and any other file is JSON, and the YAML parser does not accept every JSON document; a file is read as UTF-8, or in the encoding that a byte order mark names; files can be rewritten in place, and if the source finds a file empty while it is being rewritten, it keeps the overrides in effect until the new content is written; to remove a file's overrides, delete the file or write {}; an empty JSON file is not a valid document; replacing a YAML file in one operation avoids briefly applying a partly written file.
• Both FileOverrideSourceBuilder.filePaths methods state the extension rule and the encoding.
• Every public override API, including the builder methods, the poll interval constants and the change detection modes, says that flag overrides are currently experimental and subject to change.

Not changed: YAML cut off part way

A read that overlaps a write can see the first part of a YAML file. If the cut falls between two entries, or after flagValues:, the part is valid YAML with fewer entries; if it falls inside an unquoted value (flag: tr for flag: true), the part is valid YAML with a different value. Either is applied until the next reload. JSON cut off anywhere fails to parse. Telling a part from a whole file needs a content rule this change does not add, so the docs recommend replacing a YAML file in one operation.

Difference from the other SDKs

Go still detects the format from the content (internal/filedata/document.go), so an empty file of any name loads as no overrides there. Node chooses by extension case-sensitively (.yml, .yaml); this change, like .NET, ignores case.

Testing

• OverrideFileLoaderTest (a test clock that crosses the long overflow, on real files rewritten in place):
  • The extension chooses the parser (.yaml, .yml, .YAML, .Yml are YAML; .json, .txt, no extension and .yaml.bak are JSON).
  • An empty or white-space JSON file fails with the new message, also with no extension or another extension. A YAML file that is empty, white space, only a comment or only --- has no entries and no content; {} has content.
  • An emptied YAML file is held back at 0.95 of the retry delay and parsed at exactly the delay; an emptied JSON file is held back and then fails to parse.
  • A hold asks for its retry when the first delay ends plus a tenth (1.1, 0.7, and 0.5 with a second file emptied later), when the file's bound ends if that is sooner (0.5 at 4.6 delays after the first hold), and for the rest of the delay after a slow read of a later file (0.6).
  • White space or a byte order mark alone counts as empty, for YAML and for JSON (a read of only the byte order mark is held back, not reported); YAML with only a comment or --- is held back; {} is applied right away.
  • A file emptied again with the same modification time and size after a load read content starts a new delay, which is still running a little later (timestamps are coarse; the test sets the time back to the first emptying's).
  • Content that was read but not applied does not hold the file; a file with no content from the start (empty or only a comment) is never held, also after a new write; a file whose empty state is in effect is not held again; a file deleted and created empty is parsed.
  • A new write restarts the delay, also for a file configured through a symbolic link whose target is written.
  • The bound applies to one file kept written and to two files rewritten in turn; a load that holds nothing back starts it over, also when nothing is applied; applying data starts it over; a long-empty JSON file whose failing loads held nothing back is held back again when it is rewritten later.
  • Two files kept empty and truncated again in staggered turns are applied within the bound; a file held back during a failure of another file, then read with content, has a fresh bound when the failure ends; a file kept written while empty is applied after its bound while an earlier file fails; a load that holds a file back and then fails starts no bound over.
  • The fix of a file saved in place with a mistake is held back when it is saved (YAML and JSON); a file that fails to read starts its bound over; content that fails to parse, and a missing file, end the empty state, so a file emptied again with the earlier emptying's modification time and size starts a new delay.
  • A failing other file does not restart a file's bound; another file reaching its bound does not cut short a file's rewrite; a slow read of an earlier file does not shorten a later file's delay, and a slow read after the held file makes the retry due right away, after the margin (a reader that moves the clock).
  • Without retries an emptied file is parsed right away; files emptied together are held from the same load and parsed together; a parse failure of another file fails a load that holds a file; a failing load reports the first failure.
  • A file after a failing file starts its bound over while the other file fails, so the rewrite that comes with the fix is held back; a file after a duplicate key is read, so its bound starts over too; the fix of a file saved in place with a duplicate key is held back; a slow read of the emptied file itself does not shorten its delay.
  • A JSON document that is not an object fails with its own message; the encoding is detected from a byte order mark (UTF-8, UTF-16LE, UTF-16BE, both formats) and is UTF-8 otherwise, with a non-ASCII value; a UTF-16 file with only white space counts as empty; tab-indented JSON and \/ in a .yaml file are rejected; the content digest covers paths and presence.
• FileDataReloaderTest (scripted loader): a hold applies nothing, reports nothing, and is retried after the time the loader asks for, even when the retry delay is an hour; a failure after a hold arms the normal retry delay; the loader is given the retry delay, or zero without retries; the loader is told of applied and unchanged results only, not of failures, holds or results the consumer rejects; a hold does not end a recovery; a failure after a hold is retried and logged as a failure.
• FileDataReloaderEmptiedFileTest (reloader with the real loader on real files, rewritten in place with InPlaceWrites):
  • With a test clock and an hour-long retry delay, so that no retry fires and every load is a reloadNow or a change signal: the hold ends exactly at the retry delay; a change to another file waits while a file is held; a file rewritten in place with the same content (skipped as unchanged) is held back again six retry delays later; the requested retry is the delay plus a tenth after the first hold, the rest of the delay after a later hold, cut short by the bound (12 minutes plus a tenth), decided by the first-due of several files (40 minutes plus a tenth), the same when the hold comes from a change signal, and the retry delay after a failure.
  • With the real clock, waiting on load counts rather than fixed sleeps: an emptied YAML file is held back and the rewritten content is applied with no empty application in between; a YAML file that stays empty is applied by the retry; an emptied JSON file is not reported by the load that finds it empty, and the retry then reports it while the overrides stay; a further change signal within the delay does not apply the empty file.
  • Two rewrites whose write and second truncation come before any load reads the first new content (F1), a change to another file while a file is empty (F2), and a retry that finds a second truncation (F3) never apply the empty file.
  • Two files emptied together are applied by the first retry; a file moved to another configured path is held back when it is then rewritten in place; the same bytes moved from a .yaml to a .json file are applied, not skipped as unchanged.
  • An empty placeholder YAML file does not delay a change to another file; a file deleted and created empty is applied right away; a retry left over from a failure that finds the file emptied holds it; without retries an emptied file is applied right away.
• FileOverrideSourceTest: at startup an empty YAML file has no entries and an empty JSON file (also with no extension) fails to load.
• FileOverrideSourceInPlaceRewriteTest, end to end with real in-place rewrites (truncate, pause, write through the same stream):
  • The override source never applies a snapshot without the override, for JSON and YAML, watching (300 ms pause) and polling (1.2 s pause, longer than the 1 s poll interval, so that a poll always finds the file empty).
  • Watching, two quick rewrites (150 ms and 300 ms pauses, back to back) never apply a snapshot without the override.
  • The source with the defaults (polling every second, 1 s retry), through the public builder, keeps the override through two rewrites about a second apart.
  • A client with LaunchDarkly data and the override source serves a -> b and never the LaunchDarkly value, for JSON and YAML.
  • Where a test must finish a rewrite before the retry, it uses a 1.5 s retry delay (2.5 s for polling) through the package-private constructor of FileOverrideSourceImpl, so it survives a loaded CI machine.
• On the base branch's behavior (format by content, no JSON rule, no hold, with this PR's seams kept so the tests compile), 64 of the new tests fail, including 7 of the 8 end-to-end tests.
• Mutation check, with a clean build per mutant: each of 57 mutants fails at least one test. They are: format by content; empty JSON as an empty document; the JSON rule removed; no hold; a hold without applied content; the content check decoding without detecting the encoding; accepted recording presence instead of content; not accepted when unchanged or when applied; accepted before apply; a hold reported as a failure; a hold arming no retry; a negative retry delay passed to the loader; the retry log not chosen; its flag never reset; a case-sensitive extension; .yml not recognized; stopping at the first held file; the reloader ignoring the hold's retry time; YAML with no document counted as content; an empty mapping counted as no document; the state read without following links; an overflowing deadline comparison (now >= since + delay); the clock read once per load; a stale clock for the hold's retry; a read with content dropping the bound or keeping the delay; the bound started over by loads that hold a file back, never started over, started over by a file that stayed empty, started over after a hold and a failure, or not for a file found missing, a file that fails to read or parse, or a file parsed empty; a load stopping at the first parse failure; the last failure reported instead of the first; a hold reported before a failure; the digest without paths; JSON decoded in the platform's encoding; a JSON document that is not an object accepted; a missing file or content that fails to parse not ending the empty state; ending the empty state dropping the bound; the bound restarted by every hold; no bound; an exclusive bound or delay check; the nine-tenths slack; a state change not restarting the delay; the hold's retry at the whole delay, without the margin, not clamped at zero, ignoring the bound, or decided by the last file; and either arming path (reloadNow, change signal) ignoring the requested time. A guard that skipped holds when retries are off, and clearing the held-file records when data is applied, became redundant (a zero delay gives no grace; every file of an applied load was read by a load that held nothing back), so both were removed rather than kept as equivalent mutants.
• The five new and changed test classes (150 tests) ran 12 times in a row with no failure, while a mutation run loaded the box. An earlier loop had one failure in FileOverrideSourceTest.closeIsIdempotentAndStopsThePoller, a test of the stack (feat: Add the file-based override source #224) that this PR does not change: its baseline count of poller threads can include a poller of an earlier test that exits afterwards.
• ./gradlew build javadoc (Java 17) passes: 2262 tests, 0 failures, checkstyle and javadoc clean. CI stays red until the common and internal library PRs (feat: Add the overrideAffected indicator to EvaluationReason #218, feat: Count override-affected evaluations separately and send no individual events for them #219) are released, as for the rest of the stack.
• The harness flag overrides suite (sdk-test-harness #515 branch) passes against the contract test service built from this branch: 38 tests.
• Unrelated flakes seen on the loaded box, in code this PR does not touch: PollingProcessorTest.oneSuccessfulPollDoesNotLeaveExtendedCadence (a timing assertion), FileSynchronizerReloadBehaviorTest.everyFailedReloadIsReportedAndLoggedAsThePlainDescription (the file data source read its file mid-rewrite and applied it as an empty YAML document, the same defect on main), and FileOverrideSourceTest.closeIsIdempotentAndStopsThePoller (a poller thread of an earlier test counted in the baseline).

This PR is based on #225 and is not part of the stack's history; it can be folded into #220 and #224 or merged after them.

SDK-3246

_{Opened from termag}

---

Note

Overview
Fixes a gap where in-place file rewrites (truncate-then-write) could briefly drop all flag overrides because an empty read was parsed as YAML with no entries.

The file-based override source now defers applying a file that was previously in effect and is found empty: OverrideFileLoader throws EmptiedFileException, FileDataReloader keeps the last good snapshot, schedules a re-read (not onError), and only applies once the file has stayed empty for the retry delay or hits a per-file cap (five retry delays). Multi-file reloads still read every path before merge/fail/hold, and the content hash includes paths so format moves are not skipped as unchanged.

Parsing and format rules for overrides (not the released file data source): choose YAML vs JSON by .yaml/.yml extension (case-insensitive); detect text encoding from a BOM (UTF-8 otherwise); treat empty/whitespace JSON as invalid; treat empty YAML as a document with no entries via FlagFileRep.NO_DOCUMENT. Public FileOverrides / builder docs describe rewrite behavior, {} vs empty JSON, and that overrides remain experimental.

Large test additions cover hold timing, concurrent files, watching/polling, and end-to-end client behavior using InPlaceWrites.

^{Reviewed by Cursor Bugbot for commit 0c37016. Bugbot is set up for automated code reviews on this repo. Configure here.}

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

Stale Bugbot comment from a previous run.

[kinyoklion]

bugbot review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0c37016. Configure here.}

[kinyoklion]

The changes of this PR moved into the stack:

• feat: Add reload infrastructure for file-based flag overrides #220 (reload infrastructure): 3b81e47, fix: Hold back an override file that is emptied during an in-place rewrite (FileDataReloader, FileDataPoller, OverrideFileLoader, FileDataSourceParsing, their tests, InPlaceWrites, FileDataReloaderEmptiedFileTest).
• feat: Add the file-based override source #224 (file-based override source): 43d6c3d, fix: Keep the overrides in effect while an override file is rewritten in place (FileOverrides, FileOverrideSourceBuilder, FileOverrideSourceImpl, FileOverrideSourceTest, FileOverrideSourceInPlaceRewriteTest).

The branches above were rebased with their commits unchanged. The new #225 head (a2dbdfc) has the same tree as this PR's head (0c37016). Closing this PR; its branch stays in place.