[abelonogov-ld]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Related issues

Tier 3 of the Event Durability spec, "Tier 3 — persistence". Stacked on #403 (tier 2) and targets that branch, so this diff shows only tier 3. It will be retargeted once the branches below it merge. The iOS counterpart is launchdarkly/ios-client-sdk#538, and both write the same on-disk format.

Opened as a draft: a multi-threading and readability review turned up items still being worked through (see Known open items).

Describe the solution you've provided

Recorded events go to an append-only log on disk rather than only an in-memory buffer, so an application that dies shortly after recording an event still reports it: the next run delivers what the previous one left behind. That covers what tier 2 cannot: SIGKILL, ANR kills, native crashes, and the system reclaiming a backgrounded process.

• EventPersistence (new public enum) and EventProcessorBuilder.eventPersistence(...):
  • DISABLED (default): events stay in memory and the SDK does not touch the filesystem for events.
  • DEFERRED: writes happen on the SDK's commit thread, a moment after recording.
  • IMMEDIATE: track and identify write on the caller's thread before returning, so there is no window.
• EventStore: an append-only log under the no-backup files directory, one directory per mobile key. It handles framing, capacity, torn-tail recovery of a log interrupted by a crash, and a fallback to memory when a write fails (for example, a full disk), so a disk failure never takes the application down. Each process appends to its own open-<process> log; closed ready-<payloadId> batches are shared, so a batch left by a process that never runs again is still delivered. The byte format lives in EventLogFormat and the log this process appends to in DiskEventLog, mirroring the iOS split.
• Recording and commits (DirectEventProcessor): an evaluation appends to a pending run and updates a counter, with no encoding on the caller's thread. A commit encodes the run and its summaries together and stages them in the store. track and identify are commit points. Evaluations commit once a threshold accumulates, on flush, and before delivery.
• Delivery reads batches oldest first. A batch keeps its payload ID across retries and process restarts, so LaunchDarkly discards a repeated delivery instead of double counting. Delivery stops at the first retryable failure, schedules a single retry instead of sleeping on the events thread, and removes a batch only once it has been accepted or permanently refused. A batch whose read fails transiently is kept for a later attempt.
• Loss reporting: flushAsync() answers false when events the SDK accepted were lost for good: an event or summary that could not be serialized, or that was too large to store. Each loss is reported to exactly one caller, the one whose delivery closed its batch after the loss, as on iOS. Events dropped for capacity are counted in diagnostics, not reported as losses.
• close() commits first, then delivers with what is left of one two-second budget. Neither step waits on the caller's thread past the budget, so a stalled disk write cannot become an ANR. Work that outlasts the budget finishes in the background.
• AnalyticsEventSender posts a pre-assembled body under an explicit payload ID.
• DEFAULT_CAPACITY goes from 100 to 1000, matching iOS, since capacity now also bounds what is held on disk.

Describe alternatives you've considered

SQLite. Measured as a store and rejected: it costs more per event than an append to a log the SDK already frames, for durability properties the log gets from write alone. See sqlite-event-persistence-findings.md in the spec folder.

Fsync on every commit. What durability is defending against is the process dying, not the device losing power. Bytes handed to the kernel survive the former, and an fsync per event would dominate the cost of track.

One shared log for all processes. That is simpler, but one process's startup recovery could rename a log another live process is still appending to. A log per process avoids that without file locks.

Additional context

Tests. 879 unit tests pass locally (testDebugUnitTest). New coverage includes EventStoreMultiProcessTest (several stores on one directory, recovery, transient read failures, events still counted against capacity while being written), DirectEventProcessorTest (commit points, retry, capacity across memory and disk, persistence failing partway through, oversized frames reported as losses, a loss reported to the right flush across a retry, close() within budget on a stalled disk write), OutboundEventBufferSerializationTest, and SameContextPerContextEventSummarizerTest.

Benchmarks. EventPersistenceBenchmark (JVM and instrumented) measures recording cost per mode. It runs only when LD_EVENT_BENCH=1, so ordinary test runs are unaffected.

Test app. Now configures EventPersistence.IMMEDIATE, so the events recorded by Eval+Kill now, which tier 2 loses, are recovered and delivered on the next launch. It also gains Over-Refresh Eval, a main-thread stress loop that compares evaluation cost while filling capacity and after it is full.

Known open items (from the threading review, to be resolved before this leaves draft):

• flushAsync(), and so flushAndWait, does not commit on the caller's thread as flush() does, so a flushAndWait that times out can leave events only in memory, even at IMMEDIATE.
• Batches left by a previous run are not counted against capacity until the first listing, which an offline start postpones.
• The store and the processor both schedule commits, so a threshold commit is followed by a redundant no-op pass.

Version. EventPersistence and EventProcessorBuilder.eventPersistence have no @since tag yet; one should be added once the release version is known.

---

Note

Overview
Tier 3 event durability replaces the java-sdk-internal DefaultEventProcessor wrapper with an Android DirectEventProcessor that commits framed events to an on-disk EventStore (iOS-compatible EventLogFormat, per-process append logs, ready-* batches, BatchLocks for multi-process send/expiry). Apps opt in via public EventPersistence (DISABLED / DEFERRED / IMMEDIATE) on EventProcessorBuilder, with durable data under getNoBackupFilesDir().

Delivery is reworked around AnalyticsEventSender (stable X-LaunchDarkly-Payload-ID), DeliveryRetryState, paced/launch-gated sends, and flushAsync() / flushAndWait() semantics—including parallel multi-environment flush and LDCrashHandler flushing before delegating crashes. close() prioritizes commit then a bounded final delivery; losses and payload refusals are tracked so flush callers can get false when accepted events were dropped.

Supporting changes: DiagnosticReporter split out, launchdarkly-java-sdk-internal bumped to 1.12.2, example LDClient.init(..., waitSeconds) instead of Future.get, optional EventPersistenceBenchmark when LD_EVENT_BENCH=1, dexcount removed from the example app, and broad new JVM/android tests (including BatchLocksAndroidTest).

^{Reviewed by Cursor Bugbot for commit db668b0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[abelonogov-ld]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[tanderson-ld]

Approved, will approve officially on Monday.

[cursor]

Stale Bugbot comment from a previous run.

[tanderson-ld]

This is way too much complexity for the probability of this large context summary problem happening. We would rather just allow this to exceed the limit (apparently the customer really wants to use a large context) OR we drop such data and log a warning. We don't have precedent for dropping a large context, so we'd have to think a bit on it.

[abelonogov-ld]

Removed. Chances of this happening are pretty low. You need many events of large individually. It is fine to drop packet in 1 in million cases

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db668b0. Configure here.}

[cursor]

Track can miss close delivery

Medium Severity

At IMMEDIATE, writeAlone admits a track or identify under recordLock but does not put it in pending and writes later without commitLock. close() can set closed, commit an empty run, finish the last delivery, and release the store before that write. The accepted event then lands in a new open log after closeBatch, so this session never sends it. If the write happens after store.close(), it can also reopen a log on an abandoned store.

Additional Locations (1)

• launchdarkly-android-client-sdk/src/main/java/com/launchdarkly/sdk/android/EventStore.java#L538-L576

 

^{Reviewed by Cursor Bugbot for commit db668b0. Configure here.}

[abelonogov-ld]

Fixed in #419