Skip to content

fix: order SQLite apps deterministically when timestamps tie - #991

Open
MIRS571 wants to merge 1 commit into
apache:mainfrom
MIRS571:codex/fix-sqlite-app-order
Open

MIRS571 wants to merge 1 commit into
apache:mainfrom
MIRS571:codex/fix-sqlite-app-order

Conversation

@MIRS571

@MIRS571 MIRS571 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

SQLite's default CURRENT_TIMESTAMP has second-level precision. After #942, list_app_ids() lists each app once and orders by its latest timestamp, but apps saved within the same second still have unspecified ordering. A caller such as the YouTube-to-social-media notebook, which resumes list_app_ids("")[0], can therefore pick a different app from load("", None), whose timestamp ties were fixed in #920.

Changes

  • Make synchronous and asynchronous SQLite list_app_ids() deterministic: keep each app's maximum created_at as the primary key, then use the greatest rowid among rows at that timestamp.
  • Do not let a later-inserted, backdated checkpoint override timestamp priority or decide a tie using an older checkpoint.
  • Preserve unique app IDs, partition isolation, and the existing None / "" default-partition behavior. No schema migration or other backend changes.
  • Add 20 parameterized regression cases covering both app-name orders, repeated saves, reopening the database, same-app rows in another partition, default/named/whitespace partition keys, and timestamp-versus-insertion ordering.

How I tested this

Environment: Windows, Python 3.11.2, SQLite 3.39.4, aiosqlite 0.22.1.

Before the implementation change, the 20 new cases produced 10 failures and 10 passes. Afterward:

python -m pytest tests/core/test_persistence.py tests/integrations/persisters/test_b_aiosqlite.py -p pytest_asyncio.plugin --asyncio-mode=auto -q
94 passed

python -m pytest tests/core tests/integrations/persisters/test_b_aiosqlite.py -p pytest_asyncio.plugin --asyncio-mode=auto -q -k 'not test_visualize_dot_output'
474 passed, 4 deselected

Both commands used PYTEST_DISABLE_PLUGIN_AUTOLOAD=1. The four visualization tests were excluded because Graphviz's dot executable is unavailable locally; these are not a full-repository or cross-version CI result.

The applicable pre-commit checks passed for all four changed files (Black, isort, flake8, AST, whitespace/EOF, ASF header). The frontend-only hook was excluded; no frontend files changed. git diff --check passed.

Additional local checks: 12 diagnostic cases passed, plus 100 deterministic randomized datasets matched an independent timestamp/rowid reference ordering, including nullable timestamps and partition isolation.

Notes

  • This intentionally changes only previously unspecified timestamp-tie ordering and follows the existing SQLite load() tie-breaker. It does not promise row identity across externally rewritten databases.
  • The query first selects rows at each app's latest timestamp before aggregating rowid; independent MAX(created_at), MAX(rowid) aggregates can describe different checkpoints.
  • There is a query-cost tradeoff: on a local in-memory dataset with 100,000 rows / 100 apps, five-run median query time was about 26 ms before and 69 ms after. This is a diagnostic measurement, not a production benchmark or a performance improvement. No new index is introduced.

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal
  • Applicable pre-commit checks pass (frontend-only hook excluded as noted above)
  • Changed functionality is tested
  • Existing method docstrings describe the ordering; no new public functions
  • No placeholder implementation or deferred TODOs
  • Ordering documentation updated in both method docstrings

@github-actions github-actions Bot added area/core Application, State, Graph, Actions area/storage Persisters, state storage area/integrations External integrations (LLMs, frameworks) labels Oct 21, 2026

@enekoreto enekoreto left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I checked out the branch (5d85cdf): the new cases fail on main (8bfd726, 5 sync and 5 async) and pass here, also on Python 3.10. The aiosqlite file also passes with BURR_CI_INTEGRATION_TESTS=true, the way the persisters job runs it, and CI's main test command passes.

I also ran a randomized check against a plain-Python reference (latest timestamp first, then the last row inserted at that timestamp): 300 random save histories with lots of timestamp ties, sync and async. On main, list_app_ids("pk")[0] disagreed with load("pk", None) in 54 of them. With this change the order matched the reference and the first app matched load in all 300.

Small thing, not blocking: the Postgres persisters still order by MAX(created_at) with no tie-break (b_psycopg2.py:160, b_asyncpg.py:295). With microsecond timestamps it should be much rarer there, so fine to leave out of this PR. I didn't run Postgres.

I ran this on Python 3.13 on macOS with SQLite 3.53.0, and the tests also on 3.10.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Application, State, Graph, Actions area/integrations External integrations (LLMs, frameworks) area/storage Persisters, state storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants