Repository navigation
fix: end each psycopg2 persister call's transaction - #950
sangkyoonnam wants to merge 3 commits into
Conversation
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
elijahbenizzy
left a comment
There was a problem hiding this comment.
Right fix, and it's parity with asyncpg. Three things before merge: rebase onto #942 once it lands (same list_app_ids body); while you're in list_app_ids, map a None partition key to PARTITION_KEY_DEFAULT for both psycopg2 and asyncpg, like save and load already do; and use 'with self.connection.cursor() as cursor' so cursors stop leaking. Please call out the shared-connection behavior change in the description for the release notes. Elijah
…ion-boundaries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@elijahbenizzy thanks, all four are in. I merged |
PostgreSQLPersisternow commits or rolls back afteris_initialized,list_app_ids,loadandsave, so one failed insert no longer breaks the persister for the rest of the process, and reads no longer leave a transaction open.psycopg2 runs with autocommit off, so the first statement on the connection opens a transaction.
savecommits on success but doesn't roll back on failure. After a rejected insert, such as a duplicate(partition_key, app_id, sequence_id, position)or a state string with a NUL character, every laterload,list_app_idsandsaveon that persister raisesInFailedSqlTransaction. On main the repro is: save a row, save it again (UniqueViolation), thenloadthe same app.AsyncPostgreSQLPersisterisn't affected, since asyncpg doesn't open a transaction unless asked to.load,list_app_idsandis_initializednever end their transaction, so the session sits inidle in transactionuntil the nextsavecommits.created_atdefaults toCURRENT_TIMESTAMP, which is the transaction start time, so that save is stamped with the time of the earlier read. With two connections, a read on A, a save on B, then a save on A stores A's row with an earliercreated_atthan B's, andload(partition_key, None)returns B's row as the latest.Changes
is_initialized,list_app_ids,loadandsaveget their cursor from a small_transaction()context manager that commits on success and rolls back on any exception,KeyboardInterruptincluded. If the rollback itself fails, usually because the connection is already gone, the original exception is the one raised. It doesn't usewith self.connection:, because psycopg2 refuses to re-enter that from a caller's ownwith connection:block, and that pattern worked on main.list_app_idsmaps aNonepartition key toPARTITION_KEY_DEFAULTin bothPostgreSQLPersisterandAsyncPostgreSQLPersister, assaveandloadalready do, solist_app_ids(None)finds apps saved without a partition key instead of matchingpartition_key = NULL._transaction()andcreate_tableopen their cursor withwith self.connection.cursor() as cursor.mainto pick up fix: list each app once in SQL persister list_app_ids, newest save first #942;list_app_idskeeps itsGROUP BY app_id ORDER BY MAX(created_at) DESCquery inside_transaction().How I tested this
UniqueViolationand the nextload,list_app_idsandsavesucceed; the connection reportsTRANSACTION_STATUS_IDLEright afterload(hit and miss),list_app_idsand an uncachedis_initialized; a save on the fixture connection after a read and another connection's save gets acreated_atat or after that save's; aKeyboardInterruptraised by the cursor mid-call leaves the connection idle; and the persister still works inside a caller'swith connection:block. Two checklist_app_ids(None)after asave(None, ...)for each persister, and one records the cursors thatis_initialized,save,loadandlist_app_idsopen and checks that all four are closed. Againstmainwith this test file, seven fail and 16 pass; all 23 pass here. The status checks are the deterministic ones; thecreated_attest relies on the server clock not going backwards.postgres:16container withBURR_CI_INTEGRATION_TESTS=true, on fresh tables, from a copy of the test file with only the port changed: 23 passed.pytest tests --ignore=tests/integrations/persisters --ignore=tests/integrations/test_bip0042_bedrock.py: 687 passed, 3 skipped.pre-commit runon the changed files passes.Notes
PostgreSQLPersisternow ends its transaction after the database work inlist_app_ids,load,saveand an uncachedis_initialized. Code that passes a connection it also uses itself is affected: a read now commits whatever that code had pending on the connection, a failed query now rolls back the whole transaction, and aloadfollowed by asaveno longer shares one transaction.savealready committed pending work onmain. Giving the persister a connection of its own keeps the caller's transactions separate.create_tabledoesn't go through_transaction(). It already commits, and a failedCREATE TABLEwould still leave the transaction aborted; that seemed out of scope for this fix.Checklist