Repository navigation
fix: register datasets inserted via MetaCache.upsert_many - #116
Merged
Merged
Conversation
`_upsert_many_insert` called `list.__add__` on the circle registry, which creates a new list and discards it, instead of extending the registry. As a consequence, a second `upsert_many` for the same circle in the same session (incremental database update) inserted existing datasets again and failed with `sqlite3.IntegrityError`, and `upsert_dataset` inserted duplicates. The registry was only correct after restarting DCOR-Aid, when it is rebuilt from the databases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a bug in the local metadata database (
dbmodel/meta_cache.py) that I came across while adding type annotations in #114.Bug
MetaCache._upsert_many_insert()updated the circle registry withlist.__add__returns a new list, which was discarded, so datasets inserted in bulk were never registered with their circle. The registry decides whether a dataset is new (upsert_dataset) and whether a circle has been seen before (upsert_many). Consequences within one DCOR-Aid session:upsert_many()for the same circle, which is what the incrementalCachedAPIInterrogator.update()does, treated all datasets as new and inserted them again. This fails withsqlite3.IntegrityError: UNIQUE constraint failed: kv_store.key.upsert_dataset()for a dataset added viaupsert_many()inserted a duplicate.After a restart, the registry is rebuilt from the SQLite databases, so the problem only shows up when the database is updated more than once per session. The existing
test_cache_upsert_manyreopens the cache between the two upserts, which is why it didn't catch this.Fix
.__add__(ds_ids)→.extend(ds_ids)Tests
New tests in
tests/test_dbmodel_meta_cache.py(offline), all failing before the fix:test_cache_upsert_many_registry: the registry contains the inserted dataset IDs (was[])test_cache_upsert_many_twice_same_session: updating a dataset and adding a new one in the same session (wasIntegrityError); also checked after reopeningtest_cache_upsert_dataset_after_upsert_many: no duplicate (was 6 datasets instead of 5)All meta cache tests and the cached API interrogator tests pass locally against dcor-dev (39 passed).
🤖 Generated with Claude Code