Repository navigation
Conversation
The target gene identifier statistics test updated the first target of TEST_MINIMAL_ACC_SCORESET and TEST_MINIMAL_SEQ_SCORESET in place, so any score set test that ran later in the same process created its score set with three external identifiers and failed its response comparison. Full-suite runs collect test_score_set.py before test_statistics.py, so CI never ran a score set test after the mutation. Listing the files in another order, or a different xdist split, did. The test now deep-copies the target and passes it as a targetGenes update.
Model mixins lived in db/mixins.py, and the external identifier base for #944 needs a home too. Splitting mixins between db/ and models/ by whether they're MaveDB-specific would leave two places named mixins and a judgment call for each new one. Every model mixin now lives in models/mixins/<concept>.py, and db/ keeps what every table builds on: Base, MaterializedView and sessions. The database rule says so, and now loads for db/ too. ValidTime's code is unchanged apart from its module docstring.
Identifier search fans out from identifiers to the records that cite them, so it needs a request-scoped read check per record type. Each viewer delegates to its module's has_permission and memoizes the result, as the existing viewers do.
… that use them DOI, raw read and target gene identifiers each lived in a shared table that records linked to by id, though every column but the identifier was constant, null or derivable from it. Each identifier now lives on the row that cites it, so under row-level security (#808) it is an owned row, visible exactly when its record is. The six DOI and raw read link tables hold the identifier, keyed by (<parent>_id, identifier). The three offset tables merge into target_gene_external_identifiers, keyed by (target_gene_id, db_name). Writes build link rows from the submitted strings, replacing the find-or-create helpers. Links are computed on serialization from lib/external_identifier_urls.py: raw read links follow the accession type (#445) and gene identifier links, always null before, are filled. DOIs are validated by a reusable Doi field type and stored bare and upper-cased, the registry's form, so a record can't cite one DOI twice. Identifier search returns only identifiers on records the caller can read. Publications keep their shared table (#945), since their rows hold fetched metadata. BREAKING CHANGE: DoiIdentifier and RawReadIdentifier lose id, ExternalGeneIdentifier loses referenceHtml, and DOIs come back upper-cased.
Before this change the api accepted DOIs written with a doi: or resolver prefix and stored them as given. The fold only upper-cased them, so such a row would have become HTTPS://DOI.ORG/..., with a broken link and no match against the bare form new submissions store. Strip the prefixes idutils.normalize_doi strips. The 2026.3.0 dump has none, but production accepts them until this ships.
A search for % or _ matched every identifier. Read access is checked in Python before the result limit, so that loaded every record that cites one, on a route anyone can call.
The DOI and raw read link models were listed both among the association tables and in their own union, and the count narrowed from one to the other with a type: ignore. Give them their own typed map, so a new link goes in one place and mypy checks it.
The DOI and raw read search routes each spelled out the same three link-record joins. identifier_sources_on_records builds them from the three link models, so a new record type changes one function.
…load The three offsets these replace were one row each, so joining them in was cheap. External identifiers are a collection of up to three rows per gene, and joining a collection in multiplies the rows of every other joined collection on the target gene and its score set. Load it in its own query instead.
bencap
added
app: backend
Coverage Report for CI Build 39997023889Coverage increased (+0.1%) to 90.519%Details
Uncovered Changes
Coverage Regressions12 previously-covered lines in 3 files lost coverage.
Coverage Stats
💛 - Coveralls |
The search checks read access through the permission viewers, which import fastapi. View models import lib/identifiers for publication authors, so they could no longer load without the server extras, and the core-dependency test jobs failed to collect. The search now lives in lib/identifier_search, which only the routers import.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DOI, raw read and target gene identifiers (Ensembl, RefSeq, UniProt) each lived in a shared table that records linked to by id. Every column but the identifier was constant, always null, or derivable from it. This moves each identifier string onto the row that cites it and drops the five shared tables and the three offset tables. Under row-level security (#808) they become owned rows, visible exactly when their record is.
(<parent>_id, identifier).ensembl_offsets,refseq_offsetsanduniprot_offsetsmerge intotarget_gene_external_identifiers, keyed by(target_gene_id, db_name). Migration2dfdc4a88870stripsdoi:and resolver prefixes from stored DOIs as it folds them; its downgrade rebuilds the old tables from the new rows.Doifield type validates them and stores them bare and upper-cased, the form the DOI registry uses, so a record can't cite one DOI twice in different cases.lib/external_identifier_urls.py. Raw read links follow the accession type (BioProject, GEO, ArrayExpress, otherwise ENA), and gene identifier links are filled in; they were always null.ScoreSetViewer,ExperimentViewerandExperimentSetViewer.ExternalIdentifiermixin inmodels/mixins/.ValidTimemoves there too, in its own commit, and.claude/rules/database.mdrecords where model mixins live.API changes:
DoiIdentifierandRawReadIdentifierloseid, andExternalGeneIdentifierlosesreferenceHtml. DOIs come back upper-cased, including the DOI statistics keys. The search routes now accept an optional login. VariantEffect/mavedb-ui#755 regeneratessrc/schema/openapi.d.tsand makes the ui's duplicate-DOI check ignore case; merge the two together.Closes #944
Closes #445
Tests
tests/routers/test_record_identifiers.py: create, update and search for each identifier type, including a private record's identifier staying out of anonymous and other users' searches.Doitype and the three viewers, and a search test that%and_match literally.id,referenceHtml,*_offset).bin/checkand the PR suite (bin/t tests/ -n 6 -m "not network and not slow": 5,473 passed, 22 skipped).Checklist
release-<year>.<n>.0branch otherwise; never
main(breaking, but targets the 2026.3.3 patch on purpose, so it ships ahead of the cycle)mavedb-uiPR regeneratessrc/schema/openapi.d.ts(chore(schema): regenerate API types for identifiers stored on owner rows mavedb-ui#755)