Repository navigation
Add score_set_id to the allele-centric mapping tables before they ship #833
Description
Summary
RT's mapping_records links to its score set only through variants, which has millions of rows. A policy that goes through variants to find the score set takes about 700 ms to return 16 rows. This issue adds score_set_id to mapping_records and mapping_record_alleles on the RT branch, before 2026.3.0 ships, while the tables are still new and empty in production.
Terms are defined in the glossary on #808. RT is the allele-centric mapping work on feature/bencap/allele-centric-mapping-and-storage.
This has to merge into the RT branch before 2026.3.0 ships. After that it's a large-table backfill.
The change
-
Build a unique index on
variants (id, scoreset_id)by hand in production before the deploy, since migrations can't runCONCURRENTLY(Squash the migration history into one baseline so it replays from an empty database #877):CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS uq_variants_id_scoreset ON variants (id, scoreset_id);
-
A new migration at the head of the RT branch. RT's migrations have already run on staging, so the change can't go into the migration that creates the tables. It:
- creates the same index with
IF NOT EXISTSand withoutCONCURRENTLY, which does nothing where it was built by hand; - adds
score_set_idtomapping_records, backfills it fromvariants, sets itNOT NULL, and adds a unique index onmapping_records (id, score_set_id); - replaces the single-column
variant_idforeign key withFOREIGN KEY (variant_id, score_set_id) REFERENCES variants (id, scoreset_id), sincevariantskeeps its older column name; - does the same for
mapping_record_alleles, backfilled frommapping_records, withFOREIGN KEY (mapping_record_id, score_set_id) REFERENCES mapping_records (id, score_set_id) ON DELETE CASCADE.
The backfills are sized for staging. In production the tables are empty when it runs.
- creates the same index with
-
Update both ORM models so
score_set_idis required, and set it wherever rows are created: the mapping job, reverse translation, and the reshape script feat: Retroactive backfill — migrate MappedVariant data and run reverse translation for existing score sets #747 uses.
The foreign key covers both columns so a row can't point at one score set's variant while claiming another, public score set.
mapped_variants doesn't get the column. It stays without RLS until #747 drops it (#832).
Acceptance criteria
-
mapping_records.score_set_idandmapping_record_alleles.score_set_idexist,NOT NULL, under two-column foreign keys. - A test asserts inserting a mapping record whose
score_set_iddoesn't match its variant's is rejected. - The mapping job, reverse translation and feat: Retroactive backfill — migrate MappedVariant data and run reverse translation for existing score sets #747's reshape script set
score_set_idon every row. - The migration upgrades and downgrades on a staging-shaped database, with the hand-built index present and absent.
Background
On 2,000 score sets and 400k rows, delegating through variants took 688 ms to return 16 rows, because the plan hashes every visible variant whatever the outer query selects. With score_set_id on the row, the policy is a semi-join against about 2,000 score set ids: about 20 ms for a full scan and under 1 ms for one score set.
If RT ships before this merges, the fallback is adding the column afterwards: add it nullable, backfill as mavedb_migrate, add a validated CHECK (score_set_id IS NOT NULL), then SET NOT NULL, and add the foreign key NOT VALID then VALIDATE, with lock_timeout set.
Activity
[-]Carry scoreset_id on mapped_variants under a composite foreign key[/-][+]Carry scoreset_id on mapping_records and mapped_variants under a composite foreign key[/+][-]Carry scoreset_id on mapping_records and mapped_variants under a composite foreign key[/-][+]Add scoreset_id to the allele-centric mapping tables before they ship[/+][-]Add scoreset_id to the allele-centric mapping tables before they ship[/-][+]Add score_set_id to the allele-centric mapping tables before they ship[/+]feat(rls): carry score_set_id on mapping_records and mapping_record_a…
bencap commentedon Oct 21, 2026
Shipped in 2026.3.0 with 5d23963 (migration
45e8ec1db15e). Both columns areNOT NULLunder the two-column foreign keys,tests/lib/csv/test_variant.pycovers the mismatch rejection, and the mapping job, reverse translation and the reshape script set the column. The upgrade ran on staging and production. The downgrade check is moot now that #938 squashed the revision into the 2026.3.0 baseline.