Skip to content

Add score_set_id to the allele-centric mapping tables before they ship #833

Closed

Description

@bencap

Summary

RT's mapping_records links to its score set only through variants, which has millions of rows. A policy that goes through variants to find the score set takes about 700 ms to return 16 rows. This issue adds score_set_id to mapping_records and mapping_record_alleles on the RT branch, before 2026.3.0 ships, while the tables are still new and empty in production.

Terms are defined in the glossary on #808. RT is the allele-centric mapping work on feature/bencap/allele-centric-mapping-and-storage.

This has to merge into the RT branch before 2026.3.0 ships. After that it's a large-table backfill.

The change

  1. Build a unique index on variants (id, scoreset_id) by hand in production before the deploy, since migrations can't run CONCURRENTLY (Squash the migration history into one baseline so it replays from an empty database #877):

    CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS uq_variants_id_scoreset ON variants (id, scoreset_id);
  2. A new migration at the head of the RT branch. RT's migrations have already run on staging, so the change can't go into the migration that creates the tables. It:

    • creates the same index with IF NOT EXISTS and without CONCURRENTLY, which does nothing where it was built by hand;
    • adds score_set_id to mapping_records, backfills it from variants, sets it NOT NULL, and adds a unique index on mapping_records (id, score_set_id);
    • replaces the single-column variant_id foreign key with FOREIGN KEY (variant_id, score_set_id) REFERENCES variants (id, scoreset_id), since variants keeps its older column name;
    • does the same for mapping_record_alleles, backfilled from mapping_records, with FOREIGN KEY (mapping_record_id, score_set_id) REFERENCES mapping_records (id, score_set_id) ON DELETE CASCADE.

    The backfills are sized for staging. In production the tables are empty when it runs.

  3. Update both ORM models so score_set_id is required, and set it wherever rows are created: the mapping job, reverse translation, and the reshape script feat: Retroactive backfill — migrate MappedVariant data and run reverse translation for existing score sets #747 uses.

The foreign key covers both columns so a row can't point at one score set's variant while claiming another, public score set.

mapped_variants doesn't get the column. It stays without RLS until #747 drops it (#832).

Acceptance criteria

Background

On 2,000 score sets and 400k rows, delegating through variants took 688 ms to return 16 rows, because the plan hashes every visible variant whatever the outer query selects. With score_set_id on the row, the policy is a semi-join against about 2,000 score set ids: about 20 ms for a full scan and under 1 ms for one score set.

If RT ships before this merges, the fallback is adding the column afterwards: add it nullable, backfill as mavedb_migrate, add a validated CHECK (score_set_id IS NOT NULL), then SET NOT NULL, and add the foreign key NOT VALID then VALIDATE, with lock_timeout set.

Activity

  1. added
    app: databaseTask implementation requires database changes
    on Aug 7, 2026
  2. added theissue type on Aug 7, 2026
  3. added this to the R2026.4 milestone on Sep 24, 2026
  4. changed the title [-]Carry scoreset_id on mapped_variants under a composite foreign key[/-] [+]Carry scoreset_id on mapping_records and mapped_variants under a composite foreign key[/+] on Sep 24, 2026
  5. changed the title [-]Carry scoreset_id on mapping_records and mapped_variants under a composite foreign key[/-] [+]Add scoreset_id to the allele-centric mapping tables before they ship[/+] on Sep 28, 2026
  6. changed the title [-]Add scoreset_id to the allele-centric mapping tables before they ship[/-] [+]Add score_set_id to the allele-centric mapping tables before they ship[/+] on Sep 28, 2026
  7. bencap commented on Oct 21, 2026

    @bencap
    CollaboratorAuthor

    Shipped in 2026.3.0 with 5d23963 (migration 45e8ec1db15e). Both columns are NOT NULL under the two-column foreign keys, tests/lib/csv/test_variant.py covers the mismatch rejection, and the mapping job, reverse translation and the reshape script set the column. The upgrade ran on staging and production. The downgrade check is moot now that #938 squashed the revision into the 2026.3.0 baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backendapp: databaseTask implementation requires database changes

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions