Skip to content

Annotate test resources in LaminDB - #20

Open
rcannood wants to merge 2 commits into
mainfrom
lamin-annotation
Open

rcannood wants to merge 2 commits into
mainfrom
lamin-annotation

Conversation

@rcannood

Copy link
Copy Markdown
Member

Describe your changes

Annotate dataset artifacts in LaminDB with everything the OpenProblems website shows about a dataset, so it can be rendered from the instance. Used for the test resources for now; it moves to core once it settles.

lamin_annotate.annotate_artifact():

  • generates a schema from the src/api file format with every slot (obs, var, uns, layers, obsm, obsp, varm) and attaches it, so the hub documents the whole file. obs, var and uns are validated with AnnDataCurator first, since lamindb can't curate the other slots yet; their validation stays in validate_dataset.
  • links the ontology terms in obs (organism, cell type, tissue, disease, sex, assay, development stage, ethnicity) through bionty. Mouse development stages are looked up in MmusDv, and only CELLxGENE placeholders (na, unknown) are created by hand.
  • stores the dataset_* metadata and normalization_id from uns, plus n_vars, n_donors, n_batches, n_hvg, n_pca_components and knn_n_neighbors as feature values, and sets n_observations
  • stores obs_counts (cells per category) and structure (per slot: name, type, dtype, shape and the description from the spec, read from the zarr metadata only)
  • links the references from common/library.bib as ln.Reference records with their DOI, and labels the modality (scrnaseq)

The readers for the structure and summary are per file format; anndata_zarr is the only one for now, MuData, SpatialData, data frames and JSON/YAML follow later. Features are grouped per slot (OpenProblems > obs, uns, ...), since e.g. dataset_id means something else in obs and uns.

The notebooks pass the file format of each step (helpers.SCRNASEQ_SPECS) and record the workflow parameters on their run. Re-running them annotated the existing test resources in openproblems/data-dev without creating new versions, e.g. the mouse pancreas dataset.

Two upstream issues worked around here: bionty's Ensembl organism source lists mouse strains under the same NCBITaxon:10090, and lamindb's save() then recurses forever on the duplicate uids, so the species is registered before curating.

Issue ticket number and link

n/a

Checklist before requesting a review

  • I have performed a self-review of my code

  • Check the correct box. Does this PR contain:

    • Breaking changes
    • New functionality
    • Major changes
    • Minor changes
    • Bug fixes
    • Documentation
  • Proposed changes are described in the CHANGELOG.md

  • CI Tests succeed and look good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant