Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@

* MAJOR CHANGE: Raw scRNA-seq datasets must have CELLxGENE-style ontology term ids for the assay, cell type, developmental stage, disease, self-reported ethnicity, sex and tissue. The multimodal loaders use a new `file_multimodal_raw` spec, in which these stay optional.

* BUG FIX: The loaders store `uns['dataset_reference']` and `uns['dataset_organism']` as lists, as declared in `src/api`, since a dataset can have several references and organisms. The OP3 loader takes both as arguments instead of hardcoding them.

* BUG FIX: `uns['pca_variance']` is declared as an `object` in `file_pca`, since `processors/pca` writes a dict.

* MINOR CHANGE: The OP3 loader adds these ontology term ids, based on its cell type annotation and the donor metadata in GEO.

* MINOR CHANGE: Component tests use the `cxg_mouse_pancreas_atlas` test resources instead of the deprecated `pancreas` ones. Its notebook now keeps every processing step.
Expand Down
2 changes: 1 addition & 1 deletion scripts/create_resources/openproblems_v1_multimodal.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ param_list:
dataset_description: "Single cell RNA-seq and ATAC-seq co-profiling for HEK293T cells, NIH/3T3 cells, A549 cells across three treatment conditions (DEX 0 hour, 1 hour and 3 hour treatment)."
dataset_reference: cao2018joint
dataset_url: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE117089
dataset_organism: "[homo_sapiens, mus_musculus]"
dataset_organism: [homo_sapiens, mus_musculus]
obs_cell_type: cell_name
layer_counts: counts
var_feature_id: index
Expand Down
2 changes: 1 addition & 1 deletion scripts/create_test_resources/scicar_cell_lines.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
" \"dataset_reference\": \"cao2018joint\",\n",
" \"dataset_summary\": \"sciCAR is a combinatorial indexing-based assay that jointly measures cellular transcriptomes and the accessibility of cellular chromatin in the same cells\",\n",
" \"dataset_description\": \"sciCAR is a combinatorial indexing-based assay that jointly measures cellular transcriptomes and the accessibility of cellular chromatin in the same cells. Here, we use two sciCAR datasets that were obtained from the same study. The first dataset contains 4,825 cells from three cell lines (HEK293T cells, NIH/3T3 cells, and A549 cells) at multiple timepoints (0, 1 hour, 3 hours) after dexamethasone treatment. The second dataset contains 11,233 cells from wild-type adult mouse kidney.\",\n",
" \"dataset_organism\": \"[homo_sapiens, mus_musculus]\",\n",
" \"dataset_organism\": [\"homo_sapiens\", \"mus_musculus\"],\n",
" \"mod1\": \"GEX\",\n",
" \"mod2\": \"ATAC\",\n",
" \"do_subsample\": True,\n",
Expand Down
2 changes: 1 addition & 1 deletion src/api/file_pca.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ info:
description: The PCA loadings matrix.
required: true
uns:
- type: double
- type: object
name: pca_variance
description: The PCA variance objects.
required: true
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -47,6 +48,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Outputs
Expand Down
4 changes: 2 additions & 2 deletions src/loaders/multimodal/openproblems_neurips2021_bmmc/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
assert "batch" in output_mod1.obs.columns, "batch column not found in mod 1 output obs"
assert output_mod1.uns["dataset_name"] == "BMMC (Multiome)", "Expected: Pancreas as value for dataset_name in mod 1 output uns"
assert output_mod1.uns["dataset_url"] == "http://foo.org", "Expected: http://foo.org as value for dataset_url in mod 1 output uns"
assert output_mod1.uns["dataset_reference"] == "foo2000bar", "Expected: foo2000bar as value for dataset_reference in mod 1 output uns"
assert list(output_mod1.uns["dataset_reference"]) == ["foo2000bar"], "Expected: foo2000bar as value for dataset_reference in mod 1 output uns"
assert output_mod1.uns["dataset_summary"] == "A short summary.", "Expected: A short summary. as value for dataset_summary in mod 1 output uns"
assert output_mod1.uns["dataset_description"] == "A couple of paragraphs worth of text.", "Expected: A couple of paragraphs worth of text. as value for dataset_description in mod 1 output uns"

Expand All @@ -88,6 +88,6 @@
assert "batch" in output_mod2.obs.columns, "batch column not found in mod 2 output obs"
assert output_mod2.uns["dataset_name"] == "BMMC (Multiome)", "Expected: Pancreas as value for dataset_name in mod 2 output uns"
assert output_mod2.uns["dataset_url"] == "http://foo.org", "Expected: http://foo.org as value for dataset_url in mod 2 output uns"
assert output_mod2.uns["dataset_reference"] == "foo2000bar", "Expected: foo2000bar as value for dataset_reference in mod 2 output uns"
assert list(output_mod2.uns["dataset_reference"]) == ["foo2000bar"], "Expected: foo2000bar as value for dataset_reference in mod 2 output uns"
assert output_mod2.uns["dataset_summary"] == "A short summary.", "Expected: A short summary. as value for dataset_summary in mod 2 output uns"
assert output_mod2.uns["dataset_description"] == "A couple of paragraphs worth of text.", "Expected: A couple of paragraphs worth of text. as value for dataset_description in mod 2 output uns"
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -52,6 +53,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Outputs
Expand Down
4 changes: 2 additions & 2 deletions src/loaders/multimodal/openproblems_neurips2022_pbmc/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@
assert "batch" in output_mod1.obs.columns, "batch column not found in mod 1 output obs"
assert output_mod1.uns["dataset_name"] == "Kaggle22 PBMC (CITE-seq)", "Expected: Kaggle22 PBMC (CITE-seq) as value for dataset_name in mod 1 output uns"
assert output_mod1.uns["dataset_url"] == "https://www.kaggle.com/competitions/open-problems-multimodal/data", "Expected: https://www.kaggle.com/competitions/open-problems-multimodal/data as value for dataset_url in mod 1 output uns"
assert output_mod1.uns["dataset_reference"] == "Neurips22", "Expected: Neurips22 as value for dataset_reference in mod 1 output uns"
assert list(output_mod1.uns["dataset_reference"]) == ["Neurips22"], "Expected: Neurips22 as value for dataset_reference in mod 1 output uns"
assert output_mod1.uns["dataset_summary"] == "Neurips22 competition dataset", "Expected: Neurips22 competition dataset as value for dataset_summary in mod 1 output uns"
assert output_mod1.uns["dataset_description"] == "The dataset for this competition comprises single-cell multiomics data collected from mobilized peripheral CD34+ hematopoietic stem and progenitor cells (HSPCs) isolated from four healthy human donors.", "Expected: The dataset for this competition comprises single-cell multiomics data collected from mobilized peripheral CD34+ hematopoietic stem and progenitor cells (HSPCs) isolated from four healthy human donors. as value for dataset_description in mod 1 output uns"

Expand All @@ -95,6 +95,6 @@
assert "batch" in output_mod2.obs.columns, "batch column not found in mod 2 output obs"
assert output_mod2.uns["dataset_name"] == "Kaggle22 PBMC (CITE-seq)", "Expected: Kaggle22 PBMC (CITE-seq) as value for dataset_name in mod 2 output uns"
assert output_mod2.uns["dataset_url"] == "https://www.kaggle.com/competitions/open-problems-multimodal/data", "Expected: https://www.kaggle.com/competitions/open-problems-multimodal/data as value for dataset_url in mod 2 output uns"
assert output_mod2.uns["dataset_reference"] == "Neurips22", "Expected: Neurips22 as value for dataset_reference in mod 2 output uns"
assert list(output_mod2.uns["dataset_reference"]) == ["Neurips22"], "Expected: Neurips22 as value for dataset_reference in mod 2 output uns"
assert output_mod2.uns["dataset_summary"] == "Neurips22 competition dataset", "Expected: Neurips22 competition dataset as value for dataset_summary in mod 2 output uns"
assert output_mod2.uns["dataset_description"] == "The dataset for this competition comprises single-cell multiomics data collected from mobilized peripheral CD34+ hematopoietic stem and progenitor cells (HSPCs) isolated from four healthy human donors.", "Expected: The dataset for this competition comprises single-cell multiomics data collected from mobilized peripheral CD34+ hematopoietic stem and progenitor cells (HSPCs) isolated from four healthy human donors. as value for dataset_description in mod 2 output uns"
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -62,6 +63,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Outputs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"dataset_reference": "reference",
"dataset_summary": "summary",
"dataset_description": "description",
"dataset_organism": "[homo_sapiens, mus_musculus]",
"dataset_organism": ["homo_sapiens", "mus_musculus"],
}
meta = {
"resources_dir": "src/loaders/openproblems_v1/"
Expand Down
4 changes: 2 additions & 2 deletions src/loaders/multimodal/openproblems_v1_multimodal/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
assert output_mod1.uns["dataset_id"] == dataset_id, f"Expected: {dataset_id} as value for dataset_id in mod 1 output uns"
assert output_mod1.uns["dataset_name"] == "Pancreas", "Expected: Pancreas as value for dataset_name in mod 1 output uns"
assert output_mod1.uns["dataset_url"] == "http://foo.org", "Expected: http://foo.org as value for dataset_url in mod 1 output uns"
assert output_mod1.uns["dataset_reference"] == "foo2000bar", "Expected: foo2000bar as value for dataset_reference in mod 1 output uns"
assert list(output_mod1.uns["dataset_reference"]) == ["foo2000bar"], "Expected: foo2000bar as value for dataset_reference in mod 1 output uns"
assert output_mod1.uns["dataset_summary"] == "A short summary.", "Expected: A short summary. as value for dataset_summary in mod 1 output uns"
assert output_mod1.uns["dataset_description"] == "A couple of paragraphs worth of text.", "Expected: A couple of paragraphs worth of text. as value for dataset_description in mod 1 output uns"

Expand All @@ -78,7 +78,7 @@
assert output_mod2.uns["dataset_id"] == dataset_id, f"Expected: {dataset_id} as value for dataset_id in mod 2 output uns"
assert output_mod2.uns["dataset_name"] == "Pancreas", "Expected: Pancreas as value for dataset_name in mod 2 output uns"
assert output_mod2.uns["dataset_url"] == "http://foo.org", "Expected: http://foo.org as value for dataset_url in mod 2 output uns"
assert output_mod2.uns["dataset_reference"] == "foo2000bar", "Expected: foo2000bar as value for dataset_reference in mod 2 output uns"
assert list(output_mod2.uns["dataset_reference"]) == ["foo2000bar"], "Expected: foo2000bar as value for dataset_reference in mod 2 output uns"
assert output_mod2.uns["dataset_summary"] == "A short summary.", "Expected: A short summary. as value for dataset_summary in mod 2 output uns"
assert output_mod2.uns["dataset_description"] == "A couple of paragraphs worth of text.", "Expected: A couple of paragraphs worth of text. as value for dataset_description in mod 2 output uns"

Expand Down
2 changes: 2 additions & 0 deletions src/loaders/scrnaseq/cellxgene_census/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -132,6 +133,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: true
resources:
Expand Down
4 changes: 2 additions & 2 deletions src/loaders/scrnaseq/cellxgene_census/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ def test_cellxgene_extract_metadata_expression(run_component, tmp_path):
assert adata.uns["dataset_id"] == "test_dataset_id", "Incorrect .uns['dataset_id']"
assert adata.uns["dataset_name"] == "test_dataset_name", "Incorrect .uns['dataset_name']"
assert adata.uns["dataset_url"] == "https://test_dataset_url.com", "Incorrect .uns['dataset_url']"
assert adata.uns["dataset_reference"] == "test_dataset_reference", "Incorrect .uns['dataset_reference']"
assert list(adata.uns["dataset_reference"]) == ["test_dataset_reference"], "Incorrect .uns['dataset_reference']"
assert adata.uns["dataset_summary"] == "test_dataset_summary", "Incorrect .uns['dataset_summary']"
assert adata.uns["dataset_description"] == "test_dataset_description", "Incorrect .uns['dataset_description']"
assert adata.uns["dataset_organism"] == "test_homo_sapiens", "Incorrect .uns['dataset_organism']"
assert list(adata.uns["dataset_organism"]) == ["test_homo_sapiens"], "Incorrect .uns['dataset_organism']"


if __name__ == '__main__':
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -89,6 +90,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: true
resources:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,10 @@ def test_cellxgene_extract_metadata_expression(run_component, tmp_path):
assert adata.uns["dataset_id"] == "test_dataset_id", "Incorrect .uns['dataset_id']"
assert adata.uns["dataset_name"] == "test_dataset_name", "Incorrect .uns['dataset_name']"
assert adata.uns["dataset_url"] == "https://test_dataset_url.com", "Incorrect .uns['dataset_url']"
assert adata.uns["dataset_reference"] == "test_dataset_reference", "Incorrect .uns['dataset_reference']"
assert list(adata.uns["dataset_reference"]) == ["test_dataset_reference"], "Incorrect .uns['dataset_reference']"
assert adata.uns["dataset_summary"] == "test_dataset_summary", "Incorrect .uns['dataset_summary']"
assert adata.uns["dataset_description"] == "test_dataset_description", "Incorrect .uns['dataset_description']"
assert adata.uns["dataset_organism"] == "test_homo_sapiens", "Incorrect .uns['dataset_organism']"
assert list(adata.uns["dataset_organism"]) == ["test_homo_sapiens"], "Incorrect .uns['dataset_organism']"


if __name__ == '__main__':
Expand Down
11 changes: 9 additions & 2 deletions src/loaders/scrnaseq/openproblems_op3/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,18 @@ argument_groups:
type: string
description: "Detailed description of the dataset"
default: "The OP3 dataset is to-date the largest single-cell small molecule perturbation dataset in primary tissue with multiple donor replicates."
- name: "dataset_reference"
- name: "--dataset_reference"
type: string
multiple: true
description: "Bibtex reference of the paper in which the dataset was published."
required: false
default: GSE279945
default: [GSE279945]
- name: "--dataset_organism"
type: string
multiple: true
description: "The organism of the dataset."
required: false
default: [homo_sapiens]
- name: "--dataset_url"
type: string
description: "Link to the original source of the dataset."
Expand Down
6 changes: 4 additions & 2 deletions src/loaders/scrnaseq/openproblems_op3/script.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
"dataset_name": "OP3: single-cell multimodal dataset in PBMCs for perturbation prediction benchmarking",
"dataset_summary": "The Open Problems Perurbation Prediction (OP3) dataset with small molecule perturbations in PBMCs",
"dataset_description": "The OP3 dataset is to-date the largest single-cell small molecule perturbation dataset in primary tissue with multiple donor replicates.",
"dataset_reference": ["GSE279945"],
"dataset_organism": ["homo_sapiens"],
"output": "output.zarr",
}
meta = {}
Expand Down Expand Up @@ -151,9 +153,9 @@ def add_metadata_to_uns(adata, par):
adata.uns['dataset_name'] = par["dataset_name"]
adata.uns['dataset_summary'] = par["dataset_summary"]
adata.uns['dataset_description'] = par["dataset_description"]
adata.uns['dataset_organism'] = 'Homo sapiens'
adata.uns['dataset_organism'] = par["dataset_organism"]
adata.uns['dataset_url'] = par["dataset_url"]
adata.uns['dataset_reference'] = 'GSE279945'
adata.uns['dataset_reference'] = par["dataset_reference"]
adata.uns['dataset_version'] = '1.0.0'
adata.uns['processing_status'] = 'processed'

Expand Down
2 changes: 2 additions & 0 deletions src/loaders/scrnaseq/openproblems_v1/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -66,6 +67,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
resources:
Expand Down
2 changes: 2 additions & 0 deletions src/loaders/spatial/tenx_visium/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -49,6 +50,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Gene or spot filtering
Expand Down
2 changes: 1 addition & 1 deletion src/loaders/spatial/tenx_visium/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
assert adata.uns["dataset_name"] == dataset_name, f"Expected {dataset_name} as value"
assert adata.uns["dataset_url"] == dataset_url, f"Expected {dataset_url} as value"
assert adata.uns["dataset_summary"] == dataset_summary, f"Expected {dataset_summary} as value"
assert adata.uns["dataset_organism"] == dataset_organism, f"Expected {dataset_organism} as value"
assert list(adata.uns["dataset_organism"]) == [dataset_organism], f"Expected {dataset_organism} as value"
assert 'spatial' in adata.obsm, "Spatial spot coordinates not found in .obsm"

print(">> All tests passed successfully", flush=True)
2 changes: 2 additions & 0 deletions src/loaders/spatial/zenodo/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -45,6 +46,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Gene or spot filtering
Expand Down
2 changes: 1 addition & 1 deletion src/loaders/spatial/zenodo/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
assert adata.uns["dataset_name"] == dataset_name, f"Expected {dataset_name} as value"
assert adata.uns["dataset_url"] == dataset_url, f"Expected {dataset_url} as value"
assert adata.uns["dataset_summary"] == dataset_summary, f"Expected {dataset_summary} as value"
assert adata.uns["dataset_organism"] == dataset_organism, f"Expected {dataset_organism} as value"
assert list(adata.uns["dataset_organism"]) == [dataset_organism], f"Expected {dataset_organism} as value"
assert 'spatial' in adata.obsm, "Spatial spot coordinates not found in .obsm"

print(">> All tests passed successfully", flush=True)
2 changes: 2 additions & 0 deletions src/loaders/spatial/zenodo_slidetags/config.vsh.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ argument_groups:
required: false
- name: --dataset_reference
type: string
multiple: true
description: Bibtex reference of the paper in which the dataset was published.
required: false
- name: --dataset_summary
Expand All @@ -45,6 +46,7 @@ argument_groups:
required: true
- name: --dataset_organism
type: string
multiple: true
description: The organism of the dataset.
required: false
- name: Gene or spot filtering
Expand Down
Loading
Loading