Repository navigation
Split scRNA-seq workflows into load and process - #17
Merged
Merged
Conversation
4 of 9 tasks
3 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Split the scRNA-seq workflows into loading and processing, so a raw dataset can be published once and processed again whenever the processing changes.
workflows/scrnaseq/load_cellxgene_census,load_openproblems_op3: loader → optional subsample → validate →output_raw. Replaceprocess_cellxgene_censusandprocess_openproblems_op3. The subsample arguments are shared throughsrc/workflows/scrnaseq/arguments_subsample.yaml, and--seednow defaults to 0.workflows/scrnaseq/process_scrnaseq: raw dataset → log CP10k → hvg → pca → knn → validate →output_dataset. One normalisation only; the intermediate files are optional outputs, for test resources.validation/validate_dataset: checks a raw and/or common dataset against itssrc/apiformat withcheck_format()from theopenproblemspackage. No output, so a mismatch just fails the run before anything is published.load_cellxgene_censusrefuseslatestandstableas Census version, since those give a different dataset over time.OMP_NUM_THREADS,OPENBLAS_NUM_THREADS,MKL_NUM_THREADSandNUMBA_NUM_THREADSto 4 in every image. By default these use all cores of the host, and the PCA (and so the kNN) changed with the number of cores. Two runs with 4 threads give the same Lamin hash.cxg_*notebooks use the new workflows, and their collections are regenerated (mouse pancreasAjn1Lo8zZomVzKtQ0003), which also picks up the list-valueddataset_referenceanddataset_organismfrom Store dataset reference and organism as lists #16. Before Store dataset reference and organism as lists #16, the new workflows reproduced the raw, normalized and hvg files byte for byte; only pca and dataset changed, because of the thread pinning.scripts/create_resources/{cellxgene_census,openproblems_op3}.shnow only load the raw datasets. Processing them on de.NBI with nf-lamin comes in a next PR.process_openproblems_v1is deprecated and left as is.Issue ticket number and link
n/a
Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!