Skip to content

Split scRNA-seq workflows into load and process - #17

Merged
rcannood merged 10 commits into
mainfrom
scrnaseq-workflows
Nov 18, 2026
Merged

rcannood merged 10 commits into
mainfrom
scrnaseq-workflows

Conversation

@rcannood

@rcannood rcannood commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Describe your changes

Split the scRNA-seq workflows into loading and processing, so a raw dataset can be published once and processed again whenever the processing changes.

  • workflows/scrnaseq/load_cellxgene_census, load_openproblems_op3: loader → optional subsample → validate → output_raw. Replace process_cellxgene_census and process_openproblems_op3. The subsample arguments are shared through src/workflows/scrnaseq/arguments_subsample.yaml, and --seed now defaults to 0.
  • workflows/scrnaseq/process_scrnaseq: raw dataset → log CP10k → hvg → pca → knn → validate → output_dataset. One normalisation only; the intermediate files are optional outputs, for test resources.
  • validation/validate_dataset: checks a raw and/or common dataset against its src/api format with check_format() from the openproblems package. No output, so a mismatch just fails the run before anything is published.
  • load_cellxgene_census refuses latest and stable as Census version, since those give a different dataset over time.
  • Pin OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS and NUMBA_NUM_THREADS to 4 in every image. By default these use all cores of the host, and the PCA (and so the kNN) changed with the number of cores. Two runs with 4 threads give the same Lamin hash.
  • The cxg_* notebooks use the new workflows, and their collections are regenerated (mouse pancreas Ajn1Lo8zZomVzKtQ0003), which also picks up the list-valued dataset_reference and dataset_organism from Store dataset reference and organism as lists #16. Before Store dataset reference and organism as lists #16, the new workflows reproduced the raw, normalized and hvg files byte for byte; only pca and dataset changed, because of the thread pinning.
  • scripts/create_resources/{cellxgene_census,openproblems_op3}.sh now only load the raw datasets. Processing them on de.NBI with nf-lamin comes in a next PR.

process_openproblems_v1 is deprecated and left as is.

Issue ticket number and link

n/a

Checklist before requesting a review

  • I have performed a self-review of my code

  • Check the correct box. Does this PR contain:

    • Breaking changes
    • New functionality
    • Major changes
    • Minor changes
    • Bug fixes
    • Documentation
  • Proposed changes are described in the CHANGELOG.md

  • CI Tests succeed and look good!

@rcannood
rcannood merged commit 6b01edf into main Nov 18, 2026
23 checks passed
@rcannood
rcannood deleted the scrnaseq-workflows branch November 18, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant