Skip to content

Add new method: MCV-calibrated diffusion (mcv_diffusion) - #39

Closed
Ephemeral6 wants to merge 3 commits into
openproblems-bio:mainfrom
Ephemeral6:method/mcv-diffusion
Closed

Ephemeral6 wants to merge 3 commits into
openproblems-bio:mainfrom
Ephemeral6:method/mcv-diffusion

Conversation

@Ephemeral6

Copy link
Copy Markdown

Add new method: MCV-calibrated diffusion (mcv_diffusion)

Method

  • Graph. Cells are connected by a symmetrised kNN graph (k = 15) on the PCA (50 components) of log1p-CP10k highly
    variable genes.

  • Smoothing. The log1p-CP10k matrix Y is smoothed by a lazy random walk with restart, Z = (1 - b) Y + b P^t Y.

  • Output. expm1(Z), rescaled to each cell's library size.

  • Calibration. The amount of smoothing, (t, b), is not fixed. It is chosen per dataset by molecular
    cross-validation (Batson et al., 2019) on the training counts only:

    1. The training molecules are binomially split 8/9 : 1/9, which gives a held-out part with the depth of the
      benchmark test set.
    2. Each (t, b) on a small grid is applied to the larger part.
    3. The pair with the lowest log-CP10k MSE against the held-out part is used to denoise the full training matrix.

    This adapts the smoothing to the depth and technology of each dataset (UMI vs full-length). For example, it picks
    t = 16 on mouse_blood_olsson and t = 1, b = 0.3 on allen_brain_atlas.

It runs on CPU only: scanpy, scikit-learn exact kNN, and sparse matrix products. Runtime is 0.5–9 min per dataset on
a laptop.

Local benchmark (all 17 datasets of run 2025-08-11, official metric scripts, 3 seeds)

I ran the component on every official train.h5ad and scored test.h5ad with src/metrics/mse and
src/metrics/poisson unchanged.

Pipeline check.

  • no_denoising reproduces the published values within 1e-7 on all 17 datasets.
  • The official magic component reproduces them within 1e-5 (MSE) and 2e-4 (Poisson).

Scores are scaled with the control bounds as in the website/report (mean(clip(scaled, 0, 1))):

method mean score MSE-only scaled mean
mcv_diffusion (this PR, 3 seeds) 0.2063 0.2014
knn_smoothing 0.1350 0.0936
magic 0.1003 0.1238
dca 0.0846 0.1259
cellmapper 0.0835 0.1306
alra 0.0276 0.0006
  • vs MAGIC (MSE, scaled). Paired test-cell bootstrap (B = 200): +0.0776, 95% CI [0.0772, 0.0779]. Dataset-level
    bootstrap: 95% CI [0.057, 0.101]. Better on 15/17 datasets, tied on 2 (both scaled 0).
  • vs knn_smoothing (mean score). Dataset-level bootstrap: +0.071, 95% CI [0.033, 0.108]; better on 14/17 datasets.

Caveats (please read)

  • Poisson half of the mean score. That part depends on the current Poisson metric, whose argument order PR Fix: Correct argument order in poisson_nll_loss function call #37
    proposes to fix.
    • Under the corrected (fixed) Poisson NLL, this method beats no_denoising on 15/17 datasets but MAGIC on only
      5/17; it is slightly worse on the rest.
    • The MSE lead does not depend on this.
  • Datasets where it does not help. On allen_brain_atlas and mouse_hspc_nestorowa2016 (full-length,
    low-cell-count), its MSE is worse than no_denoising, so both score 0 there.
  • Metric-targeted calibration. The calibration directly targets the benchmark's own loss (log-CP10k MSE at test
    depth), estimated from the training molecules. No test data is used.
  • Development data. Hyperparameter grid and design were developed on splits of the training counts; the official
    test files were scored once, after the protocol was fixed.
  • Seeds. Seeds only move the MCV split, so the three seeds give near-identical outputs.
  • Comparators. scprint has no scores in run 2025-08-11, so it is not compared.
  • Not run locally. viash test was not run here (no viash on this machine). I'm happy to fix anything CI flags.

Changes

  • src/methods/mcv_diffusion/ (config.vsh.yaml, script.py).
  • Registered in src/workflows/run_benchmark/config.vsh.yaml and main.nf.
  • CHANGELOG entry.

AI assistance disclosure

An AI agent (Anthropic Claude, via Claude Code) designed, implemented, evaluated and wrote up this method, operating
the Ephemeral6 account. All numbers come from logged runs, and a separate agent independently re-scored and audited
the results.

🤖 Generated with Claude Code

panguancheng and others added 3 commits October 30, 2026 07:22
@Ephemeral6 Ephemeral6 closed this by deleting the head repository Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant