Skip to content

Port the spatial proteomics preprocessing benchmark onto the task structure - #1

Open
sergiomarco25 wants to merge 1 commit into
mainfrom
port-isp-benchmark
Open

sergiomarco25 wants to merge 1 commit into
mainfrom
port-isp-benchmark

Conversation

@sergiomarco25

Copy link
Copy Markdown

Replaces the task_template scaffold with the spatial proteomics preprocessing
benchmark developed in juliusstein77/task_isp_preprocessing.

What came across

21 components in six namespaces — five method stages with interchangeable
alternatives sharing one argument contract per stage, plus six metric families:

namespace methods
methods_image_correction none, percentile background, flat-field
methods_proteomics_segmentation threshold, watershed+expand, reference
methods_protein_aggregation mean, sum, median
methods_proteomics_qc none, fixed threshold, adaptive quantile
methods_proteomics_normalization none, total+log1p, arcsinh
metrics_proteomics 6 families, 66 declared metrics

Metric implementations live in sprotmetrics
0.1.0; the components here are the Viash contracts around them.

Multi-stage namespacing follows the precedent in task_ist_preprocessing
rather than the template's single methods/ and metrics/, because this task is
a chain of stages rather than one prediction step.

Deliberately left behind: the legacy transcriptomics components the source
repository inherited from its own iST fork (transcript assignment, cell-type
annotation, expression correction), and four superseded or unwired components.

Schema gaps closed

Score provenance. src/api/file_score.yaml now requires dataset_id,
normalization_id and method_id in uns. All three were commented out in the
source repository, so scores carried no provenance and could not be attributed
to a run.

info.metrics. Every metric component now declares the block
common/schemas/task_metric.yaml requires — 66 metrics, each with all eight
required keys, and viash config view round-trips them.

These are generated by scripts/create_component/generate_metric_info.py from
the sprotmetrics registry rather than written by hand, so declared direction
and bounds cannot drift from the code computing them. --check makes that a CI
gate.

Two deliberate behaviours there:

  • Metrics with no direction are omitted. maximize is required and the
    schema cannot express "descriptive", so declaring them would hand the scoring
    stage a ranking criterion they do not support. They are still computed and
    still appear in the score file.
  • references and links are backfilled. MetricInfo carries both fields
    but sprotmetrics 0.1.0 does not emit them, and the schema requires both. The
    upstream fix is worth making; a version bump should not block this port.

Removed: workflows/process_datasets

The template's version implements a train/test/solution split for label
projection and referenced four API files this task has no use for — left in
place it broke config parsing against those deleted files. This task takes a
SpatialData store and benchmarks preprocessing, so there is no equivalent split.
Recorded as a gap rather than faked.

Verification

All 21 components pass viash ns test in the new layout:

methods_image_correction        3 passed, 0 failed
methods_proteomics_segmentation 3 passed, 0 failed
methods_protein_aggregation     3 passed, 0 failed
methods_proteomics_qc           3 passed, 0 failed
methods_proteomics_normalization 3 passed, 0 failed
metrics_proteomics              6 passed, 0 failed

Run against openproblems/base_python:1 with the pinned dependency versions.
Tests build their own synthetic fixtures, so they need no dataset and no S3.

Known gaps, also in CHANGELOG.md

  • run_benchmark/main.nf is still the hand-written form inherited from the
    source repository — one process per stage with a case block — rather than
    the runEach / fromState idiom. Porting it is what will supply the
    method_id and dataset_id that file_score.yaml now requires, so those two
    are one job.
  • Test resources are not on S3. _viash.yaml points at the intended
    location; scripts/create_resources/isp/ builds the input locally from the
    public MCMICRO exemplar-001 download in the meantime. That download is raw
    acquisition data, so the script registers the cycles with ashlar and converts
    to a SpatialData store.
  • README.md is hand-written. scripts/create_readme.sh renders it through
    a container, which could not be run in the development environment. The file
    says so and will be overwritten when the generator is run.
  • reference segmentation is untestable on the public dataset: the download
    carries no segmentation masks, those being MCMICRO output.

🤖 Generated with Claude Code

…ucture

Replaces the task_template scaffold with the benchmark developed in
juliusstein77/task_isp_preprocessing. Only the proteomics surface came
across: the legacy transcriptomics components that repository inherited
from its own iST fork - transcript assignment, cell-type annotation,
expression correction - were left behind, along with four superseded or
unwired components.

21 components across six namespaces: five method stages with
interchangeable alternatives sharing one argument contract per stage, and
six metric families wrapping sprotmetrics 0.1.0. Multi-stage namespacing
follows the precedent set by task_ist_preprocessing rather than the
template's single methods/ and metrics/ directories, because this task is
a chain of stages rather than one prediction step.

Two gaps against the OpenProblems schemas are closed here.

file_score.yaml now requires dataset_id, normalization_id and method_id
in uns. All three were commented out in the source repository, so scores
carried no provenance and could not be attributed to a run.

Every metric component now declares the info.metrics block that
common/schemas/task_metric.yaml requires - 66 metrics across six
families. These are generated from the sprotmetrics registry by
scripts/create_component/generate_metric_info.py rather than written by
hand, so the declared direction and bounds cannot drift from the code
computing them; --check makes that a CI gate. Metrics with no direction
are deliberately omitted: maximize is a required key and the schema
cannot express "descriptive", so declaring them would hand the scoring
stage a ranking criterion they do not support. They are still computed
and still appear in the score file.

The script also backfills references and links, which MetricInfo carries
but sprotmetrics 0.1.0 does not emit, and which the schema requires. The
upstream fix is worth making, but a version bump should not block this.

The template's workflows/process_datasets was removed rather than
adapted. It implements a train/test/solution split for label projection
and referenced four API files this task has no use for; left in place it
broke config parsing against those deleted files. This task takes a
SpatialData store and benchmarks preprocessing, so there is no equivalent
split. Recorded as a gap alongside the S3 test resources.

README.md is hand-written for now: scripts/create_readme.sh renders it
through a container, and running that here is not possible, so the file
says so and will be overwritten when the generator is run.

Verified: all 21 components pass viash ns test in the new layout, and
viash round-trips the generated info.metrics blocks.

Co-Authored-By: Claude Opus 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant