Repository navigation
Port the spatial proteomics preprocessing benchmark onto the task structure - #1
Open
sergiomarco25 wants to merge 1 commit into
Open
sergiomarco25 wants to merge 1 commit into
sergiomarco25 wants to merge 1 commit into
Conversation
…ucture Replaces the task_template scaffold with the benchmark developed in juliusstein77/task_isp_preprocessing. Only the proteomics surface came across: the legacy transcriptomics components that repository inherited from its own iST fork - transcript assignment, cell-type annotation, expression correction - were left behind, along with four superseded or unwired components. 21 components across six namespaces: five method stages with interchangeable alternatives sharing one argument contract per stage, and six metric families wrapping sprotmetrics 0.1.0. Multi-stage namespacing follows the precedent set by task_ist_preprocessing rather than the template's single methods/ and metrics/ directories, because this task is a chain of stages rather than one prediction step. Two gaps against the OpenProblems schemas are closed here. file_score.yaml now requires dataset_id, normalization_id and method_id in uns. All three were commented out in the source repository, so scores carried no provenance and could not be attributed to a run. Every metric component now declares the info.metrics block that common/schemas/task_metric.yaml requires - 66 metrics across six families. These are generated from the sprotmetrics registry by scripts/create_component/generate_metric_info.py rather than written by hand, so the declared direction and bounds cannot drift from the code computing them; --check makes that a CI gate. Metrics with no direction are deliberately omitted: maximize is a required key and the schema cannot express "descriptive", so declaring them would hand the scoring stage a ranking criterion they do not support. They are still computed and still appear in the score file. The script also backfills references and links, which MetricInfo carries but sprotmetrics 0.1.0 does not emit, and which the schema requires. The upstream fix is worth making, but a version bump should not block this. The template's workflows/process_datasets was removed rather than adapted. It implements a train/test/solution split for label projection and referenced four API files this task has no use for; left in place it broke config parsing against those deleted files. This task takes a SpatialData store and benchmarks preprocessing, so there is no equivalent split. Recorded as a gap alongside the S3 test resources. README.md is hand-written for now: scripts/create_readme.sh renders it through a container, and running that here is not possible, so the file says so and will be overwritten when the generator is run. Verified: all 21 components pass viash ns test in the new layout, and viash round-trips the generated info.metrics blocks. Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the
task_templatescaffold with the spatial proteomics preprocessingbenchmark developed in
juliusstein77/task_isp_preprocessing.What came across
21 components in six namespaces — five method stages with interchangeable
alternatives sharing one argument contract per stage, plus six metric families:
methods_image_correctionmethods_proteomics_segmentationmethods_protein_aggregationmethods_proteomics_qcmethods_proteomics_normalizationmetrics_proteomicsMetric implementations live in
sprotmetrics0.1.0; the components here are the Viash contracts around them.Multi-stage namespacing follows the precedent in
task_ist_preprocessingrather than the template's single
methods/andmetrics/, because this task isa chain of stages rather than one prediction step.
Deliberately left behind: the legacy transcriptomics components the source
repository inherited from its own iST fork (transcript assignment, cell-type
annotation, expression correction), and four superseded or unwired components.
Schema gaps closed
Score provenance.
src/api/file_score.yamlnow requiresdataset_id,normalization_idandmethod_idinuns. All three were commented out in thesource repository, so scores carried no provenance and could not be attributed
to a run.
info.metrics. Every metric component now declares the blockcommon/schemas/task_metric.yamlrequires — 66 metrics, each with all eightrequired keys, and
viash config viewround-trips them.These are generated by
scripts/create_component/generate_metric_info.pyfromthe
sprotmetricsregistry rather than written by hand, so declared directionand bounds cannot drift from the code computing them.
--checkmakes that a CIgate.
Two deliberate behaviours there:
maximizeis required and theschema cannot express "descriptive", so declaring them would hand the scoring
stage a ranking criterion they do not support. They are still computed and
still appear in the score file.
referencesandlinksare backfilled.MetricInfocarries both fieldsbut
sprotmetrics0.1.0 does not emit them, and the schema requires both. Theupstream fix is worth making; a version bump should not block this port.
Removed:
workflows/process_datasetsThe template's version implements a train/test/solution split for label
projection and referenced four API files this task has no use for — left in
place it broke config parsing against those deleted files. This task takes a
SpatialData store and benchmarks preprocessing, so there is no equivalent split.
Recorded as a gap rather than faked.
Verification
All 21 components pass
viash ns testin the new layout:Run against
openproblems/base_python:1with the pinned dependency versions.Tests build their own synthetic fixtures, so they need no dataset and no S3.
Known gaps, also in CHANGELOG.md
run_benchmark/main.nfis still the hand-written form inherited from thesource repository — one process per stage with a
caseblock — rather thanthe
runEach/fromStateidiom. Porting it is what will supply themethod_idanddataset_idthatfile_score.yamlnow requires, so those twoare one job.
_viash.yamlpoints at the intendedlocation;
scripts/create_resources/isp/builds the input locally from thepublic MCMICRO
exemplar-001download in the meantime. That download is rawacquisition data, so the script registers the cycles with
ashlarand convertsto a SpatialData store.
README.mdis hand-written.scripts/create_readme.shrenders it througha container, which could not be run in the development environment. The file
says so and will be overwritten when the generator is run.
referencesegmentation is untestable on the public dataset: the downloadcarries no segmentation masks, those being MCMICRO output.
🤖 Generated with Claude Code