[MNT] Add Missing Assertions to Run Function Tests

[Akarshkushwaha]

Metadata

• Reference Issue: Fixes [MNT] Add Missing Assertions to Run Function Tests #1646
• New Tests Added: No
• Documentation Updated: No
• Change Log Entry: Added assertions to [tests/test_runs/test_run_functions.py] to improve test coverage.

Details

• What does this PR implement/fix? Explain your changes.
  This PR adds missing assertions to [tests/test_runs/test_run_functions.py] where TODO comments indicated gaps in verification.

  Specific changes:

  • Line 170 (_rerun_model_and_compare_predictions]: Added assertion to verify task.task_type_id matches supervised classification or regression.
  • Line 287 (_perform_run]: Added assertions ensuring run.run_id and run.flow_id are set correctly. Note: run.uploader is skipped as it's not populated on the local object immediately after publish.
  • Line 291 (_perform_run]: Added assertions verifying run.trace properties (id match, non-empty iterations).
  • Line 341: Added checks for downloaded.uploader, downloaded.task_id, and downloaded.flow_id to ensure integrity of the fetched run.
  • Line 514: Added assertion run.output_files["predictions"] is not None.

• Why is this change necessary? What is the problem it solves?
  The tests contained several TODO placeholders for missing validations. These assertions make the test suite more robust by verifying that runs are correctly constructed, uploaded, and retrievable with expected properties.

• How can I reproduce the issue this PR is solving and its solution?
  Run the updated tests:

  pytest tests/test_runs/test_run_functions.py

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.69%. Comparing base (4a25027) to head (85c81a3).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1658      +/-   ##
==========================================
- Coverage   81.50%   80.69%   -0.81%     
==========================================
  Files          63       63              
  Lines        5238     5238              
==========================================
- Hits         4269     4227      -42     
- Misses        969     1011      +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[AtharvaManale]

Just cross check the reviews started beside the specific changes in the tests/test_runs/test_run_functions.py

[AtharvaManale]

Holdout task isn't specific for SUPERVISED_CLASSIFICATION tasks, it is an estimation_procedure type.
For ex:
task.estimation_procedure = { "type": "holdout", "parameters": { "number_repeats": "1", "number_folds": "", "percentage": "33", "stratified_sampling": "true" }, "data_splits_url": "https://test.openml.org/..." } as seen in openml/tasks/task.py.

so you have to check about type of estimation procedure rather than task type.

[PGijsbers]

I am getting some unexpected responses from seemingly sane assertions, such as checking run.trace.run_id. The current state of tests is a bit flaky which makes it a bit harder to dig in. I am going to post pone work on this PR until next week, when I have hopefully been able to address some of the concerns that prohibit me from properly addressing the underlying issues exposed by this PR.

[AtharvaManale]

@PGijsbers , @Akarshkushwaha, I would like to suggest some changesas follows:

As holdout set/task is type of an estimation_procedure of an task so we should validate that.
line.no: 227, function(_rerun_model_and_compare_predictions)
# assert for holdout task task = openml.tasks.get_task(run.task_id) assert task.estimation_procedure["type"] == "holdout"

As for a legal race_iteration exactly one of setup_string or parameters must be provided. mentioned in openml/trace.py. So check for valid race_iteration.
line.no: 350, function( _perform_run)
# assert for trace_iteration assert run.trace.run_id == run.run_id assert run.trace.trace_iterations is not None for trace_iteration in run.trace.trace_iterations.values(): assert (trace_iteration.setup_string is not None) != (trace_iteration.parameters is not None)

As for comparing the downloaded_run_trace with existing run.trace to check whether downloaded run carries the trace as objects.
line.no: 405, function( _perform_run)
downloaded_run_trace = downloaded.trace assert downloaded_run_trace is not None assert run.trace is not None self.assertEqual(run.trace, downloaded_run_trace)

All changes are for tests/test_runs/test_run_functions.py
You can review these changes and use them as needed.
reference #1758