Skip to content

CI: unblock R 4.1 and benchmark legs - #227

Open
ms609-agent wants to merge 3 commits into
mainfrom
claude/ci-r41-benchmark
Open

ms609-agent wants to merge 3 commits into
mainfrom
claude/ci-r41-benchmark

Conversation

@ms609-agent

@ms609-agent ms609-agent commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Two CI legs have failed on main and on every PR since 23 October.

R 4.1 leg: testthat runs past the 60-minute job limit

Root cause: a base-R copy, made quadratic by CompareAll(). ape's [[.multiPhylo starts with class(x) <- NULL. On R ≤ 4.3 that assignment copies the whole list on every element access; R 4.4 and later avoid the copy. Here are 2,000 calls of a plain function(x, i) { class(x) <- NULL; x[[i]] } on a 500k-element list, without ape:

R 4.1 4.2 4.3 4.4 4.5
seconds 10.0 6.3 7.0 0.015 0.015

CompareAll() built two multiPhylo objects with one element per pair, then indexed each of them once per pair. Its time therefore grew with the square of the number of pairs. #222's 7-leaf test calls NNIDist() on all 945 trees (446,040 pairs), which took over 50 minutes on 4.1.

Fix: CompareAll() now converts x to a plain list once and indexes it with integer pair vectors. The code is no more complex than before: it builds two integer vectors instead of two pair-length tree lists, so it also uses less memory and makes fewer S3 dispatches. On GHA, NNIDist() on the 945 trees now takes 49 s on R 4.1 and 46 s on R 4.6.

The 7-leaf test also calls cpp_nni_distance() directly, because what it checks is the bounds, not the R wrapper. It now takes about 7 s instead of about 80 s. The 6-leaf case still checks the C++ results against NNIDist().

benchmark: "Set up R" times out after 15 minutes

ubuntu-latest now resolves to Ubuntu 26.04, where setup-r apt-installs about 175 packages from a slow mirror. The job's RSPM serves noble (24.04) binaries anyway, so benchmark is now pinned to ubuntu-24.04. rev-dep-check uses the same setup, so it is pinned too.

Verification

  • Local tests pass: NNI, path, tree_distance and utilities test files.
  • test-parallel.R fails locally in the same way with and without this change: its worker processes load an older globally installed TreeDist.

🤖 Generated with Claude Code

The 7-leaf NNIDist() bounds test (#222) called NNIDist() on all 945
trees: 446,040 pairs through CompareAll()'s R loop, c. 80 s locally and
over an hour on the R 4.1 leg, which then hit its 60-minute limit.  The
test now calls cpp_nni_distance() on pre-ordered edges (c. 7 s); the
6-leaf case still checks that NNIDist() agrees.

benchmark and rev-dep-check ran on ubuntu-latest, now 26.04, where
setup-r apt-installs ~175 packages and timed out at 15 minutes; their
RSPM serves noble binaries anyway, so pin ubuntu-24.04.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ms609-agent
ms609-agent requested a review from ms609 October 29, 2026 06:55
@github-actions

This comment has been minimized.

On R 4.1, ape's [[.multiPhylo (class(x) <- NULL) copies the whole list
on each access.  CompareAll() built two multiPhylo objects with one
element per pair and indexed each once per pair, so its time grew with
the square of the number of pairs: 446,040 pairs of 7-leaf trees took
over 50 minutes on the R 4.1 CI leg, against c. 80 s on R 4.6, where
the access does not copy.  It now indexes a plain list of the trees by
pair indices.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions

This comment has been minimized.

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.74%. Comparing base (2f6776d) to head (7ec6e8f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #227   +/-   ##
=======================================
  Coverage   97.74%   97.74%           
=======================================
  Files          60       60           
  Lines        6575     6577    +2     
=======================================
+ Hits         6427     6429    +2     
  Misses        148      148           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Use resolute RSPM binaries rather than noble ones on a 26.04 runner.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions

Copy link
Copy Markdown

Performance benchmark results

Call Status Change Time (ms)
ClusteringInfoDistance(tr280) ⚪ NSD 0.61% 8.81, 8.81 →
8.77, 8.75
ClusteringInfoDistance(tr50) ⚪ NSD 1.7% 6.49, 6.57 →
6.43, 6.39
LAPJV(test2000) ⚪ NSD 3.89% 131, 134 →
127, 128
LAPJV(test40) ⚪ NSD -0.53% 0.0189, 0.019 →
0.0191, 0.0191
LAPJV(test400) ⚪ NSD 0% 3.58, 3.58 →
3.59, 3.57
MutualClusteringInfo(tr280) ⚪ NSD 0.77% 18.1, 18.3 →
18.1, 18.1
MutualClusteringInfo(tr50) ⚪ NSD 2.5% 17.2, 17.4 →
17.8, 16.8
NNIMaxStep(2000, normalize = TRUE) ⚪ NSD -0.08% 0.209, 0.21 →
0.21, 0.209
NNIMaxStep(300, normalize = TRUE) ⚪ NSD -0.61% 0.0308, 0.0309 →
0.0311, 0.0309
NNIMaxStep(300) ⚪ NSD -2.24% 0.00887, 0.009 →
0.00928, 0.00893
PathDist(postTrees) ⚪ NSD -13.92% 1.64, 1.72 →
1.99, 1.65
PhylogeneticInfoDistance(tr180) ⚪ NSD -0.01% 60.8, 60.8 →
60.9, 60.8
PhylogeneticInfoDistance(tr50) ⚪ NSD 0.48% 42.4, 41.5 →
41.9, 42.2
RobinsonFoulds(tr280) ⚪ NSD 1% 4.64, 4.66 →
4.6, 4.61
RobinsonFoulds(tr280) ⚪ NSD 0.3% 4.45, 4.48 →
4.46, 4.45
RobinsonFoulds(tr50) ⚪ NSD 0.04% 4.56, 4.62 →
4.6, 4.58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant