[juan-fernandez]

What Does This Do

Preserves passing, failing and ignored ScalaTest tests run through JUnit 4, with a CI Visibility event for each test.

Motivation

ScalaTest's JUnitRunner emits test callbacks without suite lifecycle callbacks. The JUnit 4 tracing listener therefore looks up an unregistered suite and throws Could not find test suite, removing passing tests from Surefire reports and adding empty Test mechanism errors. Reported during DataDog/ddtest#255.

Additional Notes

Suites start lazily on their first test or ignore event. Fallback suites use a parallel-safe lifecycle that keeps suite state out of callback-thread scopes and finishes pending suites when the enclosing runner returns. Native suite callbacks control standard JUnit 4 suites. Regression tests cover a single suite, both sequential suite orders, parallel class execution, and callbacks dispatched to a reusable executor; they verify complete suite and test events and no leaked active scope.

Verification:

• Regression fails before the change and passes after it.
• Nine JUnit 4.10/4.13, MUnit and Cucumber test variants: 293 passed, 2 skipped. Formatting and full agent build passed.
• Maven reproduction on JDK 21/Maven 3.9.9 (Scala 2.13.15, ScalaTest 3.2.19, ScalaTestPlus 3.2.19.0, Surefire 3.5.3): agent 1.68.0 produces three mechanism errors; the built agent exactly matches the baseline report: adds passes, existing failure errors, ignored skips. Maven exits unsuccessfully for the deliberate existing failure.
• Mockdog received exactly three test events with pass/fail/skip statuses, linked to one completed suite.

To reproduce with the dependency versions above:

@org.junit.runner.RunWith(classOf[org.scalatestplus.junit.JUnitRunner])
class CalcTest extends org.scalatest.funsuite.AnyFunSuite {
  test("adds") { assert(2 + 2 == 4) }
  test("existing failure") { assert(2 + 2 == 5) }
  ignore("ignored") {}
}

Run DD_CIVISIBILITY_ENABLED=true MAVEN_OPTS="-javaagent:/path/to/built-agent.jar" mvn test.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: Not provided (reported in DataDog/ddtest#255).

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T09:27:11.111558Z	e9f1539	Manual request
🔒 Security Review	✅ Completed 2026-10-30T09:28:23.343677Z	e9f1539	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 80643ba01b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 80643ba01b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-datadog-us1-prod]

When multiple ScalaTest-style suites share an outer JUnit runner, unordered cleanup can violate span activation order, leaving suite events unfinished and scopes active.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 80643ba}

[datadog-datadog-us1-prod]

Fallback suite cleanup can lose CI Visibility suite completion events when multiple ScalaTest classes share a runner or execute in parallel. The cleanup violates the suite scope's required close order or thread ownership.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 80643ba}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	60.18	58.86	$\color{red}{\blacktriangle}$ +1.32	57.69	$\color{red}{\blacktriangle}$ +2.49	80/411
agentEvpProxy	59.97	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	39.50	38.67	$\color{red}{\blacktriangle}$ +0.83	37.91	$\color{red}{\blacktriangle}$ +1.59	41/206
agentless	40.05	37.91	$\color{red}{\blacktriangle}$ +2.14	37.15	$\color{red}{\blacktriangle}$ +2.90	41/207
agentlessCodeCoverage	48.23	46.30	$\color{red}{\blacktriangle}$ +1.93	46.30	$\color{red}{\blacktriangle}$ +1.93	41/206
agentlessLineCoverage	59.41	57.69	$\color{red}{\blacktriangle}$ +1.72	56.55	$\color{red}{\blacktriangle}$ +2.86	41/206

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.18	10.13	$\color{green}{\blacktriangledown}$ -0.95	9.92	$\color{green}{\blacktriangledown}$ -0.74	40/205

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	91.68	93.23	$\color{green}{\blacktriangledown}$ -1.55	91.39	$\color{red}{\blacktriangle}$ +0.29	42/207
agentless	83.90	84.36	$\color{green}{\blacktriangledown}$ -0.46	82.69	$\color{red}{\blacktriangle}$ +1.21	41/206
agentlessCodeCoverage	99.74	99.00	$\color{red}{\blacktriangle}$ +0.74	97.04	$\color{red}{\blacktriangle}$ +2.70	41/206
agentlessLineCoverage	115.73	116.18	$\color{green}{\blacktriangledown}$ -0.45	113.88	$\color{red}{\blacktriangle}$ +1.85	40/206

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	22.74	22.54	$\color{red}{\blacktriangle}$ +0.20	22.54	$\color{red}{\blacktriangle}$ +0.20	44/242
agentless	19.38	19.59	$\color{green}{\blacktriangledown}$ -0.21	19.59	$\color{green}{\blacktriangledown}$ -0.21	42/233
agentlessCodeCoverage	19.49	20.39	$\color{green}{\blacktriangledown}$ -0.90	20.39	$\color{green}{\blacktriangledown}$ -0.90	41/233
agentlessLineCoverage	26.39	27.52	$\color{green}{\blacktriangledown}$ -1.13	27.52	$\color{green}{\blacktriangledown}$ -1.13	25/157

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-datadog-us1-prod]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.58%

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e9f1539 | Docs | Give us feedback!}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.21	10.33	$\color{green}{\blacktriangledown}$ -1.12	10.54	$\color{green}{\blacktriangledown}$ -1.33	42/209
agentless	10.28	10.33	$\color{green}{\blacktriangledown}$ -0.05	10.33	$\color{green}{\blacktriangledown}$ -0.05	42/207
agentlessCodeCoverage	12.03	17.73	$\color{green}{\blacktriangledown}$ -5.70	17.73	$\color{green}{\blacktriangledown}$ -5.70	38/188

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	14.26	13.94	$\color{red}{\blacktriangle}$ +0.32	13.67	$\color{red}{\blacktriangle}$ +0.59	41/218
agentless	13.43	12.37	$\color{red}{\blacktriangle}$ +1.06	12.37	$\color{red}{\blacktriangle}$ +1.06	41/218
agentlessCodeCoverage	16.19	16.04	$\color{red}{\blacktriangle}$ +0.15	15.72	$\color{red}{\blacktriangle}$ +0.47	41/218
agentlessLineCoverage	18.27	18.08	$\color{red}{\blacktriangle}$ +0.19	17.73	$\color{red}{\blacktriangle}$ +0.54	40/217

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	96.17	97.04	$\color{green}{\blacktriangledown}$ -0.87	97.04	$\color{green}{\blacktriangledown}$ -0.87	45/226
agentless	91.47	91.39	$\color{red}{\blacktriangle}$ +0.08	91.39	$\color{red}{\blacktriangle}$ +0.08	44/224
agentlessCodeCoverage	101.30	101.00	$\color{red}{\blacktriangle}$ +0.30	101.00	$\color{red}{\blacktriangle}$ +0.30	45/225
agentlessLineCoverage	102.41	103.04	$\color{green}{\blacktriangledown}$ -0.63	103.04	$\color{green}{\blacktriangledown}$ -0.63	55/252

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	19.22	19.20	$\color{red}{\blacktriangle}$ +0.02	19.20	$\color{red}{\blacktriangle}$ +0.02	44/224
agentless	18.91	18.82	$\color{red}{\blacktriangle}$ +0.09	18.82	$\color{red}{\blacktriangle}$ +0.09	44/226
agentlessCodeCoverage	21.53	21.65	$\color{green}{\blacktriangledown}$ -0.12	22.09	$\color{green}{\blacktriangledown}$ -0.56	44/230
agentlessLineCoverage	37.94	37.91	$\color{red}{\blacktriangle}$ +0.03	38.67	$\color{green}{\blacktriangledown}$ -0.73	42/221

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.84 s	14.68 s	[+0.4%; +1.7%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.59 s	13.65 s	[-1.2%; +0.3%] (no difference)
startup:petclinic:appsec:Agent	17.71 s	17.53 s	[+0.2%; +1.9%] (maybe worse)
startup:petclinic:iast:Agent	17.60 s	17.61 s	[-0.9%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	17.22 s	17.32 s	[-1.7%; +0.5%] (no difference)
startup:petclinic:sca:Agent	17.59 s	17.56 s	[-0.7%; +1.1%] (no difference)
startup:petclinic:tracing:Agent	16.59 s	16.77 s	[-2.1%; -0.1%] (maybe better)

Commit: e9f1539d · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.49	16.36	$\color{red}{\blacktriangle}$ +0.13	16.36	$\color{red}{\blacktriangle}$ +0.13	42/219
agentless	10.42	9.73	$\color{red}{\blacktriangle}$ +0.69	9.92	$\color{red}{\blacktriangle}$ +0.50	42/219
agentlessCodeCoverage	13.91	13.67	$\color{red}{\blacktriangle}$ +0.24	13.67	$\color{red}{\blacktriangle}$ +0.24	42/221
agentlessLineCoverage	22.34	22.54	$\color{green}{\blacktriangledown}$ -0.20	22.54	$\color{green}{\blacktriangledown}$ -0.20	41/218

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	23.46	10.97	$\color{red}{\blacktriangle}$ +12.49	13.94	$\color{red}{\blacktriangle}$ +9.52	42/219
agentless	22.62	13.40	$\color{red}{\blacktriangle}$ +9.22	13.94	$\color{red}{\blacktriangle}$ +8.68	42/219
agentlessCodeCoverage	138.30	87.80	$\color{red}{\blacktriangle}$ +50.50	84.36	$\color{red}{\blacktriangle}$ +53.94	42/219
agentlessLineCoverage	149.12	120.92	$\color{red}{\blacktriangle}$ +28.20	123.36	$\color{red}{\blacktriangle}$ +25.76	41/218

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 73cdd95118

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[datadog-datadog-us1-prod]

More details

Per-runner suite tracking and reverse-order cleanup address the nested-runner ownership and wrong-thread scope failures discussed on the PR.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 73cdd95}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73cdd95118

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: e9f1539daa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: e9f1539daa

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[datadog-datadog-us1-prod]

More details

Fallback suites avoid activating suite scopes on callback threads, allowing the enclosing runner to finish them without violating thread-bound scope ownership. Confidence is limited to static inspection.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit e9f1539}