[juan-fernandez]

What

ddtest testdrive now runs paired validation (baseline vs reporting-only, plus the six feature probes) for Karate, ScalaTest and Scala MUnit on Maven. Before this PR they got reporting-only checks.

• Karate: native results come from Karate's target/karate-reports/*.karate-json.txt. Surefire's copies of Karate scenarios are unreliable. The probe is a feature run through the module's Karate runner with -Dkarate.options, selecting one runner method; the runner's own JUnit result is left out.
• ScalaTest: native results come from the scalatest-maven-plugin's target/scalatest-reports, and its goal counts as a test goal. The probe is an AnyFunSuite, selected with -Dsuites and -Dtest. It declares the module's JUnitRunner when Surefire runs the suites. dd-java-agent 1.67.1 and 1.68.0 currently fail every test of such a suite, and validation reports that as a suspected regression.
• MUnit: the probe is a munit.FunSuite. Skipping and disabled are reported as not supported: dd-java-agent skips JUnit 4 tests through ParentRunner, and MUnitRunner doesn't extend it.

Shared changes:

• maven.ReadReports reads the two new report formats.
• maven.IncompleteBuild accepts Surefire 3.5's failure line for a build whose failed tests are all errors. Before, such a build was treated as incomplete.

Why

Customers using these frameworks could not get a COMPLETE local verdict: features were never exercised. This finishes Maven framework coverage after #253.

E2E testing

Prerequisites: Docker, or a host with JDK 17+, Maven 3.9 and git. Build ddtest from this branch with make build. Run every command below with network access to Maven Central.

fixtures.sh creates three fixture projects under ./e2e

#!/bin/sh
# Creates the three fixtures under ./e2e: karate, scalatest, munit.
set -e
mkdir -p e2e && cd e2e
calc='package example;
public class Calc { public static int add(int a, int b) { return a + b; } }'
pom() { # artifact, dependencies, plugins
cat <<POM
<project><modelVersion>4.0.0</modelVersion><groupId>example</groupId><artifactId>$1</artifactId><version>1.0</version>
<properties><maven.compiler.release>17</maven.compiler.release><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding></properties>
<dependencies>$2</dependencies><build><plugins>$3</plugins></build></project>
POM
}
scala='<dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>2.13.15</version></dependency>'
scalac='<plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>4.9.2</version><executions><execution><goals><goal>testCompile</goal></goals></execution></executions></plugin>'
for p in karate scalatest munit; do mkdir -p $p/src/main/java/example && printf '%s\n' "$calc" > $p/src/main/java/example/Calc.java && (cd $p && git init -q); done

pom karate '<dependency><groupId>io.karatelabs</groupId><artifactId>karate-junit5</artifactId><version>1.5.2</version><scope>test</scope></dependency>' \
  '<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>3.2.5</version></plugin>' > karate/pom.xml
mkdir -p karate/src/test/java/example karate/src/test/resources/example
cat > karate/src/test/java/example/KarateTest.java <<'J'
package example;
class KarateTest {
    @org.junit.jupiter.api.Test void all() {
        com.intuit.karate.Results results = com.intuit.karate.Runner.path("classpath:example").parallel(2);
        org.junit.jupiter.api.Assertions.assertEquals(0, results.getFailCount(), results.getErrorMessages());
    }
}
J
cat > karate/src/test/resources/example/calc.feature <<'F'
Feature: Calculator
  Scenario: adds
    * match Java.type('example.Calc').add(2, 2) == 4

  Scenario: existing failure
    * match Java.type('example.Calc').add(2, 2) == 5
F

pom scalatest "$scala"'<dependency><groupId>org.scalatest</groupId><artifactId>scalatest_2.13</artifactId><version>3.2.19</version><scope>test</scope></dependency>' \
  "$scalac"'<plugin><groupId>org.scalatest</groupId><artifactId>scalatest-maven-plugin</artifactId><version>2.2.0</version><configuration><junitxml>.</junitxml></configuration><executions><execution><id>test</id><goals><goal>test</goal></goals></execution></executions></plugin>' > scalatest/pom.xml
mkdir -p scalatest/src/test/scala/example
cat > scalatest/src/test/scala/example/CalcTest.scala <<'S'
package example
class CalcTest extends org.scalatest.funsuite.AnyFunSuite {
  test("adds") { assert(Calc.add(2, 2) == 4) }
  test("existing failure") { assert(Calc.add(2, 2) == 5) }
}
S

pom munit "$scala"'<dependency><groupId>org.scalameta</groupId><artifactId>munit_2.13</artifactId><version>1.0.4</version><scope>test</scope></dependency>' "$scalac" > munit/pom.xml
mkdir -p munit/src/test/scala/example
cat > munit/src/test/scala/example/CalcTest.scala <<'S'
package example
class CalcTest extends munit.FunSuite {
  test("adds") { assertEquals(Calc.add(2, 2), 4) }
  test("existing failure") { assertEquals(Calc.add(2, 2), 5) }
}
S
for p in karate scalatest munit; do (cd $p && git add -A); done
echo "fixtures in $(pwd)"

1. Setup: run sh fixtures.sh.
2. Karate: run cd e2e/karate && ddtest testdrive --framework karate --yes.
   • Expect Compatibility (this invocation): compatible.
   • Expect all six Feature …: passed lines and Local validation: PASSED, with exit 0.
   • The baseline counts the existing scenario failure plus the runner test's failure.
3. ScalaTest: run cd e2e/scalatest && ddtest testdrive --framework scalatest --yes. Expect the same result: compatible, six features passed, Local validation: PASSED.
4. MUnit: run cd e2e/munit && ddtest testdrive --framework munit --yes.
   • Expect compatible, with auto-retries, early-flake-detection, quarantine and attempt-to-fix passed.
   • Expect Feature skipping: not supported and Feature disabled: not supported, naming MUnit's runner.
   • Expect Local validation: PASSED.
5. Karate without a runner:
   • In e2e/karate, replace src/test/java/example/KarateTest.java with a plain JUnit 5 test of Calc, then re-run step 2's command.
   • Expect compatible, every feature inconclusive with "No Karate runner class … in the test sources of the root module", and Local validation: INCOMPLETE with exit 1.
6. Isolation: after each run, git status in the fixture shows no target/ and no ddtest probe files. Only .testoptimization/ is added.
7. Cleanup: rm -rf e2e.

🤖 Generated with Claude Code

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-29T13:19:41.090831Z	f76d6a0	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0fddf15858

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

   

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 92.75%
• Overall Coverage: 88.33% (+0.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: feeca0d | Docs | View more details | Give us feedback!}

[datadog-official]

Karate’s report parser can treat different multiline assertion failures as identical and incorrectly report compatibility. Valid ScalaTest/Surefire and Karate runner configurations can also leave all feature checks inconclusive.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 0fddf15}

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d9be9f989d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

ScalaTest feature validation cannot complete for reactors with multiple ScalaTest modules or projects using modular dependencies without AnyFunSuite. The probe selection applies across the reactor, while its generated source assumes a suite dependency that those projects may lack.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit d9be9f9}

[juan-fernandez]

/datadog autotest review
@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f76d6a0713

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

ScalaTest commands that already specify -Dsuites retain that selector during probe runs, causing existing suites to run alongside the probe and making all feature checks inconclusive.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit f76d6a0}

[datadog-official]

Explicit ScalaTest suite filters and Karate helper classes can prevent isolated probe execution, leaving feature validation inconclusive for otherwise runnable projects.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit f76d6a0}