Repository navigation
Validate TestNG, Spock and Cucumber JVM with paired feature controls - #253
Conversation
Both report through Surefire, so they reuse the Maven runner: TestNG's probe is the Java probe with TestNG's annotation, and Spock's is a Groovy specification in src/test/groovy that GMavenPlus compiles. Cucumber JVM, Karate, ScalaTest and MUnit keep reporting-only checks. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Surefire 3.0.0-M5 writes no report for a Spock specification dd-java-agent skips, so the skipping and disabled probes had no native results. A missing report is accepted like an empty one for those probes, and a disabled probe without native results passes when its own execution count proves the body did not run. The Java CI inventory leaves unresolved steps after the job's last test outside its scope, as discovery does, and a script whose Maven calls only read metadata, as in VERSION=$(mvn help:evaluate ...), runs no tests. Co-Authored-By: Claude Opus 5.5 <[email protected]>
A probe build ran the selected command's later phases, so a check bound after the tests, such as license headers, failed the build on the probe file itself. Probe builds now stop at the test phase; the paired runs keep the selected command. Surefire reports a TestNG data-provider invocation as method[args](n), which dd-java-agent reports by its method name. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Onboarding eval, uncoached (
Each agent followed Earlier trials found the fixes in this PR:
skroutz/elasticsearch-skroutz-greekstemmer (TestNG 6.8.8) stays INCOMPLETE, correctly, because of dd-java-agent limitations:
|
|
/datadog autotest review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b89dd57ad3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: ea154e9 | Docs | View more details | Give us feedback! |
![datadog-prod-us1-3[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
CI validation can silently omit real Maven test executions hidden in metadata substitutions or find -exec steps. Valid Spock feature names can also be shortened incorrectly, causing false telemetry mismatches.
🤖 Bits Code Review · Commit b89dd57
The probe is a feature named after the probe class, with its step in the glue package of the module's runner class, so the project's glue, hooks and object factory still load. The runner is selected with the probe feature in place of the project's; a JUnit Platform suite also excludes the root Cucumber engine, which would run the probe twice. The agent reports a feature's suite as its URI and name, and no source file, so Java matches telemetry by feature name, probe controls use the framework's own telemetry check, and test-level skipping identifies the probe by suite and name. The language guard lets Java name Cucumber, which it shares with Cucumber.js. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- The job's last test step is found per matrix row, so a test step that exists only after expansion keeps the setup before it in scope. - An unresolved step that runs Maven, as in find ... -exec mvn test, stays in scope after the last literal test step. - A Maven call with an assignment, such as MAVEN_ARGS=test, is not a metadata-only script. - JUnit signatures and TestNG invocations are normalized only for their frameworks, so a Spock feature named parse[a](1) keeps its name. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/datadog autotest review |
One build may run several frameworks, and the selected one labels the validation, so framework-based normalization left JUnit 5 parameterized tests unmatched in a Cucumber build. A native name is now compared as written when the agent reported it so, and otherwise with its JUnit signature or TestNG arguments removed; a Spock feature named parse[a](1) still keeps its name. Co-Authored-By: Claude Opus 5.5 <[email protected]>
There was a problem hiding this comment.
Mixed-framework Maven reactors can be rejected because native test names are normalized using the selected framework. Cucumber probe generation also fails for lambda glue and default-package runners, while exported Maven arguments can hide real test executions from CI scope.
🤖 Bits Code Review · Commit be44ef5
A custom @ParameterizedTest name keeps the invocation index out of the agent's telemetry, so Surefire's method[n] names went unmatched. When neither the native nor the normalized name was reported, the name without its index is compared, still by count and status. Co-Authored-By: Claude Opus 5.5 <[email protected]>
juan-fernandez
changed the title
|
Onboarding eval, uncoached (
Each agent added only the pinned action and The CurrencyExchange trials found the last two fixes:
franiglesias/flowapowa stays INCOMPLETE, correctly: its Surefire 3.0.0-M1 cannot run its Cucumber suite even without the agent ("URI has a query component"). One trial failed on truncated Maven Central downloads and was retried. |
…ckage - A MAVEN_ARGS assignment anywhere in a script, exported or not, voids the metadata-only exemption. - A module that declares only cucumber-java8 gets a lambda probe step, since that backend has no step annotations. - Glue in the default package gets no package declaration. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 50138c4b6f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cucumber probes fail for valid projects that supply glue on the Maven command line or inherit the lambda backend from a parent POM. A matrix-expanded Maven launcher can also disappear from the CI validation inventory.
🤖 Bits Code Review · Commit 50138c4
There was a problem hiding this comment.
Inherited Maven arguments can make a supposedly metadata-only script run tests that disappear from CI coverage. Valid Cucumber configurations can also generate probes that fail compilation or remain undefined, blocking local completion.
🤖 Bits Code Review · Commit 50138c4
- The probe step joins the glue a selected command sets with -Dcucumber.glue, and a suite's glue parameter in either argument order. - The Cucumber backend is read from the module's glue sources, so a cucumber-java8 dependency inherited from a parent still gets a lambda probe step. - MAVEN_ARGS inherited from the workflow, job or step voids the metadata-only exemption. - A launcher that runs Maven through a matrix value is classified once expanded, so it stays in the CI inventory. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b0b535502
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Valid Cucumber engine lists and Maven glue arguments can leave all feature controls inconclusive. Metadata-only detection also drops CI steps that inherit test goals from .mvn/maven.config, allowing validation to omit real test executions.
🤖 Bits Code Review · Commit 8b0b535
- A metadata script stays in scope when an earlier step exports MAVEN_ARGS through GITHUB_ENV or the applicable .mvn/maven.config declares goals, since Maven then runs them too. - An explicit surefire:test is the probe build's test boundary, so a later Failsafe goal does not run the probe again. - A suite that lists cucumber after other engines is a Cucumber runner. - Separated -D and --define forms of cucumber.glue place the probe step. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da5e24d1d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Maven metadata scripts can silently omit test executions when an alternate POM selects configured test goals. Cucumber suites with annotated glue can also leave feature validation inconclusive when command-line glue differs.
🤖 Bits Code Review · Commit da5e24d
Each Java lane ran every scenario serially, 18 to 23 minutes. The scenarios are now spread over three shards per lane, as the Jest lanes do. The --all flow against a CI workflow is the same for every framework, so only JUnit 5 repeats it after its paired validation; the agent workflow covers coordinated --all. Short-lived Maven JVMs skip the optimizing compiler. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- A metadata script that selects another POM stays in scope, since that POM may read another .mvn/maven.config. - The Cucumber probe goes to the first executed module that holds a runner, which an inherited dependency hides from reactor order. - A suite's glue parameter outranks the command's -Dcucumber.glue, as JUnit Platform configuration parameters do; a JUnit 4 runner's glue option still yields to it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
What
TestNG, Spock and Cucumber JVM Maven projects get paired validation like JUnit 4/5: a paired comparison without and with dd-java-agent, application coverage, the six feature controls and
COMPLETE LOCALLYthroughonboard --agent.Probes
@org.testng.annotations.Test.src/test/groovy, compiled by the project's GMavenPlus setup.@RunWith(Cucumber.class)or a JUnit Platform suite of thecucumberengine), so the project's glue, hooks and Spring context load. The runner is selected with-Dcucumber.featurespointing at the probe. A suite also excludes the root Cucumber engine, which would run the probe twice.Validation
testphase, so checks after the tests (license headers, coverage gates) never judge the probe file.VERSION=$(mvn help:evaluate …), no assignments) runs no tests.Karate, ScalaTest and MUnit keep reporting-only checks.
Why
Follow-up to the Java split (#241, #248, #252). Projects on these frameworks could never reach
COMPLETE LOCALLY. Eval trials on real projects found the remaining fixes. Reference: https://docs.datadoghq.com/tests/setup/java/Changes to shared code:
TestLevelSkippingpolicy, Java only) identifies the probe by suite and name; the intake returns test-level skippable entries without a source file. dd-java-agent reports no source file for Cucumber.probeTelemetry: an opt-in adapter hook, set by Java, that checks probe controls with the framework's own telemetry match.maven.UpToTest(Maven package).E2E testing
Prerequisites: Docker and the
maven:3.9-eclipse-temurin-17image. Build this branch for Linux (GOOS=linux CGO_ENABLED=0 go build -o ddtest .). Run commands in the container with the fixture and binary mounted, aftergit config --global --add safe.directory '*'.TestNG. Create a Git fixture:
pom.xmlwithorg.testng:testng7.10.2 and Surefire 3.5.3;src/main/java/example/Calc.java;src/test/java/example/ExampleTest.javawith a passing@Test, a failing one, and one withenabled = false.Run
ddtest testdrive --framework testng --tracer-version 1.67.1 --yes.Compatibility (this invocation): compatible, all six features passed, andLocal validation: PASSED.Spock. Same fixture shape, but:
spock-core2.3-groovy-4.0,org.apache.groovy:groovy4.0.24, andgmavenplus-plugin4.1.1 (addTestSources,compileTests);src/test/groovy/example/ExampleTest.groovy.Run with
--framework spock.Cucumber JVM. Same fixture shape, but:
cucumber-java,cucumber-junit-platform-engine7.20.1 andjunit-platform-suite1.11.4;src/test/resources/example/calc.featurewith a passing and a failing scenario, and matching step definitions inexample;@Suite @IncludeEngines("cucumber") @SelectClasspathResource("example") @ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "example") class RunCucumberTest.Run with
--framework cucumber-java.cucumber-junitand@RunWith(Cucumber.class) @CucumberOptions(features = "classpath:example", glue = "example")instead of the suite.Post-test check. Add to the TestNG fixture
license-maven-plugin4.3 with acheckexecution bound toverifyand a header the probe lacks. Run step 1 with--command 'mvn -B verify'.Agent flow. Add a GitHub workflow (setup-java 17,
mvn -B verify), then runddtest onboard --agent --framework cucumber-java, apply the action and the.gitignoreentry, and runresume_command.complete_locally.Cleanup: remove the fixture directories.
🤖 Generated with Claude Code