Repository navigation
Add CircleCI as a CI provider for JavaScript and Ruby - #254
Conversation
|
⏳ PR checks are still running. This summary will update when all results arrive. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: 60ea844 | Docs | View more details | Give us feedback! |
|
/datadog autotest review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5460bc253c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
CircleCI onboarding can produce a configuration that CircleCI rejects when the input uses version 2. Validation can also omit test commands or certify instrumentation that conditional exports and non-Bash test steps do not load.
🤖 Bits Code Review · Commit 5460bc2
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1fac4f99fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
CircleCI setup can be inserted before a test step using sh, leaving tests uninstrumented. Runtime tracking can also check the wrong version, while failure normalization can hide differences between baseline and instrumented failures.
🤖 Bits Code Review · Commit 1fac4f9
There was a problem hiding this comment.
CircleCI validation can skip valid suites, trust stale runtime and preload state, or select different tests through command rewriting. Failure normalization also hides legitimate hexadecimal assertion differences.
🤖 Bits Code Review · Commit 1fac4f9
175b64d to
65e1d33
Compare
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65e1d3379e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
CircleCI discovery can silently omit repeated job invocations and tests in orb hooks. Shell-state handling can also validate different suites or Ruby options from CI and accept an installer whose curl dependency is conditional.
🤖 Bits Code Review · Commit 65e1d33
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17b4d76eee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Inline Ruby exports can hide a removed Datadog preload or replay different environment values from CI. CircleCI setup also skips supported jobs that install a newer Node runtime over an unsupported image runtime.
🤖 Bits Code Review · Commit 17b4d76
ddtest now reads .circleci/config.yml into the shared workflow model (executors, reusable commands, parameters, inline orbs, matrix, when/unless, $BASH_ENV, test splitting) and sets up Test Optimization with Datadog's Test Optimization orb. JavaScript setup is applied by onboard --apply; Ruby setup is applied by the coding agent from onboard --agent. Python and Java answer ci_provider_unsupported on CircleCI. The shell-state follower, image checks and setup-note wording that GitLab CI introduced are now shared by both providers. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Read circleci/node, circleci/ruby and circleci/browser-tools orbs from models of their source, provided through a language hook (ciComponents): installs are preparation, install commands select the runtime, node/test, node/run and ruby/rspec-test run the commands they run, rubocop-check only lints. Other commands and releases stay unresolved. - Treat diagnostic prints (npm --version) and type checkers (tsd) before the tests as outside the test path on every provider. - Compare test failures without object addresses, so the same failure in every run is not reported as varying outcomes. - Report Qlty's coverage upload orb as a non-test orb. - Report a rake RUBYOPT=... argument that replaces the Datadog preload. - Move version 2 configurations to 2.1 when applying the orb, and report orbs declared in a version 2 configuration. - Follow override-with, anchor matches conditions, type condition values, substitute plain $BASH_ENV assignments into later commands, reject compound test-splitting commands, keep $BASH_ENV away from sh steps and leave $BASH_ENV changes under undecided conditions unresolved. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…er choice, tracer - Report tests whose names only one run reports as unstable_test_identities, with the offending examples and the static description fix, instead of varying outcomes. - Resolve Ruby test steps that export literal variables, run exactly one test command (a rake RUBYOPT="$RUBYOPT ..." argument joins the preload) and then only move, copy or print artifacts. - Don't count documentation tooling run through bundle exec (yard, rdoc) as possible JavaScript tests, so workflows that only build docs don't make a provider a candidate. - When the latest tracer can't run on every instrumented CI row and no release passes the framework's support check, select the newest release that runs on every row, say so, and report its support gap. - Add the ksylvest/omniai fixture. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ll checks Discovery's variant expansion rejected a literal export and a $RUBYOPT read, so brakeman's fixed test-3-2 step (export, then rake RUBYOPT="$RUBYOPT ...") still needed separate CI setup. Reads of the variables the installer exports stay as written for the language's resolver, and literal exports pass through. Setup also says which tracer the CI rows' recommendation selected. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…nd runtimes - Normalize object addresses only inside object representations (Ruby #<Foo:0x...>, Python <x object at 0x...>, Java pkg.Class@hash), so differing bare hexadecimal values still differ. - Substitute $BASH_ENV shell variables only where Bash expands them, reading the shell syntax tree: never inside single quotes or escapes, escaped inside double quotes, unresolved with expansion operators. - Follow $BASH_ENV writes in steps that also run other commands, and make later steps unresolved after a write ddtest cannot follow. - Check later tests against the Node a modeled node/install selects. - Leave the runtime unknown when a modeled installer runs under a condition ddtest cannot decide. - Refuse to insert the orb before a test step that runs with /bin/sh. - Extract the split-test command with the shell parser, keeping quoted arguments as written. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…inline rake RUBYOPT
- An unstable_test_identities verdict lists each affected example once,
whatever names the runs gave it, up to 50, and counts the rest.
- A gemspec whose files are an allowlist of specific paths
(Dir.glob("{bin,lib,exe}/**/*") + %w[README.md Gemfile]) can't include
.testoptimization/: the guidance says not to edit it.
- When a CI test command passes RUBYOPT as a rake argument, the Ruby
handoff prescribes RUBYOPT="$RUBYOPT <options>" for it on every
provider.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…idation - Leave reads of a variable the step itself assigns earlier as written, instead of substituting the value an earlier step put in $BASH_ENV; read/unset/for on such variables stay unresolved. - Diagnose unstable test identities only when the one-off names are one example's variants (after stripping generated values and framework truncation); disjoint stable names stay varying outcomes. The report keeps every listed example instead of 10. - Keep both invocations when a workflow runs one job twice under the same name, and report the clash CircleCI rejects. - A rake RUBYOPT="$RUBYOPT ..." argument keeps the options the step inherits and adds its own. - Make a step unresolved when it sets a variable that text appended to $BASH_ENV reads, since Bash re-evaluates that text in each step. - Count a curl install toward the orb's downloader check only when it certainly runs. - Read workflow hooks on a non-test orb's job as the configuration's steps. - After a retried run completes, name the tests earlier attempts saw vary as a caveat in the final response. - Classify jobs that never run the framework before recording their preparation, so a build-only job is a caveat, not a blocker. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Leave Ruby test steps unresolved when they export RUBYOPT or a Datadog setting, or append to a variable; decode escapes in exported values. - Treat commands after the test as artifact handling only when they touch report, coverage or test-result paths; copying other files may prepare a later test. - Substitute $BASH_ENV variables only after top-level, unconditional assignments of the step; an assignment in a subshell, function or condition leaves the step unresolved. - Select CircleCI setup targets on the Node a modeled installer selects before the tests, not only the image's. Co-Authored-By: Claude Opus 5.5 <[email protected]>
17b4d76 to
60ea844
Compare
What
Adds CircleCI as a CI provider for JavaScript and Ruby, on the same seam as GitLab (#249).
.circleci/config.yml: executors, commands and parameters, matrix jobs, conditional steps and workflows,$BASH_ENVexports, and the commoncircleci tests split/runforms. Unknown orb commands, dynamic config (setup: true) and anything else it can't resolve are reported, never dropped. Notification, scanning and coverage orbs (slack, jira, snyk, sonarcloud, codecov, coveralls) are caveats.datadog/test-optimization-circleci-orb@2).onboard --applyadds the orb declaration and theautoinstrumentstep before the tests.ci_setup_required.DD_API_KEYisn't set, and the instructions and human action say so, including forked-PR builds and restricted contexts.@2or an exact@2.x.y, with the language inlanguages, an exact tracer version and the selected site.$BASH_ENV), slim images without curl, or unresolved images are reported, not set up. Ruby versions come from images such ascimg/ruby:3.3.ci_provider_unsupportedon CircleCI.Why
Customers whose tests run on CircleCI couldn't onboard with ddtest. Using Datadog's orb keeps CI setup on the supported installer the GitHub action and the GitLab script also use.
E2E testing
Prerequisites:
ddtest;Copy each fixture into a fresh git repository (
git init -q && git add -A), and keep the same shell, includingTMPDIR, between a run and its--report.internal/onboard/testdata/circleci-javascript(based on marp-vscode's config), then runnpm install.ddtest onboard --agent. Expected:ready.ddtest onboard --framework jest --apply --yes --tracer-version 5.133.0 --format json. Expected:complete_locally..circleci/config.ymlgets the orb declaration and anautoinstrumentstep before the tests, and.gitignoregets/.testoptimization/.--applyagain without--yes. Expected: "Jest CI setup is already configured; no edits."ddtest testdrive --all --agent --yes, thenddtest testdrive --report. Expected:complete_locallyand "Executed Jest validations: PASSED (1/1)";DD_API_KEYin CircleCI, saying the job fails without it;node:22-alpineand run the--apply --yescommand again. Expected:ci_edit_required, saying the image has no Bash so the orb step can't run.internal/onboard/testdata/circleci-ruby(based on rubyXL's config), then runbundle install.ddtest onboard --agent. Expected:ci_setup_required, with orb instructions and the RubyGems guard for the Ruby 3.0 entry.autoinstrumentstep (languages: ruby,ruby_tracer_version: 1.42.0), and/.testoptimization/to.gitignore. Then runddtest onboard --agenttwice. Expected:readyboth times.ddtest testdrive --framework rspec --all --agent --yes, then--report. Expected:complete_locallyand "PASSED (1/1)".ruby_tracer_version: latest. Expected:onboard --agentreturnsci_setup_required. Declaring the orb at@1instead makes check-only report "not a supported release".ddtest onboard --agent. Expected:ci_provider_required,required_input: [github, gitlab, circleci].--ci-provider circleci. Expected:ready..circleci/config.yml, runddtest onboard --agent. Expected:ci_provider_unsupported.Cleanup: delete the scratch repositories.
🤖 Generated with Claude Code