Repository navigation
Add Buildkite as a CI provider for JavaScript and Ruby - #256
Merged
Merged
Conversation
Contributor
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 2a7526c | Docs | View more details | Give us feedback! |
juan-fernandez
force-pushed
the
juan-fernandez/buildkite-provider
branch
from
October 29, 2026 12:22
f52250a to
a2b409a
Compare
Contributor
Author
|
/datadog autotest review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2b409a428
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-prod-us1-3[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Buildkite discovery can silently omit tests inside installer statements or annotation arguments, allowing incomplete CI coverage to appear ready. Container configuration resolution and static pipeline upload handling also produce incorrect validation results.
🤖 Bits Code Review · Commit a2b409a
juan-fernandez
force-pushed
the
juan-fernandez/circleci-provider
branch
from
October 29, 2026 13:28
17b4d76 to
60ea844
Compare
juan-fernandez
force-pushed
the
juan-fernandez/buildkite-provider
branch
from
October 29, 2026 13:28
2c3a776 to
868ebb5
Compare
ddtest now reads every Buildkite pipeline file (buildkite.yml and the YAML under .buildkite/ that defines steps) into the shared workflow model: command steps per matrix combination, statements of a step's commands sharing one shell, pipeline and step env, groups, if/skip/branches, the docker and docker-compose plugins, the image attribute and pipeline upload's interpolation of $NAME. Triggers, generated pipeline uploads, unknown plugins and Windows agents stay unresolved blockers; plugins that only fetch secrets, artifacts or caches or annotate results are caveats. There is no Datadog Buildkite plugin, so the setup is a snippet in the test step itself that downloads Datadog's installer (install_test_visibility_v15, pinned with its SHA256), runs it and exports what it prints, guarded on DD_API_KEY. JavaScript setup is applied by onboard --apply (including DD_API_KEY in the docker plugin's environment); Ruby setup by the coding agent from onboard --agent. Python and Java answer ci_provider_unsupported on Buildkite. Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Treat DD_API_KEY= (and an empty Compose value) as an empty key, not a
passthrough: only a bare name or a Compose ${DD_API_KEY} reference
forwards the agent's key.
- Resolve pipeline upload paths from the step's working directory, accept
statically quoted paths, and resolve an argument-free upload to the one
default file Buildkite finds (buildkite.yml, .buildkite/pipeline.yml,
buildkite/pipeline.yml and their .yaml/.json forms); read
buildkite/pipeline.* as pipelines too.
- Keep buildkite-agent record commands with command substitutions in scope.
- Keep a statement running the installer in an unrecognized form as an
unresolved command instead of a setup step that hides it.
- Decide Windows agents per matrix combination.
- Resolve Compose image variables from the step's env, leaving variables
only the agent or a .env file supplies unknown; expand matrix values in
Compose file paths.
- Exempt only build-level BUILDKITE_* variables from upload-time
interpolation; agent and job variables differ between upload and test.
- Bound a matrix's combinations before building them.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
juan-fernandez
force-pushed
the
juan-fernandez/buildkite-provider
branch
from
October 29, 2026 13:39
868ebb5 to
e651771
Compare
Contributor
Author
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e651771fd3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Buildkite discovery can silently omit pipelines whose filenames contain “compose” and validate containerized tests using the wrong environment or working directory. These cases can produce successful onboarding without validating the tests CI actually runs.
🤖 Bits Code Review · Commit e651771
- Recognize Docker Compose files by their top-level keys (services or version without steps), so pipeline files whose name mentions compose are read. - Leave a step unresolved when a Compose service, or the docker or docker-compose plugin, sets an entrypoint other than an emptied one or a known passthrough (tini --, dumb-init --, docker-entrypoint.sh, env). - Report a setup after an earlier command that clears (export DD_API_KEY=, unset) or writes DD_API_KEY, and synthesize no exports for it. - Give a container step the environment its container receives: the plugin's environment entries (NAME=value as written, bare names with the pipeline's value), the Compose service's environment, and the pipeline's env only with propagate-environment. - Map a Compose service's working_dir back into the checkout through the volume mounting it; one outside the checkout is unresolved. - Accept propagate-environment as passing DD_API_KEY, with a note that it carries only the pipeline's environment (such as the Buildkite UI's), never a key an agent hook or secrets plugin exports. - Decide Windows agents from the effective agents, the step's keys overriding the pipeline's. Co-Authored-By: Claude Opus 5.5 <[email protected]>
juan-fernandez
marked this pull request as ready for review
October 29, 2026 14:40
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c250f4626
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Buildkite Compose steps can validate the wrong package or reject a valid Datadog setup. Pipeline-level image interpolation can also approve an image that remains unresolved after upload.
🤖 Bits Code Review · Commit 8c250f4
Uncoached trials on freshshell/fresh and buildkite/pipeline-schema showed three gaps when a docker-compose step's service has build: instead of image:. - ddtest now reads the Dockerfile (build context, dockerfile, dockerfile_inline, args, target) and uses the image its final stage, or target, builds FROM, following stage aliases and build arguments with a literal default. That image feeds the existing static runtime rules (ruby:3.4.7-slim-trixie selects Ruby 3.4.7, node:18 selects Node 18). A FROM ddtest cannot resolve leaves the image unknown, and the Ruby and Node runtime findings name that cause instead of asking for a static Ruby image the job may already have. Without a Compose working_dir, the commands start in the Dockerfile's last WORKDIR, mapped back into the checkout through the mounted directory (now read relative to the Compose file, as Compose does) or, without one, the Dockerfile's COPY . of the build context. - The setup check applies the image rules (no Bash on Alpine, BusyBox or distroless; no curl or wget on official slim images) to that image, unless a RUN instruction installs bash, or curl or wget. The blocker names the Dockerfile and the fix, and the setup instructions list such steps before the snippet is written. - The release manual CI edits pin follows every known CI Node, including steps ddtest could not resolve, and is written in the provider's syntax (DD_SET_TRACER_VERSION_JS="5.133.0" on node:18) and kept when unresolved scope sends the agent to the manual instructions. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review findings on what upload interpolation and the docker-compose
plugin actually give a step:
- Agent targeting is read after upload interpolation, so
queue: "$TARGET_QUEUE" with a known windows value selects Windows
agents, and a target reading a value ddtest cannot know leaves the step
unresolved instead of assuming a POSIX shell.
- ${NAME:-default} and ${NAME-default} follow upload's rules for known
variables (an empty value takes the :- default, - keeps it); an unknown
variable stays unknown.
- The pipeline image is interpolated once, so image: $$NODE_IMAGE stays the
literal $NODE_IMAGE, an image ddtest cannot resolve, and an unresolved
image no longer reaches the runtime rules.
- A docker-compose plugin that only builds or pushes runs no container:
the step's commands run on the agent host, so apply no longer adds a
DD_API_KEY environment list the re-check would then ignore.
- A plugin that lists DD_API_KEY decides what the container receives over
the Compose service's value, in the mapping and list forms; a key
written into the Compose file is still reported, since it is committed.
- Compose paths (bind mounts, the build context, the Dockerfile) are
relative to the Compose file's directory, covered for ci/ Compose files.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
gnufede
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Buildkite as a CI provider for JavaScript and Ruby.
Discovery. ddtest reads every Buildkite pipeline file (
.buildkite/*.yml,buildkite.yml, …), including matrices, groups,ifconditions, step and pipelineenv, YAML anchors, and the Docker and Docker Compose plugins. These are reported, never dropped:buildkite-agent pipeline uploadof generated or unread pipelines);Artifact, cache, annotation and secrets plugins are caveats.
Setup. There is no official Datadog Buildkite plugin, so ddtest uses the same Datadog installer the GitLab script and the CircleCI orb wrap (
install_test_visibility_v15.sh, pinned with its SHA256).DD_API_KEY, so tests run untraced without the key.onboard --applyinserts it, addingDD_API_KEYto a Docker plugin's environment list.ci_setup_required.API key.
DD_API_KEYcomes from an agent environment hook, a secrets plugin, or pipeline env; a literal key in the pipeline is refused.Complete setup. One shared check: the exact recognized snippet runs before the test command in the same step, with the language, an exact tracer version and the selected site.
Runtime. It comes from the Docker or Compose image, or from the final
FROMof the Dockerfile a Compose service builds, with the same Bash/curl rules as GitLab and CircleCI (a Dockerfile that installs curl counts). The dd-trace version is chosen for that Node. Steps that run on the agent host reportruntime_unknown.Other languages. Python and Java answer
ci_provider_unsupportedon Buildkite.Other providers. GitHub, GitLab and CircleCI output is unchanged.
Why
Customers whose tests run on Buildkite couldn't onboard with ddtest. Using Datadog's installer keeps setup on the same supported install path as the GitHub action, the GitLab script and the CircleCI orb.
E2E testing
Prerequisites:
ddtest;Copy each fixture into a fresh git repository (
git init -q && git add -A), and keep the same shell between a run and its--report.internal/onboard/testdata/buildkite-ruby(based on buildkite/rspec-junit-example), then runbundle install.ddtest onboard --agent. Expected:ci_setup_required, with the setup snippet and the RubyGems guard for the Ruby 3.0 entry.bundle exec rspecin the test step,DD_API_KEYto the Docker plugin'senvironment, and/.testoptimization/to.gitignore. Then runddtest onboard --agenttwice. Expected:readyboth times.ddtest testdrive --framework rspec --all --agent --yes, then--report. Expected:complete_locallyand "Executed RSpec validations: PASSED (1/1)".latest. Expected:onboard --agentreturnsci_setup_required.internal/onboard/testdata/buildkite-javascript(based on eslint-plugin-heap's pipeline), then runnpm install.ddtest onboard --framework jest --apply --tracer-version 5.133.0. Expected: a preview that inserts the snippet with itsDD_API_KEYguard ("DD_API_KEY is not set; Datadog Test Optimization is disabled for this step.") and addsDD_API_KEYto the Docker plugin environment.--yes, then the same command without--yes. Expected: "Jest CI setup is already configured; no edits."ddtest testdrive --all --agent --yes, thenddtest testdrive --report. Expected:complete_locallyand "Executed Jest validations: PASSED (1/1)". The build-only:typescript: Buildstep is reported as outside the test path../steps.sh | buildkite-agent pipeline upload, runddtest onboard --framework jest --apply --yes --format json. Expected:preflight_unresolved, with no files written.bundle exec rspecthroughdocker-compose#v4.16.0(run: app). Add adocker-compose.ymlwhoseappservice hasbuild: .and mounts.:/app, and aDockerfilewithFROM ruby:3.3-slimandWORKDIR /app.ddtest onboard --agent. Expected:ci_setup_required, and the instructions say the image built from Dockerfile (FROM ruby:3.3-slim) has neither curl nor wget, with the fix to install curl.RUN apt-get update && apt-get install -y curlto the Dockerfile and rerun. Expected: the curl note is gone.ddtest onboard --agent. Expected:ci_provider_unsupported.Cleanup: delete the scratch repositories.
🤖 Generated with Claude Code