DDTest

DDTest helps you scale CI workloads down when Datadog Test Optimization skips tests: it plans how to run the remaining test files across CI nodes or local workers in the most efficient way.

Use ddtest plan once to create a reusable .testoptimization/ plan, then use ddtest run in each CI job to run only the files assigned to that job.

DDTest can also write file lists for another runner, such as Knapsack Pro or parallel_tests.

Currently supported:

• Ruby with RSpec or Minitest.
• Python with pytest.
• JavaScript with Cucumber, Cypress, Jest, Mocha, Playwright, or Vitest.

Try Test Optimization locally

With your project's test dependencies and any required browsers/services ready:

ddtest onboard    # GitHub Actions setup instructions
ddtest onboard --framework vitest --apply       # Preview exact setup edits
ddtest onboard --framework vitest --apply --yes # Apply reviewed setup and validate
ddtest testdrive  # Preview, confirm, and run tests locally

For agent-driven execution, use ddtest onboard --framework vitest --apply --yes or ddtest testdrive --framework vitest --all --agent --yes. The shared coordinator returns a resolved plan, precise blockers and bounded resume commands; it checks source preservation, report exclusions and cleanup. Finish with ddtest testdrive --report to verify freshness and render the factual verdict. --format json is available to hosts. Plain testdrive retains its existing execution behavior; onboard --instructions prints manual guidance. See agent onboarding.

For Jest, Vitest, Mocha, Playwright and Cypress, testdrive compares the original test suite without and with instrumentation, then validates supported advanced features using temporary probes. Playwright reports TIA skipping and per-test coverage as unsupported. Cypress TIA requires application coverage from the tested page (Istanbul window.__coverage__) and a test-level skip of the probe. Matching existing failures are acceptable. Other frameworks keep their existing reporting checks.

No Datadog account or Agent is needed. Testdrive links to an HTML report under .testoptimization/report.html with a compact .testoptimization/testdrive.json. Use --yes after reviewing the preview in non-interactive sessions, --framework to select a runner, and --command for package scripts or custom setup.

For Jest, start with testdrive configuration checks.

For repositories with several Jest configurations, use ddtest testdrive --framework jest --all --yes to validate discovered CI commands and their prerequisites. Unknown setup stays explicitly unvalidated; --command selects a focused command. See configuration validation.

Testdrive reuses a detected project tracer; otherwise it installs the release every instrumented CI job pins, or the latest release (--tracer-version selects a fallback release or git:<ref>). JavaScript/Python installs are isolated. Ruby runs bundle add datadog-ci, updating the project's Gemfile and lockfile.

Prerequisites

Before using ddtest plan or ddtest run, you must have Datadog Test Optimization already set up and enabled with a Datadog Test Optimization library for your language and framework. DDTest relies on this integration to discover your tests and plan test execution accordingly.

Minimum supported library and runtime requirements:

• Building DDTest from source requires the Go version specified in go.mod or later.
• Ruby requires the datadog-ci gem 1.31.0 or higher.
• Python requires the ddtrace package 4.11.0 or higher and pytest.
• JavaScript requires the dd-trace package 5.111.0 or higher and Node.js. Cucumber support is tested with @cucumber/cucumber 7 through 13; Cypress support requires Cypress 12 or higher; Mocha support requires Mocha 8 or higher; Playwright support requires Playwright 1.18 or higher; Vitest support requires Vitest 1.6 or higher and dd-trace 5.125.0 or higher. Vitest uses a direct Node API integration; see configuration and migration instructions.

For instructions on setting up Test Optimization, see the Datadog Test Optimization documentation.

Usage

DDTest ships as a CLI tool ddtest with two primary sub-commands: plan and run.

Use plan to create a reusable .testoptimization/ plan without running tests. Use run to execute that plan locally or in CI. If a plan is missing, run will generate it on the fly.

Both commands accept files, directories, or quoted globs to narrow the configured test discovery. When a saved plan exists, use plan to change the selection; run rejects positional arguments.

ddtest plan spec/models spec/requests
ddtest run

DDTest is meant to run in CI. Local runs are possible when you want to reuse CI's skippable tests on your machine; see Running locally with CI skippable tests. For planning-step performance tips and framework-specific setup notes, see Best practices.

Platform and framework are detected from the current directory when --platform and --framework are omitted, including for plan and run. Projects with multiple platforms or test frameworks must select one explicitly. Flags and DD_TEST_OPTIMIZATION_RUNNER_PLATFORM / DD_TEST_OPTIMIZATION_RUNNER_FRAMEWORK override detection; an explicit framework also identifies its platform. Detection does not execute project scripts or change the test command. Use --command when your project requires a custom invocation.

Available commands

ddtest plan

Creates a reusable execution plan under .testoptimization/ without running tests. The plan contains the runnable test files, the selected CI node or worker count, and any per-node file lists needed by ddtest run or another runner.

Example:

ddtest plan \
  --platform ruby \
  --framework rspec \
  --min-parallelism 8 \
  --max-parallelism 32

For Python/pytest:

ddtest plan \
  --platform python \
  --framework pytest \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Jest:

ddtest plan \
  --platform javascript \
  --framework jest \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Vitest:

ddtest plan \
  --platform javascript \
  --framework vitest \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Mocha:

ddtest plan \
  --platform javascript \
  --framework mocha \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Cypress:

ddtest plan \
  --platform javascript \
  --framework cypress \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Playwright:

ddtest plan \
  --platform javascript \
  --framework playwright \
  --min-parallelism 8 \
  --max-parallelism 32

For JavaScript/Cucumber:

ddtest plan \
  --platform javascript \
  --framework cucumber \
  --min-parallelism 8 \
  --max-parallelism 32

This prepares the plan and writes it to .testoptimization/ folder for later reuse. Copy .testoptimization/ to any CI job that runs ddtest run or reads DDTest's plan file lists. For the full file layout and formats, see Plan file layout.

ddtest run

Runs tests using the framework you specify. If .testoptimization/ exists, DDTest uses its precomputed plan; otherwise it first runs plan and then executes.

ddtest run --platform ruby --framework rspec

For Python/pytest:

ddtest run --platform python --framework pytest

For JavaScript/Jest:

ddtest run --platform javascript --framework jest

For JavaScript/Vitest:

ddtest run --platform javascript --framework vitest

For JavaScript/Mocha:

ddtest run --platform javascript --framework mocha

For JavaScript/Cypress:

ddtest run --platform javascript --framework cypress

For JavaScript/Playwright:

ddtest run --platform javascript --framework playwright

For JavaScript/Cucumber:

ddtest run --platform javascript --framework cucumber

For CI-node mode, worker environment variables, custom commands, and parallelism details, see Running DDTest.

Common settings

CLI flag	What it does
--platform	Language/platform. Currently supported: ruby, python, javascript.
--framework	Test framework. Currently supported: rspec, minitest, pytest, cucumber, cypress, jest, mocha, playwright, vitest, webdriverio (wdio).
--command	Override the default base command for supported framework modes. Used by RSpec and Minitest run/discovery, Cucumber, Cypress, Jest, Mocha, and Playwright run/discovery. Vitest plan/run reject this option; testdrive accepts an explicit Vitest CLI or supported ddtest run command.
--vitest-config	Vitest config file used by both planning and execution; defaults to Vitest config discovery. See migration instructions.
--min-parallelism	Minimum CI node or worker count DDTest considers when planning.
--max-parallelism	Maximum CI node or worker count DDTest considers when planning.
--target-time	Target wall time DDTest tries to satisfy when selecting parallelism.
--ci-node	Run only the files assigned to CI node N.
--tests-location	Override the default test file discovery glob.
--tests-exclude-pattern	Exclude matching test files from discovery.
--strict-discovery	Fail planning when full test discovery fails.

For all flags, environment variables, and defaults, see Settings.

Installation

To install from a checkout:

make install                          # GOBIN, or the first GOPATH entry's bin
make install GOBIN="$HOME/.local/bin"  # Custom destination; add it to PATH

This project uses GitHub Releases for distribution.

Use gh command line tool to download the latest release in GitHub actions:

- name: Download ddtest binary
  run: |
    mkdir -p bin
    gh release download --repo DataDog/ddtest --pattern "ddtest-linux-amd64" --dir bin
    mv bin/ddtest-linux-amd64 bin/ddtest
    chmod +x bin/ddtest
  env:
    GH_TOKEN: ${{ github.token }}

...or use curl:

mkdir -p bin
curl -fsSL https://github.com/DataDog/ddtest/releases/latest/download/ddtest-linux-amd64 -o bin/ddtest
chmod +x bin/ddtest

The list of available precompiled artifacts is on release page.

CI configuration examples

• GitHub Actions
• CircleCI

More documentation

• Running DDTest
• Settings
• Plan file layout
• Third party test runners
• GitHub Actions example
• CircleCI example
• Best practices
• Running locally with CI skippable tests and runtime tags
• DDTest 1.0 upgrade guide

Vitest testdrive supports baseline/reporting comparison and controlled retries, early flake detection, skipping, quarantine, disabled tests and attempt-to-fix. Use ddtest testdrive --framework vitest --all --check-only --yes before ddtest testdrive --framework vitest --all --yes. For focused diagnostics, replace --all with --command "npx vitest run --config ci.ts" or a supported ddtest run invocation. plan/run retain their Node API behavior and reject their own --command. See Vitest validation.

Mocha testdrive validation

See Mocha validation for commands, coverage requirements and limitations.

Cucumber testdrive also compares native scenario outcomes and exercises all six controls, requiring real application source coverage through the project's NYC setup. See Cucumber validation.

WebdriverIO validation

Use ddtest onboard --framework webdriverio for setup instructions, then ddtest testdrive --framework webdriverio --all --check-only --yes for the configuration inventory. Execute a focused command with --command 'npm test' without --check-only. The local runner with Mocha or Jasmine is supported; standalone automation, custom adapters and browser runners remain unvalidated. Five controlled features are required; TIA skipping and coverage are unsupported. See WebdriverIO validation.

Developer onboarding experiments

The ddtest onboarding evaluation skill compares uncoached Codex/Claude onboarding in fresh repositories. It is maintainer tooling, separate from customer onboarding and the ddtest binary.

Use the container runner with a clean trusted image, no cloud/GitHub/npm credentials or MCP servers, and an explicitly selected throwaway agent login. Repository install scripts and auto-approved agents must not run on the host with personal credentials. CLI permission modes alone do not isolate a trial.

The evaluator reads the skill; the tested agent receives only a simple prompt pointing to the binary. Evidence includes redacted public tool activity, current product report verdicts and owned cleanup. The runner pins/checks CLI versions, isolates configuration and removes the container on interruption or timeout. See the skill for setup, independent grading and marker-verified cleanup. CI uses fake CLIs and offline fixtures, never real model sessions.