Skip to content

Add conda/micromamba:v3 template with layered conda builds - #1146

Merged
pditommaso merged 21 commits into
masterfrom
002-conda-micromamba-v3-layers
Oct 2, 2026
Merged

pditommaso merged 21 commits into
masterfrom
002-conda-micromamba-v3-layers

Conversation

@pditommaso

@pditommaso pditommaso commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds conda/micromamba:v3, an opt-in Docker build template. It installs the Conda environment like v2, then ships it as one image layer per package group (each ≤ 500 MB before compression) instead of one multi-GB layer. Existing templates and container ids are unchanged.

Design: spec (with a sequence diagram of the build) and ADR.

How it works

  • condasplit (Go, standard library only) runs in the install step. It moves /opt/conda into /layers/00, /layers/01, …, grouped by package, and prints the layer plan in the build log.
  • condasplit-frontend is a BuildKit frontend, selected by the template's first line # syntax=<condasplit image>. It repeats the template's single COPY --link --from=build /layers/NN/ / line once per directory, so the image gets exactly one layer per directory. The build itself is done by BuildKit's built-in Dockerfile frontend, and micromamba runs once.
  • Both ship in one ~8 MB image, set with wave.build.condasplit-image (default public.cr.seqera.io/wave/condasplit:v1, published by the build-condasplit workflow on the next [release] commit; public.cr.stage-seqera.io/wave/condasplit is only for tests and local development). The buildctl arguments and the build pod settings don't change.
  • Lock file URLs are fetched with ADD, because micromamba can't read an explicit lock from a URL.

Results

Wave-rendered builds on the production BuildKit (v0.25.2-rootless) with the Kubernetes pod's settings. Reference: gatk4 + gcnvkernel, 323 packages, linux/amd64.

v2 v3
Layers 2 20 (1 base + 19)
Largest layer 1,988 MB 399 MB
Total size 2,018 MB 1,861 MB
Build time 283 s 146 s
  • /opt/conda is identical to v2 (97,285 paths) apart from the unused package cache pkgs/.
  • A small environment (samtools) gets 3 layers, with no empty layers.
  • The tool isn't in the image.
  • Wave tests and the Go tests of both modules pass. Golden files confirm that v1, v2, pixi and cran output is unchanged.

Also

  • run.sh: works with AWS SSO (AWS_PROFILE=<profile> ./run.sh, no static keys) by pushing local builds through the ECR dual-stack endpoint with the session's ECR login token. The Gradle progress bar is off.

Before merge / follow-ups

🤖 Generated with Claude Code

pditommaso and others added 11 commits September 30, 2026 12:51
Add a new opt-in Docker build template, conda/micromamba:v3, that splits
the conda prefix into up to 32 per-package layers (each under ~500 MB)
instead of a single multi-GB layer.

- conda-layers: static Go tool (stdlib only) that plans layers from
  conda-meta ownership and moves files into per-slot dirs inside the
  install RUN; shipped as a tiny multi-arch image mounted at build time
  and never copied into the final image
- Wave: new template files, CondaHelper/ContainerHelper routing,
  wave.build.conda-layers-image config (stage image by default),
  registry credentials for the RUN --mount image
- v3 fetches lock-file URLs with ADD, since micromamba can't read an
  explicit lock from a URL; v1/v2 templates are unchanged (golden tests)
- Docs, ADR and a GitHub workflow to publish the tool image

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Resolve conflict in ContainerControllerTest by keeping both the
micromamba v3 tests and the multi-platform singularity tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Rename the Go tool, folder, module path, image and workflow to
  condasplit; the config property is now wave.build.condasplit-image
  (stage image public.cr.stage-seqera.io/wave/condasplit:v1)
- Spec/ADR: v3 adds lock-file URLs with ADD and installs the local copy
- Spec/ADR: the prefix is 0755 root:root as in v2, only its times are kept
- Update the layer plan example with the integration run numbers
- Note that directory link counts are not compared (overlayfs nlink=1)
- R8: the workflow runs the Go tests in a golang:1.27 container as root

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Content changes that belong to the previous commit, which only
recorded the file moves.

- Rename references to the tool, module path, image, workflow and
  config property (wave.build.condasplit-image)
- Spec/ADR: v3 adds lock-file URLs with ADD and installs the local copy
- Spec/ADR: the prefix is 0755 root:root as in v2, only its times are kept
- Update the layer plan example with the integration run numbers
- Note that directory link counts are not compared (overlayfs nlink=1)
- R8: the workflow runs the Go tests in a golang:1.27 container as root

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- condasplit: drop the EXDEV copy fallback (copyNode, copyFile,
  lutimes, times_*.go): rename() never crosses a filesystem in the
  build step, overlayfs copies up lower-layer files (verified on the
  rootless BuildKit), and directories are created, never renamed
- condasplit: drop atime tracking, the rename test seam and the argument
  guards the template never needs; use flag.Func, slices, maps and cmp
- condasplit: one unit type for packages, chunks and layers, and a single
  warnings list instead of per-kind bookkeeping; fixed two-digit slot names
- condasplit: chmod the prefix parents, mkdir inherits the setgid bit of /
  in a rootless BuildKit step (regression test added)
- Wave: findMountRepositories uses two patterns instead of nested loops,
  and containerFileV2/V3 share the micromamba 2.x options setup
- Spec/ADR/README: describe overlayfs copy-up instead of the EXDEV fallback

Co-Authored-By: Claude Opus 5.5 <[email protected]>
BuildKit adds a layer for every COPY, even an empty one, so v3 images
always had 32 conda layers, most of them empty in small environments.

- condasplit/frontend: BuildKit frontend, its own Go module pinned to
  BuildKit v0.25.2. It builds the install stage, lists /layers/NN,
  removes the COPY lines of the empty slots and builds the image, both
  times through the built-in Dockerfile frontend; the install stage runs
  once as the second build reuses the first
- The condasplit image ships the frontend as its entrypoint, and the v3
  templates select it with `# syntax={{layers_image}}`: the buildctl
  invocation, the config and the credentials are unchanged
- Dockerfile, Makefile and CI workflow build and test both modules
- Spec, ADR and docs: one layer per used slot, frontend decision

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- Read the single result of the install stage, Wave builds one
  platform at a time, and drop the nil options guard
- One COPY for both binaries, no separate module download step
- One CI step for both Go modules, leaner frontend tests

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The v3 templates listed 32 COPY lines, most of them dropped at build
time. They now have a single `COPY --link --from=build /layers/NN/ /`
line, which the frontend repeats for each directory under /layers.

- condasplit creates only the directories of the layers; --slots is the
  maximum number of layers
- The frontend lists /layers once and fails when the line is missing;
  built without the frontend the Dockerfile fails on /layers/NN
- Tests, spec, ADR, README and docs updated

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The v3 lock-file variant differed from the conda-file one in three lines,
and only lock-file URLs reached it. One template now serves both inputs
with the conda_file_add and conda_file_opts bindings; the rendered
Dockerfiles are byte-identical.

- TemplateUtils.condaToDockerFileUsingV3 replaces the two v3 methods,
  and the v2 condaPackagesTemplate1 is back to its original form
- v3 tests assert the template text once, in TemplateUtilsTest; the
  helper tests check the delegation

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Shorter spec (499 to 256 lines): build flow with a sequence diagram,
the roles of the template, condasplit and the BuildKit frontend, the
key decisions with their reasons, and verification results from the
end-to-end builds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
pditommaso and others added 3 commits September 30, 2026 20:45
With AWS_PROFILE set and no static AWS keys, push local builds through
the ECR dual-stack endpoint, which Wave treats as a plain registry, using
the ECR login token of the SSO session. The token stays in the
environment; the generated config (build/config-sso.yml) only
references it. Also disable the Gradle progress bar.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The build-and-push job now runs on a `[release]` commit on master or
release-* (same rule as Wave CI) instead of a manual dispatch. It runs
`make release`, which publishes the condasplit/VERSION tag unless it
already exists, since tags are immutable.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The AWS SDK clients, e.g. the SES mailer, can't read an SSO profile, so
export the session credentials when run.sh uses AWS SSO.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Comment thread condasplit/inventory.go Resolved
Comment thread condasplit/frontend/main.go Resolved
pditommaso and others added 7 commits October 1, 2026 07:52
A symlinked --src was not descended, so the plan had no layers and the
verify step passed with no conda content in the image. The frontend now
sets cmdline instead of deleting it, so the built-in Dockerfile frontend
can never forward the build back to it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The cmdline option set by forward already stops the built-in Dockerfile
frontend from forwarding the build again, so the Dockerfile is now passed
through with its syntax directive.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pditommaso
pditommaso merged commit 5ffc92c into master Oct 2, 2026
@pditommaso
pditommaso deleted the 002-conda-micromamba-v3-layers branch October 2, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants