Skip to content

fix(security): update runtime certificate trust in v0.10.x - #3320

Open
rtemperini wants to merge 3 commits into
kagent-dev:release/v0.10.xfrom
rtemperini:codex/fix-emudhra-trust-v0.10
Open

rtemperini wants to merge 3 commits into
kagent-dev:release/v0.10.xfrom
rtemperini:codex/fix-emudhra-trust-v0.10

Conversation

@rtemperini

@rtemperini rtemperini commented Oct 22, 2026 •

Copy link
Copy Markdown

Follow-up to #3310. The Go verifier fix does not update the independent trust stores used by Python, uv, Node, or git/curl. The published 0.10.4 ADK image still accepts the affected roots in its system store, certifi, and ensurepip wheel.

  • Upgrade uv to 0.13.5, certifi to 2026.10.19, and boto3/botocore to 1.43.117. The updated uv catalog installs Python 3.13.17 with pip 26.2.2, including ensurepip. Require the fixed certifi in the published core package.
  • Refresh pinned runtime bases and require fixed Wolfi/Alpine CA packages. Select the Node 24 package with a 24.22.1 minimum, allowing later Node 24 patches; the full Python image uses Node 26.12.1. Keep the Python runtime on Debian 12 for glibc compatibility and copy the updated upstream Debian CA bundle from its builder; the current Debian 12 distroless base still has the affected roots.
  • Replace the floating gcr.io/distroless/static:nonroot Go runtime base with gcr.io/distroless/static-debian13:nonroot@sha256:d62067879de890a1def9da0e7766b7d20810a5e5555cef8506b752437d7a3e88 for the controller and Go ADK. The pinned Debian 13 base has the corrected CA bundle; its AMD64 and ARM64 bundles were verified.
  • Enforce Debian CA package minimums through native APT dependency constraints: 20250419+deb13u1 in the Trixie builder and 20250419~deb12u2 in the Bookworm full image.
  • Point Python gRPC at the updated system bundle using its supported GRPC_DEFAULT_SSL_ROOTS_FILE_PATH setting. Its upstream roots fix has no published wheel yet. No certificate files are filtered or patched.
  • Keep Bookworm sample images on Bookworm using official Python 3.13.17 plus the upstream uv binary. uv 0.13.5 no longer publishes Bookworm Python images.

Local validation: the published kagent-adk:0.10.4 image retains affected CA trust. Rebuilt ARM64 slim/full and AMD64 slim images pass the five local trust checks. The rebuilt system, certifi, and botocore bundles each reject all 12 published compromised chains offline (36 checks). Lockfile checks, Python lint, and git diff --check pass.

This needs a new release and rollout of rebuilt images. Inactive bundled gRPC/httplib2 fallback files remain upstream-owned; these updates address effective trust and do not remove every bundled PEM. Full application tests and vulnerability scans were not run locally.

References: incident and fixed versions, uv 0.13.5, certifi, botocore.

Main-branch counterpart: #3321.

@rtemperini
rtemperini requested review from a team, Charlesthebird and peterj as code owners October 22, 2026 12:24
@github-actions github-actions Bot added the bug Something isn't working label Oct 22, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 22, 2026
Comment thread .github/workflows/image-scan.yaml Outdated
Comment thread scripts/check_python_image_trust.py Outdated
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 22, 2026
@rtemperini
rtemperini requested a review from EItanya October 23, 2026 07:16
Comment thread ui/Dockerfile Outdated
### STAGE 1: Dependencies and Build
ARG BASE_IMAGE_REGISTRY=cgr.dev
ARG TOOLS_NODE_VERSION=24
ARG TOOLS_NODE_VERSION=24.22.1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we avoid pinning the exact nodejs patch version?

Comment thread go/Dockerfile
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
FROM gcr.io/distroless/static:nonroot
FROM gcr.io/distroless/static-debian13:nonroot@sha256:d62067879de890a1def9da0e7766b7d20810a5e5555cef8506b752437d7a3e88

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could the static-debian13 swap be mentioned in the description?

@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants