[politerealism]

Summary

Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to a destination signed by that CA fail upstream TLS establishment (curl: (52) Empty reply from server), and the only workaround is rebuilding the supervisor image with the CA baked in per upgrade or rotation.

This adds a Podman-only additional_ca_bundle setting that restores that trust without a custom image.

Related Issue

Fixes #3781.

This is a deliberately narrower, Podman-only alternative to the in-flight cross-driver implementation in #3292 (Docker/Podman/Kubernetes/VM delivery, gateway-level config schema, lifecycle rotation reconciliation), which remains the intended long-term direction — see the note posted on #3781. There are a few other things in flight that need to settle before taking on a change of #3292's size, so this fix is scoped to just what #3781 needs.

Changes

• Add additional_ca_bundle to PodmanComputeConfig, independent of the existing proxy_ca_bundle (which requires https_proxy to be set) — this applies to direct, inspected egress regardless of whether a corporate proxy is configured.
• Validate the bundle on the gateway host when the Podman driver starts (read and parse it, not just check non-empty), so a misconfigured path or certificate-free bundle fails gateway startup immediately instead of producing a failed sandbox once the supervisor container starts.
• Bind-mount the bundle read-only into the supervisor container and pass its path via a new --additional-ca-bundle supervisor flag. The mount path is listed in trusted_mount() so the supervisor keeps it and the workload does not.
• Fold it into the supervisor's upstream TLS trust store alongside (not instead of) the system CA bundle, reusing the exact additive-append mechanism already proven for the corporate proxy CA case — no changes to tls.rs's trust-store construction itself.
• The CA is trusted for all destinations by both the supervisor and the workload (it also lands in the sandbox trust bundle), so the docs advise using a CA you control or one with name constraints.
• Document the new setting in docs/how-it-works/gateways/configuration.mdx and docs/how-it-works/sandboxes/runtimes.mdx.

Deliberately out of scope: Docker/Kubernetes/VM delivery, gateway.toml schema, new proto fields, and lifecycle/rotation reconciliation for stopped sandboxes — all left to #3292 or a future follow-up.

Testing

• mise run pre-commit passes
• cargo fmt --all -- --check passes
• cargo clippy -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --all-targets -- -D warnings passes
• cargo test -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --lib — 528 + 232 + 128 + 1458 tests pass, 0 failures
• Real end-to-end TLS regression test added (additional_ca_enables_trust_of_privately_signed_upstream in tls.rs): a genuine rustls TlsAcceptor/TlsConnector handshake over a real loopback connection, reproducing bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change #3781's exact failure without the CA trusted and proving it succeeds once it is
• Config→mount→argv wiring tested directly (container.rs), including the no-https_proxy case that's the actual point of this fix
• Fail-closed validation tested for empty, nonexistent, and certificate-free bundle paths
• build_isolation_specs() tested directly: the final supervisor spec keeps the bind mount and --additional-ca-bundle argv, and the workload spec excludes the mount

Checklist

• Follows Conventional Commits
• Commit is signed off (DCO)
• Architecture/user-facing documentation updated

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[politerealism]

Rebased onto current main (the run_sandbox arguments moved into SandboxRunConfig, so additional_ca_bundle is threaded through that struct now) and clarified in the docs that the CA is trusted for all destinations by both the supervisor and the workload. New head is 729061e.

@sjenning @elezar could one of you take a look when you have a chance, and run /ok-to-test 729061e830f5b525b655455ec64d5d4fa28894e7 so CI can run on the new commit? This is the narrower Podman-only fix for #3781.

[elezar]

/ok-to-test 729061e

[TaylorMutch]

The additive CA trust approach makes sense, but the final Podman container split removes the CA mount from the supervisor, so enabling the setting prevents sandbox startup. Please fix the final-spec wiring and add coverage for build_isolation_specs(). The linked issue #3781 is accepted. I reviewed the full diff and surrounding provisioning code; I did not run tests locally.

[politerealism]

Thanks, both addressed in ec7691f. ADDITIONAL_CA_MOUNT_PATH is now in trusted_mount(), and a new test runs build_isolation_specs() to check the final supervisor keeps the mount and argv while the workload doesn't. The docs and code comment now say the bundle is validated when the driver starts, and I added Podman troubleshooting to the debug-openshell-cluster skill (startup validation failure, and how to check the supervisor mount and argv).

[TaylorMutch]

/ok to test ec7691f

[TaylorMutch]

Re-reviewed ec7691f. Both findings are addressed: the additional CA mount now survives the final supervisor/workload split, with a regression test covering the final specs, and the validation timing plus deployment troubleshooting guidance are corrected. No remaining blocking concerns found. All executed checks and required gates passed on this head, including Rust tests on Linux x86_64, Linux ARM64, and macOS ARM64. E2E suites were skipped by workflow configuration; I did not run tests locally.