Repository navigation
Antalya 26.8 - Store the submodule cache in Hetzner Object Storage - #2740
Merged
strtgbb merged 4 commits intoOct 31, 2026
Merged
Conversation
Most runners are in Hetzner, so every job that downloads the submodule archive from AWS S3 pays egress. Keep the archive in a Hetzner bucket instead. Buckets listed in `S3_EXTERNAL_BUCKETS` are served by a boto3 client with their own endpoint and credentials. `S3.put` and `S3.head_object` use it for those buckets, and keep using the AWS CLI for every other bucket. With `public_read`, reads are unsigned, so community PRs can use the archive without credentials; they never create it. A missing credential is an error, so ambient AWS credentials are never sent to the Hetzner endpoint. The archive is public, so `_prepare_submodule_cache` refuses to upload it if the clone token ended up in a git config file. The workflows export `HETZNER_S3_ACCESS_KEY` and `HETZNER_S3_SECRET_KEY` from GitHub secrets. The bucket is `altinity-builds-cache` in hel1. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
strtgbb
added
antalya
cicd
The first CI run failed with a 403 on the existence check for the submodule archive. The bucket is public for object reads, but answers a HEAD of a missing key with 403 to callers who cannot list it. The config job read the bucket unsigned because the bucket is marked `public_read`, even though it has the keys. Reads are now unsigned only for jobs without the bucket's keys. Those jobs treat a 403 from `head_object` as a miss; jobs with the keys use a signed client, so a 403 is still an error for them. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
20 of 30 tasks
The Altinity PR workflow has full-suite sanitizer jobs that use CAS storage (`Stateless tests (amd_asan_ubsan, cas s3 storage, ...)`), which the test rejected as unexpected full-suite jobs. It failed in every CI run, including without other changes. Related: https://github.com/Altinity/ClickHouse/actions/runs/41066953900 Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
strtgbb
added a commit
that referenced
this pull request
Oct 30, 2026
Attached volumes tied the cache to whichever builder a job landed on, and the runner scaler never reused a volume for builders made from a snapshot image. The cache now travels as one archive per branch, build type and debug-info flavour in the Hetzner bucket from the submodule cache: `altinity-builds-cache/ci_ch_cache/thinlto/<branch>/<build type>-<g|g0>.tar.zst`. Every ThinLTO build that links restores the archive before `ninja`. PR builds use their base branch's archive. Pushes and internal PRs then upload it again, so `-g` archives come from `MasterCI` and `-g0` ones from PR builds; community PRs read anonymously and write nothing, like the shared compiler cache. The cache-warmup builds only compile, so they skip it. The upload keeps exactly the entries the build used: restored entries get an access time of zero, so under `relatime` the next read sets it, and the entries still at zero are dropped before compressing with `zstd --long`. That keeps an archive at one build's working set, about 15 GB raw for `-g`, and lld's own pruning is off. A restore that fails leaves an empty cache rather than a partly extracted one, and an archive over 20 GB (`-g`) or 5 GB (`-g0`) is not uploaded. This removes the praktika `--volume-if-exists` setting, the cache mounts and the per-target builder labels, and puts `amd_cfi` and the cache-warmup builds back on `ARM_LARGE`. Related: #2740 Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce egress costs associated with the new submodule cache.
Sccache and LTO cache can also be moved after this PR.
Changelog category (leave one):
CI/CD Options
Exclude tests:
Regression jobs to run: