Skip to content

Stable 23.8 - Bump brotli to v1.0.10, update runner labels, use minio mirror image - #1864

Merged
zvonand merged 25 commits into
releases/23.8.16from
update-runner-labels-23.8
Oct 28, 2026
Merged

zvonand merged 25 commits into
releases/23.8.16from
update-runner-labels-23.8

Conversation

@strtgbb

@strtgbb strtgbb commented Jun 2, 2026 •

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Critical Bug Fix (crash, data loss, RBAC)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Bump brotli from v1.0.9 to v1.0.10; fix CVE-2026-111497

Description

CI fixes for releases/23.8.16:

  • Update runner labels; read CI secrets from env instead of SSM; update artifact actions.
  • Use the clickhouse/minio-minio mirror image for integration tests. minio/minio was deleted from Docker Hub, so every with_minio=True test failed in cluster.start() with pull access denied for minio/minio. The mirror (upstream Mirror the MinIO integration-test images into the clickhouse/ namespace ClickHouse/ClickHouse#119999) has no RELEASE.2021-09-23T04-46-24Z tag, so minio is bumped to RELEASE.2024-07-31T05-46-26Z, same as releases/24.8.14.
  • Take the test_storage_delta fix from upstream Upgrade MinIO to support accepting non signed requests ClickHouse/ClickHouse#55245 (the 2021 → 2023 minio upgrade): newer minio does not match a list_objects prefix with a leading slash.
  • Take the stateless-test minio/mc binaries from clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z instead of dl.min.io, which now returns 410 Gone, so the image could no longer be rebuilt. Same as upstream Update minio in stateless tests ClickHouse/ClickHouse#67975 (stateless tests on 2024 minio), it also takes the renamed mc admin policy attach / mc anonymous set commands in setup_minio.sh and the S3ObjectStorage::listObjects fix that skips a "/" prefix.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@strtgbb strtgbb added stable cicd Improvements and fixes to the CICD process 23.8 labels Jun 2, 2026
minio/minio was deleted from Docker Hub, so every with_minio=True test
fails in cluster.start() on the image pull. Point docker_compose_minio.yml
at the clickhouse/minio-minio mirror (upstream ClickHouse#119999). The mirror has no
RELEASE.2021-09-23T04-46-24Z tag, so bump to RELEASE.2024-07-31T05-46-26Z,
the same image releases/24.8.14 uses.

Also take the test_storage_delta change from upstream ClickHouse#55245 (the
2021 -> 2023 minio upgrade): newer minio does not match a list_objects
prefix with a leading slash.
@zvonand zvonand changed the title Stable 23.8 - update runner labels Stable 23.8 - update runner labels, use minio mirror image Oct 26, 2026
dl.min.io returns 410 Gone for every minio/mc binary, so the
stateless-test image can no longer be rebuilt. Copy minio and mc from
clickhouse/minio-minio:RELEASE.2024-07-31T05-46-26Z instead, the same
image the integration tests use.

The newer mc and minio need the same changes upstream made when it
moved stateless tests to 2024 minio (ClickHouse#67975): the renamed mc policy
commands, and not sending a "/" prefix in S3ObjectStorage::listObjects.
@zvonand zvonand changed the title Stable 23.8 - update runner labels, use minio mirror image Stable 23.8 - Bump brotli to v1.0.10, update runner labels, use minio mirror image Oct 26, 2026
@altinity-robot

altinity-robot commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

This is an automated comment for commit 581c21f with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
ClickHouse build checkBuilds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process✅ success
Compatibility checkChecks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help✅ success
Docker image for serversThe check to build and optionally push the mentioned image to docker hub✅ success
Install packagesChecks that the built packages are installable in a clear environment✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Mergeable CheckChecks if all other necessary checks are successful✅ success
Push to DockerhubThe check for building and pushing the CI related docker images to docker hub✅ success
Ready for releaseThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Check nameDescriptionStatus
CI runningA meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR⏳ pending
Stress testRuns stateless functional tests concurrently from several clients to detect concurrency-related errors❌ failure

server-jepsen-test was not listed as a test-base dependent, so it was built
from the stale test-base:latest with an outdated Kitware apt key.

Also push per-arch images without provenance attestation: an index with an
attestation is rejected by `docker manifest create`.
Restore the test images fixture and the expected changed images from
upstream, with altinityinfra names: the fixture referenced images it did
not define. Run test_get_changed_docker_images as a push to an existing PR,
since PR number 0 rebuilds every image. Mock reading the build log on
failure, and expect --provenance=false and the :latest tag in the build
command.
Azurite's transitive dependency @dabh/diagnostics >= 2.0.5 pulls
@so-ric/colorspace, which uses syntax Node.js 12 from Ubuntu 22.04 can't
parse, so azurite failed to start.
DOCKER_PASSWORD belongs to the altinityinfra account (the image push
scripts log in with it), so pairing it with secrets.DOCKER_USERNAME failed
with 'incorrect username or password'.
rustup >= 1.28 prints the toolchain list as '(active, default)', which the
bundled contrib/corrosion does not parse, so cmake failed with
"Could not find toolchain ''". Same pin as upstream 0511e72, plus
disabling auto self-update, since 'rustup toolchain install' otherwise
updates rustup back to the latest version.
Each crate is copied to the build directory on its own, outside of the
workspace, so cargo ignored rust/Cargo.lock and resolved the latest
dependency versions. Those now include crates using the 2024 edition
(cpufeatures 0.3.1), which the pinned nightly-2023-07-04 cargo can't parse.
altinityinfra/python-bottle:latest is an arm64-only image now, so on amd64
runners the S3 mock servers in the resolver container fail to start.
mongod refuses to start with an expired certificate. Taken from 6439fec.
PHP 8.5 prints a deprecation warning for PDO::MYSQL_ATTR_DIRECT_QUERY, which breaks test_mysql_protocol::test_php_client.
get-deb-url.py from the clickhouse-regression release branch sets clickhouse_path, so --clickhouse-binary-path was empty.
The two inserts used now()+1 and now(); when they landed in different
seconds the values matched and count() returned 1 instead of 0.
zvonand and others added 3 commits October 28, 2026 06:54
The AWS_REPORT_* keys are no longer valid, so every regression job failed
on the S3 upload after the suite passed.
These tests came with upstream backports but depend on changes that 23.8
does not have:
- 02888: the mysql pool throws POCO_EXCEPTION, not ALL_CONNECTION_TRIES_FAILED
- 02907: there is no 'backups' disk in the test config; use the default disk
- 02920: projections with nullable keys need allow_nullable_key
(cherry picked from commit 2c86d7d)
@zvonand

zvonand commented Oct 28, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

CI triage for 581c21f

Summary: 2 failing checks. Neither is caused by a code change in this PR.

Check Verdict
DCO Process issue in this PR: the head commit has no sign-off. Simple fix, see below.
Stress test (asan) Known false assertion in the 23.8 filesystem cache. Upstream already removed it, but it was never backported to 23.8. Not caused by this PR's code. Hit twice on this branch.

Several regression jobs (Alter (attach), S3 (aws_s3), S3 (minio), Common (aggregate_functions)) are still running, so this triage does not cover them. All other finished checks pass: builds, stateless/stateful/integration, the other 4 stress runs, compatibility and docker.


1. DCO: failure caused by this PR (sign-off only)

:red_circle: 581c21f  Remove bad test (3)  Sign-off not found

The other 24 commits are skipped as "sign-off not required for members". The head commit is the cherry-pick of upstream 2c86d7db ("Remove bad test (3)"). Its author is Alexey Milovidov, so the members exemption doesn't apply, and the cherry-pick added no Signed-off-by.

Fix (any one of these):

  • An org admin overrides DCO for this PR. For a cherry-picked upstream commit, this is the usual route.
  • Add a DCO remediation commit, which needs no force-push. See cncf/dco2 "individual remediation".
  • Amend with git commit --amend -s and force-push. I can't do this one (no force-push), and it would also restart CI.

For future cherry-picks, use git cherry-pick -x -s.

2. Stress test (asan): not caused by this PR (stale assertion in fs cache)

Report · fatal_messages · gdb.log

<Fatal> : Logical error: 'inserted'.
<Fatal> Application: Child process was terminated by signal 6.
Thread 400 "LoadMarksThread" received signal SIGABRT
#3  DB::abortOnFailedAssertion   src/Common/Exception.cpp:50
#4  DB::CleanupQueue::add        src/Interpreters/Cache/Metadata.cpp:376
#5  DB::LockedKey::~LockedKey    src/Interpreters/Cache/Metadata.cpp:677
#11 DB::FileSegment::complete    src/Interpreters/Cache/FileSegment.cpp:718
#17 DB::CachedOnDiskReadBufferFromFile::~CachedOnDiskReadBufferFromFile

The other two red rows in that report are effects of this one crash, not separate failures:

  • "Hung check failed": the hung check got only Connection refused because the server had already died.
  • "Fatal message in log": the same Logical error: 'inserted' line.

Why it's not this PR's code:

  • The PR touches nothing under src/Interpreters/Cache/ or src/Disks/IO/. The only server-side change is the path != "/" prefix guard in S3ObjectStorage::listObjects, which is not on this stack. The rest is CI/docker/tests plus the brotli bump and its gtest.
  • The assertion is chassert(inserted) in CleanupQueue::add. Upstream deleted it in Sept 2023 as "Remove incorrect assertion" (kssenii, commit d368dda185c, part of Add assertion ClickHouse/ClickHouse#54985, merged into master after the 23.8 branch point). It was never backported, so releases/23.8.16 still has it. chassert only aborts in debug/sanitizer builds, so release binaries are not affected.
  • It also failed the same way on this branch's earlier run on 296f2e9 (00:10:34 ... Logical error: 'inserted'). So it is reproducible on 23.8 stress/asan, not a one-off.

Caveat: releases/23.8.16 has no stress run of its own to compare against. Earlier ASan stress runs on this branch also used the new CI images, so I can't show it failing before this PR. The likely reason it surfaces now is that the refreshed stress environment (minio mirror, new images) exercises the S3 + filesystem-cache path more. That is a guess. The code-level evidence above is solid: the asserted invariant is one upstream itself judged incorrect.

Suggested fix: backport the one-line removal of chassert(inserted) (plus its comment) from d368dda185c into src/Interpreters/Cache/Metadata.cpp. You could add it to this PR, or do it in a small separate 23.8 PR. The code right after the assertion already handles !inserted correctly (if (inserted) { ... }). If you prefer not to change server code in this infra PR, this failure is safe to treat as known/pre-existing for 23.8.


I didn't push anything. Just ask if you want me to add the assertion-removal backport to this branch (or open it as a separate blau/* PR). Either way, CI on that commit would confirm the fix.

@zvonand
zvonand merged commit 855bd6e into releases/23.8.16 Oct 28, 2026
291 of 296 checks passed
@zvonand zvonand added the verified Approved for release label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

23.8 23.8.16 23.8.16.10044 cicd Improvements and fixes to the CICD process stable verified Approved for release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants