Skip to content

CKS: reconcile node capacity after live resize - #14223

Open
andrijapanicsb wants to merge 11 commits into
mainfrom
cks-live-resize-capacity-reconciliation
Open

andrijapanicsb wants to merge 11 commits into
mainfrom
cks-live-resize-capacity-reconciliation

Conversation

@andrijapanicsb

@andrijapanicsb andrijapanicsb commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Context

#13226 added live compute-offering changes for CKS nodes, building on #12975. The VM could gain CPU/RAM while Kubernetes still reported the old capacity. Operators had to activate guest resources and restart kubelet manually. #14213 tracks that gap.

Changes

CKS now activates and verifies the added guest CPU/RAM, restarts kubelet when needed, and waits for the Node to report the new capacity and allocatable resources before completing the scale operation.

Nodes are handled one at a time without draining resident Pods. An existing operator cordon is preserved. If reconciliation fails after resize, the node stays CKS-cordoned; retry finishes that node before continuing with the others.

QA also exposed a pre-existing cluster CPU/RAM accounting bug, not a regression from this PR. Totals are now calculated from the configured per-node-type offerings and counts, so the API and UI reflect mixed control/worker offerings correctly.

Functional and regression testing

Test environment: Cloud-managed KVM, isolated network, published packages from f92e52d736b9.

Test Result
Two-worker CPU-only, RAM-only and combined live increases (API) Guest and Kubernetes capacity/allocatable updated correctly
Control CPU increase (API) and RAM increase (UI) Automatic reconciliation; UI scale task successful
Injected kubelet failure and retry (API) Expected failure left the first node cordoned and second untouched; retry completed both; temporary fault cleaned
Operator cordon, cap-only and same-offering requests Cordon preserved; no unnecessary kubelet restart
Stop/start, stopped offering change, worker scale-out/in and live migration Passed; scale-in used worker count and the current worker offering
Invalid parameters and insufficient CPU capacity Correctly rejected without resource changes
Final UI Refresh, API, guest and Kubernetes checks Correct 6 CPU / 7168 MiB total; both Nodes Ready; no VM reboot; 17 Pod UIDs/restart counts unchanged; HTTP/DNS passed

VPC/direct-access and multi-control HA were not tested. Dedicated etcd live resize remains unsupported. The separate existing nodeids-only scale-in issue found during QA is tracked in #14458, with its fix in #14459.

UI evidence

Successful scale task and correct aggregate capacity:

Successful UI scale and correct CPU/RAM total

Scale request and result after Refresh

Control RAM: 2048 → 3072 MiB; worker unchanged at 3 CPU / 4096 MiB.

Control RAM-only scale request

After Refresh: 6 CPU / 7168 MiB.

Capacity after UI Refresh

@codecov

codecov Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 59.20000% with 102 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.47%. Comparing base (af9764b) to head (f92e52d).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...er/actionworkers/KubernetesClusterScaleWorker.java 26.54% 76 Missing and 7 partials ⚠️
...s/KubernetesClusterNodeCapacityReconcilerImpl.java 84.87% 11 Missing and 7 partials ⚠️
...utils/KubernetesClusterNodeCapacityReconciler.java 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14223      +/-   ##
============================================
+ Coverage     20.45%   20.47%   +0.02%     
- Complexity    21415    21520     +105     
============================================
  Files          6487     6489       +2     
  Lines        589303   589540     +237     
  Branches      72245    72307      +62     
============================================
+ Hits         120525   120733     +208     
- Misses       455643   455646       +3     
- Partials      13135    13161      +26     
Flag Coverage Δ
uitests 3.70% <ø> (ø)
unittests 21.78% <59.20%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@andrijapanicsb andrijapanicsb changed the title CKS: reconcile node capacity after live resize (WIP)CKS: reconcile node capacity after live resize Sep 22, 2026
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

ACSHomeBot commented Sep 22, 2026 •

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform
PASS RPM EL (EL8/9/10)
PASS DEB Ubuntu, Debian

Test packages are available at:

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🔴 Test Coverage Grade: D — Marginal

Metric Value
Line coverage 25.53%
Branch coverage 19.89%

Grade Scale

Grade Line Coverage Meaning
🟢 A ≥ 80% Excellent - this code sleeps well at night 😴
🟡 B 60-79% Good - almost there, don't stop now 😉
🟠 C 40-59% Acceptable - your code is wearing a seatbelt, but no airbags 😬
🔴 D 20-39% Marginal - boldly shipping where no test has gone before 🖖
⛔ F < 20% Failing - tests? what tests? 🔥

Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run

@andrijapanicsb
andrijapanicsb force-pushed the cks-live-resize-capacity-reconciliation branch from 40b50b0 to 7e5a95a Compare October 20, 2026 05:18
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

Copy link
Copy Markdown

Accepted: package kvm G for 548011910846.

Planned: RPM + DEB, KVM SystemVM.

Status: Queued for worker G. Jobs ahead: 0.

@ACSHomeBot

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform Build Time
NOT RUN RPM EL (EL8/9/10) n/a
NOT RUN DEB Ubuntu, Debian n/a

Package building did not start. No repository was published.

Status: Infrastructure failure before package building started.

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

Copy link
Copy Markdown

Accepted: package kvm G for 2a404aa1ffcc.

Planned: RPM + DEB, KVM SystemVM.

Status: Queued for worker G. Jobs ahead: 0.

@ACSHomeBot

Copy link
Copy Markdown

Package building started on worker G.

@ACSHomeBot

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform Build Time
PASS RPM EL (EL8/9/10) 31 min
PASS DEB Ubuntu, Debian 22 min

Test packages are available at:

@sonarqubecloud

Copy link
Copy Markdown

@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@ACSHomeBot package G

@ACSHomeBot

Copy link
Copy Markdown

Accepted: package kvm G for f92e52d736b9.

Planned: RPM + DEB, KVM SystemVM.

Status: Queued for worker G. Jobs ahead: 0.

@ACSHomeBot

Copy link
Copy Markdown

Package building started on worker G.

@ACSHomeBot

Copy link
Copy Markdown

Packaging results:

Result Artifact Platform Build Time
PASS RPM EL (EL8/9/10) 30 min
PASS DEB Ubuntu, Debian 22 min

Test packages are available at:

@andrijapanicsb andrijapanicsb changed the title (WIP)CKS: reconcile node capacity after live resize CKS: reconcile node capacity after live resize Oct 21, 2026
@andrijapanicsb
andrijapanicsb marked this pull request as ready for review October 21, 2026 20:09
@andrijapanicsb andrijapanicsb added this to the 24.0 milestone Oct 21, 2026
@andrijapanicsb

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19660

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants