Skip to content

feat(tenant): support configurable readiness probe timing - #296

Merged
GatewayJ merged 1 commit into
rustfs:mainfrom
GatewayJ:feat/tenant-readiness-probe
Nov 12, 2026
Merged

GatewayJ merged 1 commit into
rustfs:mainfrom
GatewayJ:feat/tenant-readiness-probe

Conversation

@GatewayJ

@GatewayJ GatewayJ commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Type of Change

  • New Feature
  • Bug Fix
  • Documentation
  • Performance Improvement
  • Test/CI
  • Refactor
  • Other:

Related Issues

Refs #295

Summary of Changes

Operator-generated readiness probes currently inherit Kubernetes' one-second timeout, and Tenant configuration cannot adjust it. Add optional spec.readinessProbe timing settings so the reported five-second workaround can be maintained through normal reconciliation.

  • Expose initialDelaySeconds, timeoutSeconds, periodSeconds, failureThreshold, and successThreshold, with defaults 0/5/10/3/1 and Kubernetes-compatible minimum values. The Operator continues to select the readiness handler according to Tenant TLS, including mTLS exec and TCP probes.
  • Validate the configuration in the CRD and controller, and detect effective readiness changes on existing StatefulSets, including removal of overrides and historical Kubernetes defaults.
  • Apply readiness timing and all other Pod template changes atomically. Retry with force only when Kubernetes reports conflicts exclusively on the five readiness timing fields, using the same complete request, UID, and resourceVersion. Preserve conflicts on unrelated fields and existing lifecycle/update validation.
  • Compare only Operator-managed readiness fields so externally managed HTTP headers and host settings remain intact and reconciliation converges. Cover mTLS exec and TCP handlers, timing overrides, and override removal.
  • Keep structured StatefulSet apply conflict causes with a boxed Kubernetes error source so the error return type passes Clippy.
  • Update the generated CRD, native Tenant Helm examples, upgrade documentation, changelog, and tests. Configuration rules remain in Tenant types, rendering/comparison in workloads, update coordination in reconcile, and API transport in Context.

Checklist

  • I have read and followed the CONTRIBUTING.md guidelines
  • Passed make pre-commit (fmt-check + clippy + test + console-lint + console-fmt-check)
  • Added/updated necessary tests
  • Documentation updated (if needed)
  • CHANGELOG.md updated under [Unreleased] (if user-visible change)
  • CI/CD passed (if applicable)

Impact

  • Breaking change (CRD/API compatibility)
  • Requires doc/config/deployment update
  • Other impact: existing Tenants that omit readiness settings receive a five-second timeout and an updated StatefulSet Pod template.

The API addition is optional. Install the updated Tenant CRD explicitly before upgrading or using the new settings; Helm does not upgrade CRDs from the chart's crds/ directory. Tenants without spec.serviceAccountName use enforced RollingUpdate with partition zero for default ServiceAccount token protection. Tenants with a custom ServiceAccount retain their existing update strategy and partition. Plan upgrades around available cluster quorum. Set spec.readinessProbe.timeoutSeconds: 1 to retain or restore the previous timeout.

Verification

RUSTUP_TOOLCHAIN=1.99.0 make -j1 pre-commit
cargo test --lib readiness_ssa_live_preserves_values_ownership_and_external_fields -- --ignored --nocapture
cargo test --lib readiness_rollout_live_updates_complete_templates -- --ignored --nocapture
  • The current revision passes the full local gate with Rust 1.99.0 and serial Make targets: release metadata, Rust formatting, Clippy, unit/integration tests, e2e harness checks, and frontend tests/lint/build/format checks. An unchanged leader-election timing test failed during the parallel run; its isolated rerun and the complete serial gate passed.
  • Previously ran the opt-in tests against a dedicated Kubernetes v1.35.1 Kind cluster; these live checks were not rerun for the current update. API tests verify historical defaults, manual 5 -> Tenant 5 -> Tenant 8 -> default 5, shared ownership, external HTTP fields, custom-ServiceAccount update strategy preservation, unrelated image conflicts without partial changes, and concurrent updates or same-name replacement between the rejected ordinary apply and force retry.
  • Python HTTP server Pod regression tests cover Parallel and OrderedReady, changing probe response delay and timeout together under external timing ownership. Watch events verify complete templates, and all replacement Pods must reach Ready at the final revision. CONTRIBUTING.md documents the context, image, and commands.
  • Kubernetes server-side dry-run accepted the five parameter boundaries and a legacy Tenant, and rejected each below-minimum value: 11 checks passed.
  • Focused tests cover runtime rejection/status, HTTP/HTTPS rendering, override removal, default normalization, full-template request contents, strict structured conflict filtering, identity preconditions, retry failure propagation, external-field convergence, and Helm values reaching generated Pod probes.
  • Reviewed the boxed error propagation and the integration with current main: mTLS handlers remain intact, timing comparison converges, and external HTTP fields remain unmanaged.

Additional Notes

This addresses the Operator configuration and reconciliation gap described in #295. The reporter's Nginx HTTP 404 cause and end-to-end read/write availability during the original RKE2/Longhorn node reboot remain unverified. A five-second kubelet probe timeout does not bound all server-side readiness work. The existing mTLS exec command also retains its separate two-second curl timeout. The issue remains open for that investigation.

CI results are pending for the updated commit.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T17:08:45.169036Z 7189a84 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@GatewayJ
GatewayJ force-pushed the feat/tenant-readiness-probe branch from ac74d35 to 28de289 Compare November 12, 2026 02:20
@GatewayJ
GatewayJ added this pull request to the merge queue Nov 12, 2026
Merged via the queue into rustfs:main with commit ab9c599 Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant