Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ All notable changes to RustFS Operator are documented in this file. The format i

### Added

- Tenant `spec.readinessProbe` with optional initial delay, timeout, period, failure
threshold, and success threshold settings, preserved during reconciliation.
- Full mutual TLS (client-to-Pod and Pod-to-Pod) support when `caTrust.clientCaSecretRef` is
configured with cert-manager, including client certificate/key volume projections, automatic
`client auth` usage on managed default certificates, mTLS-aware container probes, and operator
Expand Down Expand Up @@ -58,6 +60,16 @@ All notable changes to RustFS Operator are documented in this file. The format i
opt-out on Tenants whose budgets you manage yourself, or remove your own budgets that select
pool Pods, because Kubernetes rejects evictions of Pods selected by more than one budget.

### Changed

- The default Tenant readiness probe timeout is five seconds, increased from the
previous effective one-second default. Upgrades update existing StatefulSet Pod
templates. Tenants without `spec.serviceAccountName` use enforced `RollingUpdate`
with partition zero for default ServiceAccount token protection; Tenants with
a custom ServiceAccount retain their existing update strategy and partition.
Apply the updated Tenant CRD explicitly before upgrading the Operator or
configuring readiness fields; Helm does not upgrade installed CRDs.

### Fixed

- Check an existing Tenant bucket with HeadBucket before CreateBucket, so a bucket under
Expand Down
29 changes: 29 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,35 @@ Optional quick compile (not a separate `make` target):
cargo check --all-targets
```

The readiness field-management and rollout tests use a real Kubernetes API and
are ignored by default. Run them against a disposable cluster with the current
Tenant CRD:

```bash
kind create cluster --name rustfs-readiness --kubeconfig /tmp/rustfs-readiness.kubeconfig
export KUBECONFIG=/tmp/rustfs-readiness.kubeconfig
export RUSTFS_READINESS_TEST_CONTEXT=kind-rustfs-readiness
kubectl --context "$RUSTFS_READINESS_TEST_CONTEXT" apply --server-side -f deploy/rustfs-operator/crds/tenant-crd.yaml
cargo test --lib readiness_ssa_live_preserves_values_ownership_and_external_fields -- --ignored
docker pull python:3.12-slim
kind load docker-image python:3.12-slim --name rustfs-readiness
cargo test --lib readiness_rollout_live_updates_complete_templates -- --ignored
kind delete cluster --name rustfs-readiness
```

Both tests create and delete isolated namespaces. The field-management test uses
zero-replica StatefulSets and checks
readiness timing ownership, manual configuration migration, and preservation of
external rollout policies and unrelated field ownership.

The rollout test uses two Python HTTP server Pods with `imagePullPolicy: Never`,
so `python:3.12-slim` must be available on every node. It submits readiness timing
and server configuration changes through the production update helper, observes
all stored StatefulSet templates and Pod events, and waits for both updated Pods
to become Ready without manual Pod deletion. It exercises `Parallel` and
`OrderedReady` policies with an external manager sharing the readiness timeout.
Neither test requires Operator or RustFS container images.

### 🔒 Git hooks (optional)

The repository does **not** ship a `make setup-hooks` target. To run checks automatically on `git commit`, add your own `.git/hooks/pre-commit` that invokes `make pre-commit` (or the individual commands above).
Expand Down
35 changes: 35 additions & 0 deletions deploy/rustfs-operator/crds/tenant-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3158,6 +3158,41 @@ spec:
priorityClassName:
nullable: true
type: string
readinessProbe:
description: Readiness timing for every RustFS Pod. Omitted fields use operator defaults.
nullable: true
properties:
failureThreshold:
description: Consecutive failures required to report NotReady. Defaults to 3.
format: int32
minimum: 1.0
nullable: true
type: integer
initialDelaySeconds:
description: Seconds before readiness checks begin. Defaults to 0.
format: int32
minimum: 0.0
nullable: true
type: integer
periodSeconds:
description: Interval between readiness checks, in seconds. Defaults to 10.
format: int32
minimum: 1.0
nullable: true
type: integer
successThreshold:
description: Consecutive successes required to report Ready. Defaults to 1.
format: int32
minimum: 1.0
nullable: true
type: integer
timeoutSeconds:
description: Timeout for each readiness check, in seconds. Defaults to 5.
format: int32
minimum: 1.0
nullable: true
type: integer
type: object
rpcSecret:
description: |-
Optional Secret key selector for RustFS internode RPC authentication.
Expand Down
36 changes: 36 additions & 0 deletions deploy/rustfs-tenant/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ has no `appVersion`: it does not select a RustFS image through chart metadata.
| `tenant.metadata.name` | Tenant name; defaults to release name; DNS-1035, max 55 characters |
| `tenant.metadata.labels`, `annotations` | Native metadata maps for the Tenant |
| `tenant.spec` | Native Tenant spec, rendered without `tpl` evaluation or environment injection |
| `tenant.spec.readinessProbe` | Optional readiness timing fields; the Operator supplies defaults and the health endpoint |
| `ingress.api`, `ingress.console` | Independent Ingress settings; disabled by default |
| `ingress.*.hosts` | Explicit DNS hostnames; no catch-all rule |
| `ingress.*.ingressClassName`, `tls` | Native Ingress fields; TLS uses existing Secrets |
Expand All @@ -101,6 +102,41 @@ Keep the complete desired lists in your values file. Clearing a default map uses
source. The installed Tenant CRD remains the authority for validating native spec
fields; chart schema validation covers the chart-owned settings.

### Readiness timing

Configure readiness through the native Tenant spec in your values file:

```yaml
tenant:
spec:
readinessProbe:
initialDelaySeconds: 0
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
```

These are the Operator defaults. Each field is optional, so a values file can
override only `timeoutSeconds`, for example. The Operator selects the readiness
handler from Tenant TLS: HTTP/HTTPS uses `/health/ready` on port `9000`, while
mTLS uses the authenticated exec probe or configured TCP probe. The five-second
timeout is the time Kubernetes allows for a probe response; the mTLS exec
command retains its separate two-second curl timeout, and server-side readiness
work has its own time budgets.

Install the updated Tenant CRD explicitly before applying these values; Helm
does not upgrade CRDs from the Operator chart's `crds/` directory. Upgrading the
Operator also changes existing Tenants' default readiness timeout from one to
five seconds, which updates their StatefulSet Pod templates. When
`spec.serviceAccountName` is omitted, the Operator enforces `RollingUpdate` with
`rollingUpdate.partition: 0` for default ServiceAccount token protection, and
eligible Pods are replaced automatically. With an explicit custom ServiceAccount,
the Operator preserves the existing `RollingUpdate`, `rollingUpdate.partition`,
or `OnDelete` settings.
See [readiness timing and upgrade guidance](../../docs/operator-user-guide.md#readiness-probe-timing)
for validation rules, rollout behavior, and availability checks.

## Upgrade and remove

Keep the release name, namespace and Tenant name stable. Renaming a Tenant creates
Expand Down
8 changes: 8 additions & 0 deletions deploy/rustfs-tenant/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ tenant:
spec:
credsSecret:
name: rustfs-credentials
# Optional timing overrides; omitted fields use these Operator defaults.
# The Operator selects the readiness handler from the Tenant TLS configuration.
# readinessProbe:
# initialDelaySeconds: 0
# timeoutSeconds: 5
# periodSeconds: 10
# failureThreshold: 3
# successThreshold: 1
pools:
- name: pool-0
servers: 1
Expand Down
61 changes: 61 additions & 0 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,7 @@ Useful Tenant-level fields:
| `createServiceAccountRbac` | Deprecated compatibility field; ignored. Manage any custom ServiceAccount RBAC explicitly. |
| `priorityClassName` | Tenant-level priority class. |
| `lifecycle` | Kubernetes container lifecycle hooks. |
| `readinessProbe` | Optional readiness timing overrides for every RustFS Pool. |
| `podManagementPolicy` | StatefulSet pod management policy. |
| `podDeletionPolicyWhenNodeIsDown` | Node-down pod deletion behavior. |
| `disablePodDisruptionBudget` | Set to `true` to stop creating per-pool PodDisruptionBudgets and delete the ones the operator created. |
Expand Down Expand Up @@ -768,6 +769,66 @@ Multi-pool Tenants cannot contain a single-node single-disk pool. The Operator r

Force deletion can have data consistency implications. Use it only when the storage backend and operational procedure are designed for that failure mode. Force deletion requires the Node object to be deleted or marked with an effective `node.kubernetes.io/out-of-service` taint that the target Pod does not tolerate, so volume detach fencing is explicit. Before using force policies, confirm the node is powered off or otherwise isolated; deleting the Node object is treated as that operational assertion.

#### Readiness probe timing

Set `spec.readinessProbe` to configure the readiness probe on every RustFS
container in the Tenant. Each field is optional; omitted fields use the defaults
below.

| Field | Default | Minimum | Meaning |
| --- | --- | --- | --- |
| `initialDelaySeconds` | `0` | `0` | Delay before readiness checks begin. |
| `timeoutSeconds` | `5` | `1` | Time Kubernetes allows for each probe response. |
| `periodSeconds` | `10` | `1` | Interval between readiness checks. |
| `failureThreshold` | `3` | `1` | Consecutive failures required to mark the container unready. |
| `successThreshold` | `1` | `1` | Consecutive successes required to become ready after a failure. |

```yaml
spec:
readinessProbe:
initialDelaySeconds: 0
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
```

The Operator selects the probe handler from the Tenant TLS configuration.
HTTP/HTTPS readiness probes use `/health/ready` on port `9000`. mTLS readiness
uses the existing authenticated exec probe or configured TCP probe. The five
timing fields apply to each readiness handler. The mTLS exec command retains
its separate two-second curl timeout. Startup and liveness retain their existing
HTTP or TCP handlers. Kubernetes stops routing Service traffic to a Pod when it
becomes unready; a readiness failure alone does not restart its container.

The five-second default gives distributed health checks more time to respond
during a peer failure. It is the Kubernetes probe timeout, not an upper bound on
the RustFS server's entire readiness handler. Choose timing values using measured
probe durations during normal operation and node failures, alongside the required
failure detection time. A successful readiness check reports the server's readiness
decision; verify S3 reads and writes separately when testing availability.

Apply the updated Tenant CRD before using these fields or upgrading the Operator,
following [the upgrade instructions](#4-install-the-operator). Helm does not
upgrade an installed CRD from a chart's `crds/` directory.

For existing Tenants that omit `readinessProbe`, upgrading changes the generated
readiness timeout from Kubernetes' previous effective default of `1` second to
`5` seconds. This changes the StatefulSet Pod template. When
`spec.serviceAccountName` is omitted, the Operator enforces `RollingUpdate` with
`rollingUpdate.partition: 0` as part of its default ServiceAccount token
protection, so eligible Pods are replaced automatically. With an explicit custom
ServiceAccount, the Operator preserves the existing StatefulSet update strategy:
`RollingUpdate` updates eligible Pods, `rollingUpdate.partition` limits which
ordinals are updated, and `OnDelete` applies the new template when Pods are
manually replaced. Plan the update according to the cluster's available quorum.
Set `spec.readinessProbe.timeoutSeconds: 1` explicitly if the deployment requires
the previous timeout.

Keep the desired settings in the Tenant resource. Reconciliation restores them
after a manual StatefulSet probe change. Removing an override restores that
field's default and can also change the Pod template.

#### Pod disruption budgets

The Operator creates one PodDisruptionBudget per pool, named after the pool StatefulSet and selecting the pool Pods. `minAvailable` is the pool's server count minus the number of servers that can be unavailable at once while every erasure set keeps write quorum for the `STANDARD` storage class. It is computed from the drive placement RustFS reports through the admin API:
Expand Down
66 changes: 66 additions & 0 deletions src/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ use crate::types;
use crate::types::v1alpha1::encryption::LocalKmsMasterKeySecretRef;
use crate::types::v1alpha1::tenant::{OidcExtraCaCertSecretRef, RpcSecretRef, Tenant};
use k8s_openapi::NamespaceResourceScope;
use k8s_openapi::api::apps::v1::StatefulSet;
use k8s_openapi::api::core::v1::Secret;
use k8s_openapi::apimachinery::pkg::apis::meta::v1::{Status, StatusCause};
use kube::api::{DeleteParams, ListParams, ObjectList, Patch, PatchParams, PostParams};
use kube::runtime::events::{Event, EventType, Recorder, Reporter};
use kube::{Resource, ResourceExt, api::Api};
Expand All @@ -30,6 +32,22 @@ use std::io::Cursor;
use std::path::{Component, Path, PathBuf};
use tracing::info;

#[derive(Debug, Snafu)]
#[snafu(display("Kubernetes StatefulSet apply error: {}", source))]
pub(crate) struct StatefulSetApplyError {
pub(crate) source: Box<kube::Error>,
pub(crate) causes: Vec<StatusCause>,
}

impl From<kube::Error> for StatefulSetApplyError {
fn from(source: kube::Error) -> Self {
Self {
source: Box::new(source),
causes: Vec::new(),
}
}
}

#[derive(Debug, Snafu)]
pub enum Error {
#[snafu(display("Kubernetes API error: {}", source))]
Expand Down Expand Up @@ -696,6 +714,54 @@ impl Context {
.await
}

/// Preserves structured SSA conflict causes omitted by kube's ErrorResponse.
pub(crate) async fn apply_statefulset(
&self,
resource: &StatefulSet,
namespace: &str,
params: &PatchParams,
) -> Result<StatefulSet, StatefulSetApplyError> {
let api: Api<StatefulSet> = Api::namespaced(self.client.clone(), namespace);
let request = kube::core::Request::new(api.resource_url())
.patch(&resource.name_any(), params, &Patch::Apply(resource))
.map_err(kube::Error::BuildRequest)?;
let response = self
.client
.send(request.map(kube::client::Body::from))
.await?;
let response_status = response.status();
let body = response.into_body().collect_bytes().await?;
if response_status.is_success() {
return serde_json::from_slice(&body)
.map_err(kube::Error::SerdeError)
.map_err(Into::into);
}

let error_response = serde_json::from_slice::<kube::core::ErrorResponse>(&body)
.unwrap_or_else(|_| kube::core::ErrorResponse {
status: response_status.to_string(),
message: String::from_utf8_lossy(&body).into_owned(),
reason: "Failed to parse error data".to_string(),
code: response_status.as_u16(),
});
let causes = if response_status == http::StatusCode::CONFLICT
&& error_response.code == 409
&& error_response.reason == "Conflict"
{
serde_json::from_slice::<Status>(&body)
.ok()
.and_then(|status| status.details)
.and_then(|details| details.causes)
.unwrap_or_default()
} else {
Vec::new()
};
Err(StatefulSetApplyError {
source: Box::new(kube::Error::Api(error_response)),
causes,
})
}

/// Force-applies a narrow security patch without taking ownership of the full resource.
pub(crate) async fn force_apply_security_fields<T, P>(
&self,
Expand Down
3 changes: 3 additions & 0 deletions src/reconcile.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ mod phases;
mod pod_disruption_budget;
mod pool_lifecycle;
mod provisioning;
mod readiness;
mod tls;

const OUT_OF_SERVICE_TAINT_KEY: &str = "node.kubernetes.io/out-of-service";
Expand Down Expand Up @@ -971,6 +972,7 @@ pub fn error_policy(object: Arc<Tenant>, error: &Error, _ctx: Arc<Context>) -> A
types::error::Error::ImmutableFieldModified { .. }
| types::error::Error::InvalidTenantName { .. }
| types::error::Error::InvalidAdditionalVolumeSpec { .. }
| types::error::Error::InvalidReadinessProbe { .. }
| types::error::Error::KmsMigrationBlocked { .. }
| types::error::Error::InvalidWorkloadSecurityProfile { .. }
| types::error::Error::WorkloadSecurityIncompatible { .. }
Expand Down Expand Up @@ -1028,6 +1030,7 @@ fn reconcile_error_reason(error: &Error) -> &'static str {
types::error::Error::InvalidAdditionalVolumeSpec { .. } => {
"InvalidAdditionalVolumeSpec"
}
types::error::Error::InvalidReadinessProbe { .. } => "InvalidReadinessProbe",
types::error::Error::ImmutableFieldModified { .. } => "ImmutableFieldModified",
types::error::Error::PoolDeleteBlocked { .. } => "PoolDeleteBlocked",
types::error::Error::KmsMigrationBlocked { .. } => "KmsMigrationBlocked",
Expand Down
Loading