diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index 7599b3e..0d38f7f 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -6,67 +6,60 @@ sidebar_label: Architecture # Architecture overview -This page is a single-screen-ish view of how the schedkit components fit together. If the Concepts pages on `sched_ext` and OCI explained the *what*, this one explains the *where*. +schedkit has two tools for running OCI-packaged `sched_ext` CPU schedulers: schedctl on a host, and sked in Kubernetes. Both run scheduler images, but they manage them independently. -## The two layers +## On one host: schedctl -schedkit is, deliberately, two pieces of software at two different layers. +schedctl talks to a local Podman or containerd runtime. It verifies scheduler images, starts and stops containers, and provides host diagnostics and status commands. -**schedctl** is a host-level CLI. It runs on a single machine, talks to a local container runtime (Podman or containerd), and is responsible for verifying the scheduler image's signature, pulling it, starting the binary in a privileged container, and managing its lifecycle. It has no opinion about whether your machine is part of a cluster. As far as schedctl is concerned, it's just a Linux box. - -**sked** is a Kubernetes operator. It exposes a CRD (`SchedExt`) that names the scheduler image you want to run, and creates a privileged DaemonSet that runs that image on every node the DaemonSet's pod-scheduling rules allow. The kubelet on each node pulls the image and starts the container; the kernel takes care of attaching `sched_ext` from there. - -The two projects are aligned in design — both run OCI-packaged sched_ext schedulers — but **independent in implementation today**. sked does not invoke schedctl, share its trust policy, or use its kernel preflight. A future version of the operator could plausibly use schedctl as the in-pod runtime (and pick up its signature verification and discrepancy reporting for free), but that integration does not exist now. - -## A diagram, in words - -On a single machine, the chain looks like this: - -``` -user → schedctl CLI → container runtime (Podman/containerd) → scheduler binary → sched_ext (kernel) +```text +user -> schedctl -> Podman/containerd -> scheduler binary -> sched_ext ``` -In a Kubernetes cluster, sked is the user-facing interface and the chain is shorter than you might expect: - -``` -cluster admin - ↓ -SchedExt CRD (in the API server) - ↓ -sked controller (one replica, control plane) - ↓ -DaemonSet (one per SchedExt) - ↓ -kubelet → container runtime → scheduler binary → sched_ext (per node) +The CLI does not require Kubernetes. Use it for a workstation, a standalone server, or host management driven by scripts. See [schedctl](../schedctl/overview). + +## In Kubernetes: sked + +sked watches cluster-scoped `Scheduler` resources. Each names a scheduler image and can select a node pool, tolerate taints, and configure container arguments, environment, pull secrets, and resources. + +```text +cluster administrator + | + v +API server <-> sked admission webhook <-> trust policy + image registry + | + v +Scheduler resource + | + v +sked leader: assign nodes, verify image, reconcile workload + | + v +DaemonSet in the configured workload namespace + | + v +kubelet -> container runtime -> scheduler binary -> sched_ext ``` -There is no per-node sked component. The kubelet runs the scheduler pod; the pod runs privileged; the binary inside attaches `sched_ext` directly. schedctl is not in this picture. - -## What's in each repository - -| Repository | Layer | Language | Role | -| --- | --- | --- | --- | -| [schedkit/schedctl](https://github.com/schedkit/schedctl) | Host | Go | CLI, container runtime client, scheduler lifecycle, cosign verification, kernel preflight | -| [schedkit/sked](https://github.com/schedkit/sked) | Cluster | Go | kubebuilder operator: `SchedExt` CRD, controller that creates a privileged DaemonSet per resource | -| [schedkit/plumbing](https://github.com/schedkit/plumbing) | — | JSON | Catalog manifest (mapping short names to image URIs) consumed by `schedctl list`/`run`/`versions` | -| [schedkit/docs](https://github.com/schedkit/docs) | — | TypeScript | This site | - -The two code repositories are independently versioned and currently independent in their wire-level concerns: sked does not import schedctl as a Go module, nor does it shell out to it. Aligning their lifecycle and trust models is a deliberate longer-term direction; today they coexist rather than co-operate. +The [Helm chart](../sked/installation) installs two controller replicas with leader election and a PodDisruptionBudget. Both replicas serve admission webhooks; only the leader reconciles. The kubelet runs the privileged scheduler pods directly. There is no per-node sked agent and no schedctl invocation. -## Why the layers are separate +sked verifies signatures at admission and during reconciliation, then places the verified digest in the DaemonSet. It resolves overlapping node selections by creation time and reports DaemonSet rollout counts, conditions, and events. It does not inspect node kernels: a ready pod is not proof that its scheduler attached. -There's a temptation, when designing this kind of system, to do everything in one binary. We didn't, for a few reasons. +See [sked architecture](../sked/architecture) for the reconcile loop and failure handling, and the [CRD reference](../sked/crds) for the API. -The single-machine use case is real and standalone. Plenty of people want to run a different scheduler on their workstation or on a single bare-metal box without any cluster involvement. Forcing them into a Kubernetes operator dependency would be perverse. +## What the tools share -The cluster use case has its own complexity surface — CRD design, controller patterns, RBAC, helm charts — that has nothing to do with `sched_ext` itself. Keeping it in a separate repo means we can iterate on it without disturbing schedctl, which can stay small and focused. +The common input is an OCI-packaged scheduler, not a controller-to-agent protocol. sked neither imports schedctl nor shells out to it. Both implement signature verification, but their trust-policy formats and defaults differ. Configuring one tool does not configure the other. -The boundary between the two is a stable contract (the schedctl CLI), not a private API. If you wanted to write a different orchestrator that drives schedctl from the outside — Nomad, a custom Ansible setup, whatever — schedctl is happy to be driven by it. We've kept the CLI deliberately scriptable for exactly this reason. (sked itself does not yet drive schedctl; it runs scheduler images directly via the kubelet. The contract is open and stable enough for an operator that *did* want to drive schedctl on each node to do so.) +Do not let both manage a scheduler on the same node. sked's conflict handling only sees Scheduler resources; it cannot account for a scheduler started by schedctl or another host process. -## What goes where +Neither tool replaces Kubernetes pod scheduling. They manage Linux CPU schedulers, which allocate CPU time to threads after workloads have been placed on a node. -A rough rule of thumb when deciding which repo a feature belongs in: +## Repositories -- If it's about a single machine, it goes in schedctl. -- If it requires knowing about more than one node, it goes in sked. -- If the answer is "both" — kernel preflight, signature verification, lifecycle/discrepancy reporting — it lives in schedctl today. A longer-term goal is for sked to consume those by driving schedctl on each node, rather than reimplementing them. +| Repository | Role | +| --- | --- | +| [schedkit/schedctl](https://github.com/schedkit/schedctl) | Host CLI, container runtime clients, image verification, and diagnostics. | +| [schedkit/sked](https://github.com/schedkit/sked) | Scheduler CRD, operator, admission validation, trust policy, and Helm chart. | +| [schedkit/plumbing](https://github.com/schedkit/plumbing) | Scheduler image publishing and the catalog mapping names to image references. | +| [schedkit/docs](https://github.com/schedkit/docs) | This documentation site. | diff --git a/docs/intro.md b/docs/intro.md index 2c40102..7723eda 100644 --- a/docs/intro.md +++ b/docs/intro.md @@ -18,7 +18,7 @@ The documentation is split into three sections. **schedctl** documents the host-side command-line tool. If you have a single machine and you want to try out a different scheduler for the duration of a build, a game session, or a test run, this is what you're looking for. -**sked** documents the Kubernetes operator. If your nodes already live in a cluster and you'd like to declare which scheduler runs on them as a Kubernetes resource, this is the layer for that. sked is at an early, deliberately-thin stage today — see its [overview](/docs/sked/overview) for what's actually implemented before planning a rollout. +**sked** documents the Kubernetes operator. It uses cluster-scoped Scheduler resources to select node pools, verifies scheduler images, and reports DaemonSet rollout status. Start with the [overview](/docs/sked/overview) or [install it with Helm](./sked/installation). You can read the sections in order or jump straight to whichever tool you came here for. The Concepts pages are prerequisite-free; the project pages assume you've at least skimmed Concepts but try not to lean on it too hard. diff --git a/docs/sked/architecture.md b/docs/sked/architecture.md index 94500f4..7bfc188 100644 --- a/docs/sked/architecture.md +++ b/docs/sked/architecture.md @@ -5,100 +5,128 @@ title: Architecture # Architecture -This page is for people who want to understand what's actually happening inside sked, either because they're contributing or because they're debugging a behaviour the user-facing docs don't cover. - -If you only want to *use* sked, [Usage](./usage) is enough. Come back here when something is doing things you didn't expect. - -## A single-process operator - -sked is, today, **one binary** running as a Deployment in `sked-system`. There is no per-node agent. There is no controller-to-agent protocol. The operator's only job is to watch `SchedExt` resources and create the corresponding `DaemonSet` for each one. The kubelet on each node then handles starting the scheduler pod, just like any other pod. - -It's a kubebuilder-scaffolded operator using `controller-runtime` (currently v0.19.x). The relevant pieces: - -- `cmd/main.go` — sets up the manager, registers the `SchedExt` scheme, configures metrics/healthz, optionally enables leader election. -- `internal/controller/schedext_controller.go` — the only reconciler. Watches `SchedExt` and writes `DaemonSet`. -- `api/v1/schedext_types.go` — the CRD type. Single field on the spec, empty status struct. - -That's the whole operator. There is no controller-to-agent protocol because there is no agent. - -## The reconcile loop - -The reconciler is small enough to summarise in a few lines. - -```go -func (r *SchedExtReconciler) Reconcile(ctx, req) { - var scx SchedExt - if err := r.Get(ctx, req.NamespacedName, &scx); err != nil { - return // notfound is ignored - } - - ds := &DaemonSet{Name: req.Name, Namespace: req.Namespace} - CreateOrPatch(ctx, ds, func() error { - SetControllerReference(&scx, ds, scheme) - ds.Labels["managed-by"] = "sked-controller" - ds.Spec = DaemonSetSpec{ - Selector: matchLabels{"name": req.Name}, - Template: PodTemplate{ - Labels: {"name": req.Name, "managed-by": "sked-controller"}, - Spec: PodSpec{ - Containers: []Container{{ - Name: "scx", - Image: scx.Spec.Sched, - SecurityContext: {Privileged: true}, - }}, - }, - }, - } - return nil - }) -} +sked is one manager binary built with controller-runtime. The process hosts a Scheduler reconciler, a validating admission webhook, a trust-policy refresher, metrics, and health endpoints. There is no node agent or controller-to-node protocol. + +```text +kubectl create/update Scheduler + | + v +API server ---> sked validating webhook ---> policy + image verification + | + v +cluster-scoped Scheduler + | + v +sked leader ---> node assignment + image verification + | + v +DaemonSet in workload namespace + | + v +kubelet ---> container runtime ---> scheduler binary ---> sched_ext + | + +--- DaemonSet rollout status ---> sked ---> Scheduler status/events ``` -That's it. There's no status update. No conditions. No selectors derived from the resource. The reconciler is invoked on `SchedExt` events; it watches `SchedExt` resources only via `For(&SchedExt{})` and does not own or watch `DaemonSet` events explicitly. (Owner references mean the kubebuilder default does watch DaemonSets owned by SchedExts implicitly, but the reconciler doesn't react to DaemonSet status changes.) +The kubelet runs the scheduler image directly. sked does not install or invoke schedctl. -## What the manager configures +## Processes and availability -The manager wiring in `cmd/main.go` enables a few standard pieces: +Helm and Kustomize deploy two manager replicas by default. The lease `b4c95cb3.schedkit.io` elects one reconciler; both replicas serve admission requests and refresh their trust policy. Extra replicas provide failover and webhook capacity, not parallel controller leaders. -- **Leader election** via `--leader-elect` (off by default). The lease ID is `b4c95cb3.schedkit.io`, in whatever namespace the manager runs. -- **Metrics** on `--metrics-bind-address` (defaults to `0`, i.e. disabled, but the kustomize patch in `config/default` flips it to `:8443`). With `--metrics-secure=true` (default), an authn/authz filter from `controller-runtime/pkg/metrics/filters` is applied. -- **Health probes** on `--health-probe-bind-address` (defaults to `:8081`). Both `/healthz` and `/readyz` are wired to `healthz.Ping`. -- **HTTP/2 disabled by default** for both the metrics and webhook servers (`--enable-http2=false`), as a precaution against the HTTP/2 Stream Cancellation/Rapid Reset CVEs. +The default PodDisruptionBudget keeps one controller available during voluntary evictions. A soft topology-spread rule prefers different nodes but permits co-location. Neither setting protects against all replicas sharing a failed node. -There is no webhook server *content* today. Validating/mutating webhook configurations are scaffolded in `config/webhook/` but commented out in the kustomize entrypoint, so the install doesn't deploy them. +If all controllers stop, existing DaemonSets and scheduler pods keep running. Reconciliation and status updates stop, and Scheduler creates and updates fail because the validating webhook has `failurePolicy: Fail`. Deletion is not handled by the webhook; Kubernetes owner-reference garbage collection does not require sked to be running. -## Communication with nodes +## Admission and image resolution -There is none, in any direct sense. The operator writes to the API server; the kubelets watch their own DaemonSet pods; the kubelets pull the scheduler image and start the container. From the operator's perspective, the work ends when the DaemonSet object is patched. +The API schema validates the image reference. The webhook validates selected spec fields, checks the requesting identity and image repository, and verifies the signature when policy requires it. Admission does not mutate the image reference. -This means: +The reconciler verifies again before writing the workload. It uses the digest it verified, so a tag changing after that verification cannot make the kubelet pull different content. The webhook and reconciler may see different digests if a tag moves between requests; each verifies the digest it resolves. Use a digest in `spec.sched` to fix the requested content. -- Changes to `SchedExt.spec.sched` produce a DaemonSet patch, which produces a DaemonSet rolling update, which produces pod restarts on each node — at whatever cadence the DaemonSet's update strategy permits. The operator does not coordinate the rollout itself. -- Failures on a specific node (image pull failure, BPF verifier rejection, etc.) show up on the *pod* on that node, not on the `SchedExt` resource. `kubectl describe pod` and `kubectl logs` are where you look. -- The operator has no view of `/sys/kernel/sched_ext` on any node. It does not know whether the scheduler successfully attached. +The controller needs registry access as well as Kubernetes API access. [Trust policy](./trust-policy) documents verification, credential handling, policy reloads, and fallback behaviour. -## Failure modes +## Reconciliation -**Controller crash.** Recoverable. The Deployment restarts the manager. Existing DaemonSets keep their pods, so already-attached schedulers stay loaded across the restart. +The controller watches Scheduler objects, Nodes, and owned DaemonSets. Node and Scheduler events enqueue Schedulers so changes to labels or competing resources can change assignments. DaemonSet status updates trigger reconciliation even though they do not change its generation. -**API server unreachable.** Standard `controller-runtime` behaviour: the manager retries with backoff. New `SchedExt` events are queued by the watch's resync; existing scheduler pods on the nodes keep running because their lifecycle is now between the kubelet and the kernel. +For each Scheduler, the reconciler: -**Pull failure on a node.** Surfaces on the pod, not the resource. `kubectl describe pod` shows `ErrImagePull` / `ImagePullBackOff`. +1. Reads the resource and records its generation in status. +2. Lists nodes and Schedulers, then computes conflicts from node selectors. The oldest matching Scheduler wins each node; equal timestamps are ordered by name. +3. Deletes the workload and reports `SchedulerConflict` if it loses every selected node. +4. Otherwise resolves and, by default, verifies the image. +5. Creates or patches a same-named DaemonSet in the workload namespace. It copies container settings from the spec and excludes partially conflicting nodes with required node affinity. +6. Copies DaemonSet counts into status, sets conditions, and records events when the reported state changes. -**Scheduler fails to attach (BPF verifier, missing CONFIG_*, etc.).** The pod stays running (the entrypoint usually exits non-zero, which CrashLoops the pod) but the kernel does not have a scheduler attached. There's no signal back to the operator. `kubectl logs` on the pod is the only signal. +Conflict arbitration does not consider taints, tolerations, image validity, pod health, or schedulers started outside sked. A blocked older Scheduler does not automatically yield to a healthy newer one. See [overlapping selectors](./crds#overlapping-selectors). -**Two `SchedExt` resources whose pods land on the same node.** The kernel attaches one and refuses the other. The losing pod will crashloop. Avoid this with cluster-level scheduling rules (taints, labels) until the CRD grows node selection. +The controller replaces the generated DaemonSet spec on reconciliation. Manual changes to its pod template are not a supported configuration path. Owner references connect Scheduler to DaemonSet to pods; there is no custom cleanup finalizer. -**Missing RBAC for DaemonSets.** The shipped ClusterRole grants verbs on `schedexts*` but does not currently include `apps/daemonsets`. On a strict cluster, the controller will get `forbidden` from the apiserver when patching DaemonSets. The fix is a kustomize patch extending the ClusterRole. There's a tracked issue for adding this to the generated manifests. +### Rollouts and status -## What sked is *not* doing +A change to the generated pod template triggers Kubernetes' default DaemonSet rolling update. sked observes that rollout; it does not implement a separate rollout engine, atomic scheduler handoff, or automatic rollback. -Worth being explicit about a few things, because they come up: +`status.nodes` and the `Ready`, `Progressing`, and `Degraded` conditions come from DaemonSet status. `Active` records whether the Scheduler is fully blocked by conflicts. Partial conflicts can produce both `Ready=True` and `Degraded=True`: the remaining workload is ready, but some requested nodes belong to another Scheduler. -- It's not a Kubernetes scheduler. The Kubernetes scheduler decides which pod runs on which node. sked decides which `sched_ext` scheduler the *kernel* on each node uses to multiplex CPU between threads. The two operate at completely different layers. -- It's not a service mesh, sidecar, or admission controller. The DaemonSet pod does not interpose on workload pods. -- It's not collecting workload telemetry. There is no logic to choose schedulers based on observed workload behaviour; the choice is exactly what you put in `spec.sched`. -- It does not invoke `schedctl`. The host CLI and the operator are independent code paths today. -- It does not verify image signatures. If signature verification is a requirement for your cluster, configure it at admission time (e.g. with `policy-controller`). +There is no direct kernel observation. A container that stays alive after failing to attach can appear ready. Conversely, image-pull failures and crashes appear through pod availability, without a kernel-specific diagnosis. Use pod logs and node kernel state to establish the cause. -If you want any of those things, sked is not the right tool yet, and we'd rather you knew that than tried to make it fit. +Events for cluster-scoped Schedulers are stored in `default`, not the workload namespace. Normal reasons include `RolloutInProgress`, `DaemonSetReady`, and `NoNodesScheduled`; warnings include conflicts, verification and reconcile failures, unavailable nodes, and `StatusUpdateFailed`. State-change events are not repeated just because a pod count changes. Events expire; conditions describe the last recorded state. + +## Manager settings + +The installed defaults differ from some standalone binary defaults: + +| Setting | Standalone default | Helm / Kustomize default | +| --- | --- | --- | +| Leader election | Off | On | +| Workload namespace | `POD_NAMESPACE`, falling back to `default` | Release namespace / `sked-system` | +| Metrics bind address | `0` (disabled) | `:8443` | +| Metrics security | HTTPS with authentication and authorization when enabled | Enabled | +| Health probes | `:8081`, `/healthz` and `/readyz` | Same | +| Webhook | HTTPS on port 9443 | Service port 443 forwards to 9443 | +| HTTP/2 | Disabled | Disabled | +| Logging | Slog JSON at info on stderr | Same | + +Both health endpoints use a ping check; they do not certify registry access, image verification, or node scheduler health. + +Reconcile controls are exposed as flags and corresponding `controller` chart values: + +| Flag | Default | +| --- | --- | +| `--reconcile-timeout` | `1m` | +| `--max-concurrent-reconciles` | `2` | +| `--reconcile-rate-limit-qps` | `5` | +| `--reconcile-rate-limit-burst` | `10` | +| `--reconcile-retry-base-delay` | `1s` | +| `--reconcile-retry-max-delay` | `5m` | + +The timeout bounds a reconcile attempt. Failed reconciles use exponential backoff plus a shared retry rate limit. These QPS and burst settings limit failed-reconcile requeues, not all Kubernetes API traffic. + +## Failure handling + +| Failure | Behaviour | +| --- | --- | +| Leader exits | Another manager can acquire the lease. Existing scheduler pods continue running. | +| Registry or signature verification fails at admission | The create or update is rejected. | +| Image verification fails during reconciliation | `Degraded=True` with `ImageVerificationFailed`; the controller retries. Any existing DaemonSet is left unchanged. | +| API read or DaemonSet write fails | A reconcile error is retried; sked attempts to report `ReconcileFailed`. | +| Status write fails | The attempt returns an error. A `StatusUpdateFailed` event is attempted except for conflict/not-found responses. Status can be stale. | +| Scheduler pod fails to start or attach | DaemonSet availability may expose the failure. Kernel attachment itself is not checked. | +| Policy cannot be reloaded | The previous policy remains, or built-in defaults apply at startup. See [policy failure behaviour](./trust-policy#policy-updates-and-failure-behaviour). | + +## Code map + +Paths are relative to [schedkit/sked](https://github.com/schedkit/sked): + +| Path | Responsibility | +| --- | --- | +| `cmd/main.go` | Manager setup, flags, servers, policy store and verifier wiring. | +| `api/v1/scheduler_types.go` | Spec, status, conditions, and CRD generation markers. | +| `internal/controller/scheduler_controller.go` | Reconciliation, workload creation, watches, and status writes. | +| `internal/controller/nodes.go` | Node selection and winner selection. | +| `internal/controller/status.go`, `events.go` | Conditions, counters, and event transitions. | +| `internal/webhook/v1/scheduler_webhook.go` | Admission checks. | +| `internal/trust/` | Policy parsing, reloads, and Cosign verification. | +| `internal/logging/` | Slog backend and logging flags. | +| `charts/sked/`, `config/` | Helm chart and Kustomize manifests. | diff --git a/docs/sked/crds.md b/docs/sked/crds.md index 6583b84..db81423 100644 --- a/docs/sked/crds.md +++ b/docs/sked/crds.md @@ -5,71 +5,145 @@ title: CRD reference # CRD reference -This page documents the Custom Resources sked introduces. There is exactly one today. +sked defines one custom resource: `Scheduler`. The trust policy is a ConfigMap, not another CRD. -## `SchedExt` +## `Scheduler` -`SchedExt` is the operator's only CRD. It tells sked: "run *this* scheduler image as a privileged DaemonSet in this namespace." - -| | | +| Property | Value | | --- | --- | -| Group | `sked.schedkit.io` | -| Version | `v1` | -| Kind | `SchedExt` | -| Plural | `schedexts` | -| Singular | `schedext` | -| Scope | Namespaced | -| Subresources | `status` (declared, but currently never written to) | - -The CRD has no short names, no printer columns, and no additional kubectl output customisation. `kubectl get schedexts` lists by name, age, and the standard metadata. +| API version | `sked.schedkit.io/v1` | +| Kind / list kind | `Scheduler` / `SchedulerList` | +| Plural / singular | `schedulers` / `scheduler` | +| CRD name | `schedulers.sked.schedkit.io` | +| Scope | Cluster | +| Subresource | `status` | +| Printer columns | Ready, Desired, Available, Age | -### Example +There are no short names. Scheduler objects have no namespace. Their DaemonSets run in the controller's workload namespace: the Helm release namespace by default, or the namespace set by `controller.workloadNamespace` / `--workload-namespace`. ```yaml apiVersion: sked.schedkit.io/v1 -kind: SchedExt +kind: Scheduler metadata: name: scx-rusty - namespace: default spec: - sched: ghcr.io/schedkit/scheds/scx_rusty:latest + sched: ghcr.io/schedkit/scx_rusty:latest + nodeSelector: + schedkit.io/pool: batch ``` -Applying this causes the controller to create a DaemonSet named `scx-rusty` in the `default` namespace. Every node that the DaemonSet's pod-scheduling rules let it run on will end up with a privileged pod running the `scx_rusty` scheduler. +This creates a DaemonSet named `scx-rusty` in the workload namespace, targeting nodes labelled `schedkit.io/pool=batch`, subject to taints and [conflict handling](#overlapping-selectors). Use a verified digest rather than a mutable tag when you need repeatable deployments. -### Spec fields +## Spec fields -| Field | Type | Required | Description | +| Field | Type | Required | Behaviour | | --- | --- | --- | --- | -| `spec.sched` | string | yes | Fully-qualified OCI image reference for the scheduler. Used verbatim as the container image. | +| `spec.sched` | string | Yes | OCI image reference. Tags are accepted; if a digest is supplied, it must be `sha256:` followed by exactly 64 hex characters. | +| `spec.nodeSelector` | map of strings | No | Every label must match. An omitted or empty selector matches all nodes for conflict arbitration; pod placement also depends on tolerations and Kubernetes scheduling. | +| `spec.tolerations` | Kubernetes `Toleration[]` | No | Passed to the pod. Allows placement on matching tainted nodes; does not select nodes. Standard DaemonSet tolerations also apply. | +| `spec.args` | string array | No | Arguments for the `scx` container. The image's entrypoint is unchanged. | +| `spec.env` | Kubernetes `EnvVar[]` | No | Container environment, including `valueFrom` references. | +| `spec.imagePullSecrets` | `LocalObjectReference[]` | No | Named pull secrets in the **workload namespace**. Used by the kubelet, not by the controller's signature verifier. | +| `spec.resources` | Kubernetes `ResourceRequirements` | No | Compute resources for the scheduler container, including requests and limits. These do not set CPU policy for other workloads. | + +No other scheduler configuration fields are defined. In particular, there is no `priority`, `imagePullPolicy`, `command`, `affinity`, `volumes`, `serviceAccountName`, probe configuration, or update strategy. Fields inherited from Kubernetes types may depend on cluster feature gates; sked does not provide a general-purpose pod template. + +### Container configuration example + +```yaml +apiVersion: sked.schedkit.io/v1 +kind: Scheduler +metadata: + name: scx-rusty-batch +spec: + sched: ghcr.io/schedkit/scx_rusty:latest + nodeSelector: + schedkit.io/pool: batch + tolerations: + - key: dedicated + operator: Equal + value: batch + effect: NoSchedule + args: + - --verbose + env: + - name: RUST_LOG + value: info + resources: + requests: + cpu: 100m + memory: 128Mi +``` + +The resource request is an example, not a sizing recommendation. Check the scheduler's own flags and memory requirements. Treat this as an alternative to the first example, not a second Scheduler for the same pool. + +### Validation + +The CRD schema rejects empty or malformed image references and invalid digest formats. The validating webhook also checks: + +- node-selector label keys and values; +- environment-variable names and duplicate names; +- nonempty, valid image-pull-secret names; +- the caller's identity, the image repository allowlist, and its signature according to the [trust policy](./trust-policy). + +The webhook handles creates and updates with `failurePolicy: Fail`. If it cannot be reached or verification fails, the API request is rejected. It does not validate Scheduler deletion. Kubernetes still validates the generated DaemonSet and pods; admission of a Scheduler does not guarantee its pod configuration will run. + +## Status fields + +| Field | Meaning | +| --- | --- | +| `status.observedGeneration` | Scheduler generation most recently processed by the reconciler, including failed attempts. | +| `status.resolvedImage` | Image reference last successfully written to the DaemonSet. Digest-pinned when signature verification is enabled; otherwise the requested reference. Cleared when all selected nodes are lost to conflicts. | +| `status.nodes.desired` | DaemonSet `desiredNumberScheduled`. | +| `status.nodes.ready` | DaemonSet `numberReady`. | +| `status.nodes.available` | DaemonSet `numberAvailable`. | +| `status.nodes.unavailable` | DaemonSet `numberUnavailable`. | +| `status.conditions` | Conditions keyed by `type`, with `status`, `reason`, `message`, `observedGeneration`, and `lastTransitionTime`. | + +There is no phase or per-node status list. Counts come from the DaemonSet, not a count of kernels with an attached scheduler. After a reconcile failure, image and node counts may still describe the last successful reconciliation; read the conditions too. + +### Conditions + +| Type | Meaning when `True` | +| --- | --- | +| `Active` | The Scheduler has not lost all selected nodes to another Scheduler. It can also be true when no nodes match, or when image verification fails; it is not a health signal. | +| `Ready` | The DaemonSet has observed its current generation, has at least one desired node, and all desired pods are updated and available. | +| `Progressing` | The DaemonSet rollout is incomplete. | +| `Degraded` | A conflict, reconcile failure, or unavailable pods need attention. | + +Common reasons are `ActiveScheduler`, `SchedulerConflict`, `NoNodesScheduled`, `RolloutInProgress`, `DaemonSetReady`, `NodesUnavailable`, `SpecInvalid`, `ImageVerificationFailed`, `ReconcileFailed`, and `ReconcileSucceeded`. -That's the entire spec. There is no `imagePullPolicy`, no `nodeSelector`, no `tolerations`, no `priority`. If you need to constrain which nodes the scheduler runs on, do it at the cluster level (taint the nodes you don't want, or expect a future version of the CRD to expose these fields). +With partial conflicts, `Active` remains true and `Degraded=True` reports `SchedulerConflict`. `Ready` can also be true if the rollout on the remaining nodes is complete. With no desired nodes, `Ready=False` reports `NoNodesScheduled`. -There is also no validation on `spec.sched` — the field accepts any string. If the value isn't a valid image reference, the kubelet will fail to pull at pod creation time, and you'll see the error on the DaemonSet's pods rather than at admission. +:::warning +`Ready=True` does not prove that the scheduler attached to the kernel. sked adds no kernel-aware readiness probe. Verify `/sys/kernel/sched_ext` on the nodes as described in [Usage](./usage#verify-the-kernel-state). +::: -### Status fields +## Generated workload -`status` is declared on the type, the CRD has the `status` subresource enabled, and the operator never writes to it. Every `SchedExt` you create will report an empty `status: {}` block. Conditions, phase, and matched/running node counts are not reported today. +Each Scheduler normally owns one DaemonSet with: -This is one of the rougher edges of the current operator and is on the to-do list. +- the Scheduler's name, in the configured workload namespace; +- a controller owner reference to the cluster-scoped Scheduler; +- `managed-by: sked-controller` on the DaemonSet; +- pod labels `name: ` and `managed-by: sked-controller`; +- a selector matching the `name` label; +- one privileged container named `scx`; +- the spec fields above copied into the pod and container, plus node affinity excluding conflict nodes when needed. -### What gets created +Kubernetes supplies the default DaemonSet rollout behaviour. Edit the Scheduler, not its generated DaemonSet: reconciliation replaces the DaemonSet spec. Deleting the Scheduler lets Kubernetes garbage-collect its DaemonSet and pods; sked has no custom cleanup finalizer. -When you apply a `SchedExt`, the controller creates (or patches, on update) one resource: +## Overlapping selectors -A `DaemonSet` with: +For each node, the oldest matching Scheduler wins. Equal creation timestamps are ordered lexicographically by name. Schedulers being deleted are ignored. -- the same name and namespace as the `SchedExt`; -- an owner reference to the `SchedExt` (so deleting the `SchedExt` cascades to deleting the DaemonSet and its pods); -- a label `managed-by: sked-controller` on the DaemonSet itself; -- a pod selector `matchLabels: {name: }`; -- pod template labels `name: ` and `managed-by: sked-controller`; -- one container, named `scx`, with `image: ` and `securityContext.privileged: true`. +- **Partial overlap:** the losing Scheduler excludes conflict nodes through required node affinity on `kubernetes.io/hostname`, and runs on the remaining nodes. +- **Complete overlap:** the losing Scheduler's DaemonSet is deleted. Its node counts become zero, `resolvedImage` is cleared, and `Active=False`, `Ready=False`, and `Degraded=True` report `SchedulerConflict`. -The DaemonSet has no env vars, no volumes, no init containers, no resource requests, and no node selector or tolerations on the pod template. Pod scheduling is whatever the cluster's defaults are. +Arbitration compares node labels only. It does not consider tolerations, kernel support, image validity, or pod health. An older Scheduler that cannot run can still block a newer one. Use non-overlapping selectors rather than relying on age as a priority mechanism. Label changes trigger reassignment, but pod termination and startup are asynchronous; this is not an atomic handoff. -## Multiple `SchedExt` resources +## Migrating from `SchedExt` -The CRD doesn't currently coordinate between resources. If you create two `SchedExt` resources and the resulting DaemonSet pods both end up on the same node, both will try to attach a `sched_ext` scheduler — and the kernel only attaches one at a time. The losing pod will fail to attach and presumably crashloop. +The old namespaced `SchedExt` kind is not served by the current CRD. There is no conversion webhook or automatic workload adoption. -Until the operator gains real node-selection logic, the right way to use it is "one `SchedExt` per intended node set, with cluster-level rules (taints, labels) keeping their pods apart." If you only have one cluster-wide scheduler in mind, the simplest thing is also the working thing: one `SchedExt`, every node gets the same scheduler. +Export the old resources before removing the old installation. Recreate each as a cluster-scoped `Scheduler`: change the kind, remove `metadata.namespace`, and add an explicit node selector. Check image references against the current trust policy. Stop the legacy DaemonSets before creating replacement workloads so both do not attempt to attach on the same nodes. Plan for a gap in `sched_ext` scheduling during the cutover. diff --git a/docs/sked/development.md b/docs/sked/development.md index 8ec5acd..c84b2fa 100644 --- a/docs/sked/development.md +++ b/docs/sked/development.md @@ -5,48 +5,13 @@ title: Development # Development -This page is for contributors to sked itself. If you're trying to use sked, [Usage](./usage) and [Installation](./installation) are the right places. +Run these commands from the [sked repository](https://github.com/schedkit/sked), not this documentation repository. -## Layout +The module is `github.com/schedkit/sked`. The current checkout uses Go 1.27 and controller-runtime v0.25.2, with Kubebuilder v4 layout. Check `go.mod` and the Makefile for the versions required by the revision you are building. -sked is a kubebuilder v4 project. The layout is the standard one: +## Build -``` -sked/ -├── api/v1/ # CRD types (SchedExt) -├── cmd/main.go # the manager entrypoint (one binary) -├── config/ # kustomize manifests for deployment -│ ├── crd/ # CRD YAML (controller-gen output) -│ ├── default/ # the kustomize entrypoint installed by `make deploy` -│ ├── manager/ # Deployment + Service + Service for metrics -│ ├── rbac/ # ClusterRole, RoleBinding, ServiceAccount, leader-election RBAC -│ ├── prometheus/ # ServiceMonitor (commented out by default) -│ ├── webhook/ # webhook scaffolding (commented out by default) -│ ├── network-policy/ # optional NetworkPolicies (commented out by default) -│ └── samples/ # sample SchedExt YAML -├── internal/controller/ # the reconciler -├── test/e2e/ # ginkgo e2e suite (kind-based) -├── hack/ # boilerplate header for generated files -├── Dockerfile # single-stage builder + distroless runtime -├── Makefile # standard kubebuilder make targets -├── go.mod -└── PROJECT # kubebuilder project metadata -``` - -There is one binary, `manager`, built from `cmd/main.go`. The "agent" implied by older docs and earlier designs does not exist as a separate binary: the per-node component is the scheduler image itself, run as a privileged DaemonSet pod. - -## Module info - -- Module: `github.com/schedkit/sked` -- Go: 1.22 (`go 1.22.0` in go.mod) -- controller-runtime: v0.19.0 -- kubebuilder layout: `go.kubebuilder.io/v4` -- API group: `sked.schedkit.io`, version `v1` -- Single Kind: `SchedExt` - -## Building locally - -You'll need Go 1.22+, `make`, and either Docker or Podman for building images. For testing, [kind](https://kind.sigs.k8s.io/) is the path of least friction. +Install Go and a C compiler for race-enabled tests. Docker is used for image and Kind tests; Helm is needed for chart tests. The Makefile downloads Go-based generation and test tools into `bin/` as needed. ```bash git clone https://github.com/schedkit/sked.git @@ -54,96 +19,100 @@ cd sked make build ``` -`make build` runs `manifests`, `generate`, `fmt`, `vet`, then `go build -o bin/manager cmd/main.go`. The first call seeds tools (`kustomize`, `controller-gen`, `setup-envtest`, `golangci-lint`) into `./bin/`. - -To build a container image: +`make build` regenerates manifests and deepcopy code, formats and vets Go code, then writes `bin/manager`. ```bash -make docker-build IMG= +make docker-build IMG=ghcr.io/your-org/sked:dev +make docker-push IMG=ghcr.io/your-org/sked:dev ``` -Defaults to `IMG=controller:latest` if you don't override. The Makefile also exposes a `docker-buildx` target for multi-arch images (`linux/arm64,linux/amd64,linux/s390x,linux/ppc64le`). +`CONTAINER_TOOL=podman` selects Podman for image targets, though the cluster tests use Docker and Kind. Published controller images target Linux AMD64 and ARM64. -## Running against a cluster +## Run in a development cluster -The fastest dev loop runs the controller from your laptop against a real cluster: +Install cert-manager, make the image available to the cluster, then install the local chart: ```bash -make install # install CRDs into the cluster -make run # run the controller locally with `go run` +helm install sked ./charts/sked \ + --namespace sked-system --create-namespace \ + --set image.repository=ghcr.io/your-org/sked \ + --set image.tag=dev \ + --wait --timeout 5m ``` -`make run` runs the same `cmd/main.go` you'd run in the manager pod, but against your local kubeconfig. Apply a `SchedExt` from another terminal and watch the reconcile log on stdout. - -To deploy the controller into the cluster proper: +Alternatively, use `make deploy IMG=ghcr.io/your-org/sked:dev` for Kustomize. Do not run both installations in one cluster. -```bash -make deploy IMG= -``` - -`make deploy` patches `config/manager` with the image you pass, then applies `config/default`. `make undeploy` reverses it. +For host-side debugging, `make install` installs only the CRD, and `make run` starts the manager with your kubeconfig. This is not a complete installation: the always-on webhook server needs `tls.crt` and `tls.key` under `/tmp/k8s-webhook-server/serving-certs`, and admission requests need a webhook endpoint reachable by the API server with the corresponding CA configuration. Set the workload and trust-policy namespaces explicitly when running outside a pod. Avoid running an unelected local reconciler alongside an installed controller. -Note: kind nodes don't have `sched_ext` and the kernel can't host a `sched_ext` scheduler regardless. For testing the *operator's* logic — that the right DaemonSet gets created — kind is fine. For end-to-end testing with an actual scheduler attached, you need a cluster running on machines (or VMs) with `sched_ext` kernels. +Kind is useful for API, reconciliation, and chart tests. Its nodes share the host kernel; Kind does not supply `sched_ext` support. Tests that establish actual attachment need a compatible host kernel, scheduler image, and runtime permissions. A passing chart test alone does not establish kernel compatibility. -## Code generation +## Generate manifests -CRD types live in `api/v1/`. After editing a type: +After changing API types or generation markers: ```bash -make generate # regenerate zz_generated.deepcopy.go -make manifests # regenerate CRD YAML and RBAC +make generate +make manifests +make helm-sync ``` -Both invocations are `controller-gen` wrappers (`v0.16.4` is pinned by the Makefile). The build target depends on these, so `make build` and `make test` regenerate before running. Commit the regenerated files alongside your type changes; CI will fail if they drift. +Generated outputs include: + +- `api/v1/zz_generated.deepcopy.go` +- `config/crd/bases/sked.schedkit.io_schedulers.yaml` +- `config/rbac/role.yaml` +- `config/webhook/manifests.yaml` +- `charts/sked/crds/` and `charts/sked/files/` + +Edit the API, markers, or source files under `config/`, not the generated chart copies. `make verify-generated` fails if regenerated files under `api/` or `config/` have uncommitted changes; `make verify-helm` checks chart copies. Include generated changes with the source changes. -## Tests +## Test ```bash -make test # unit tests + envtest-backed integration -make test-e2e # full kind cluster, ginkgo +make test +make coverage +go build ./... +make lint +make test-helm ``` -`make test` runs `go test` with `KUBEBUILDER_ASSETS` pointed at a downloaded `setup-envtest` binary, which spins up a local etcd + apiserver pair to test the reconciler against. It excludes the `test/e2e` package. +- `make test` runs tests excluding `/e2e`, with the race detector and an envtest API server. It writes `cover.out`. +- `make coverage` checks total statement coverage against a 40% minimum. Set `COVERAGE_THRESHOLD` to try another threshold. +- `make lint` runs the pinned golangci-lint version; `make lint-fix` applies supported automatic fixes. +- `make test-helm` checks generated chart files, lints the chart, and tests rendered resources against Kustomize. CI tests Helm 3 and 4. -`make test-e2e` requires a running kind cluster (`kind create cluster`). It builds the manager image, loads it into kind, deploys, and runs the ginkgo suite. By default it also installs Prometheus Operator and CertManager — both can be skipped with environment variables: +Cluster tests have separate entrypoints: ```bash -PROMETHEUS_INSTALL_SKIP=true CERT_MANAGER_INSTALL_SKIP=true make test-e2e +make test-e2e +make test-helm-e2e ``` -## Code style +`make test-e2e` expects an existing Kind cluster named `kind`; it builds and loads the manager image and deploys test dependencies. Prometheus and cert-manager installation can be skipped with `PROMETHEUS_INSTALL_SKIP=true` and `CERT_MANAGER_INSTALL_SKIP=true` when those dependencies are already available. -Standard Go style plus what `golangci-lint` enforces. The Makefile pins lint version `v1.59.1`. +`make test-helm-e2e` creates a temporary Kind cluster with two workers, exercises installation, leader failover, eviction protection, upgrade, rollback, and uninstall, then removes the cluster. -- `make fmt` and `make vet` run before `make build`. Don't push code that fails them. -- `make lint` for the full linter set. -- `make lint-fix` applies the auto-fixable subset. +## Logging -Project-shape opinions: +The default is newline-delimited Slog JSON at info level on stderr. For local debugging, pass: -- Keep the reconciler small. Right now it's about 60 lines; if it grows past ~150, that's a sign to factor. -- Avoid log-and-return-nil. If reconciliation can't proceed, return the error so the manager can requeue. -- New fields on `SchedExtSpec` need a real reason — once a field is published it has to be supported. New CRDs even more so. - -## Manifests +```text +--log-format=text --log-level=debug +``` -The `make manifests` target regenerates: +`--log-level=debug` enables Logr `V(1)` through `V(4)`. Use `--log-verbosity=2` instead for an exact `V(2)` threshold. Do not explicitly combine `--log-level` and `--log-verbosity`; startup rejects that configuration. All legacy `--zap-*` flags are rejected too. -- `config/crd/bases/sked.schedkit.io_schedexts.yaml` — the CRD itself. -- `config/rbac/role.yaml` — the ClusterRole derived from `+kubebuilder:rbac` markers in `internal/controller/`. +JSON fields include `time`, `level`, `msg`, optional `logger`, and `err`. Logger names are slash-separated and severity names use Slog's native form. No-op reconciles log at `V(1)`; workload changes remain visible at info. -Note that the rbac markers in the reconciler currently only declare verbs on `schedexts*` resources. The reconciler creates DaemonSets, but no RBAC marker covers that — so the generated `role.yaml` does not authorise DaemonSet writes. Adding the missing marker (and regenerating) is on the to-do list. Until it's done, real-cluster installs need a kustomize patch granting `apps/daemonsets` to the controller's ClusterRole. +Use `log.FromContext(ctx)` to preserve reconcile context. Log object references or selected fields rather than whole Kubernetes objects. Do not log Secret contents or complete trust policies. See the [upstream logging reference](https://github.com/schedkit/sked#controller-logging) for migration details. -## Building a single-file installer +## Package a release ```bash -make build-installer IMG= +make build-installer IMG=ghcr.io/your-org/sked:dev +bash hack/package-helm.sh v1.2.3 /tmp/sked-chart ``` -Writes `dist/install.yaml`, the kustomize-rendered output of `config/default` with the controller image patched. This is the file to ship for users who want a single `kubectl apply -f` install. - -## Submitting changes - -Fork, branch, PR. One logical change per PR. Tests for new behaviour. Reasonable commit messages. +The first command renders `dist/install.yaml`. The second illustrates chart packaging for a release tag: `v1.2.3` produces chart version `1.2.3` and `appVersion: v1.2.3` without changing the checked-in `Chart.yaml`. -If your change crosses the schedctl/sked boundary — for example, growing schedctl with a feature that the operator wants to consume — expect that to be coordinated work across the two repos. Today the two are independent (sked does not depend on schedctl as a Go module), so the integration is mostly conceptual; that may change as the two converge. +The release workflow publishes and signs the controller image, attaches the chart to the GitHub Release, and publishes it to `https://charts.schedkit.io`. Chart and controller versions are released together. Follow the [release instructions](https://github.com/schedkit/sked#releases-and-helm-repository) for tag rules and repository setup. diff --git a/docs/sked/installation.md b/docs/sked/installation.md index da0e4c0..cba77e5 100644 --- a/docs/sked/installation.md +++ b/docs/sked/installation.md @@ -5,99 +5,180 @@ title: Installation # Installation -sked is currently shipped as Kustomize manifests under `config/` in the [sked repository](https://github.com/schedkit/sked). There is no Helm chart yet. Installing the operator means getting the controller-manager Deployment and the `SchedExt` CRD into the cluster; from there, applying `SchedExt` resources causes per-scheduler DaemonSets to come into existence. +Install sked with the Helm chart or the Kustomize manifests. Use **one installation per cluster**: every controller watches all Scheduler objects. Do not install the chart alongside a Kustomize deployment; Helm does not adopt that installation's resources. -## Cluster prerequisites +## Prerequisites -Before installing sked, the cluster needs to satisfy a few things: +- Helm 3 or 4 and `kubectl`, with permission to create CRDs, cluster RBAC, and admission webhooks. +- [cert-manager](https://cert-manager.io/docs/installation/) for the default webhook certificate setup. You can instead [supply a certificate](#webhook-certificates). +- Nodes with `sched_ext` support: Linux 6.12 or later with `CONFIG_SCHED_CLASS_EXT=y`, BPF support, and BTF, plus any requirements of the scheduler image. sked does not check node kernels. +- A workload namespace that permits privileged pods. The manager runs non-root with capabilities dropped; scheduler pods run privileged. Helm does not configure a Pod Security exemption for them. +- Network access from controllers to the image registry and Sigstore trust-root services for verification, and from nodes to the registry for image pulls. See [private and mirrored registries](./trust-policy#private-and-mirrored-registries) for restricted networks. -- Every node where you intend to run a scheduler must have a kernel with `sched_ext` support — 6.12 or later, with `CONFIG_SCHED_CLASS_EXT=y` and the supporting `CONFIG_BPF*` and `CONFIG_DEBUG_INFO_BTF` flags. sked itself does not check this; the scheduler container will simply fail to attach if the kernel can't host it. -- Every node must have a working container runtime (containerd is the typical choice for Kubernetes nodes; this is fine). -- Your cluster needs to allow privileged DaemonSet pods. Each scheduler pod is created with `securityContext.privileged: true`. If you have a Pod Security policy or admission controller that strips privileges, the scheduler will fail to load BPF. +You do not need schedctl on the nodes. -You do *not* need to install schedctl on the nodes. The cluster path runs the OCI scheduler image directly via the kubelet's container runtime; it does not invoke schedctl on the host. - -## Installing the operator - -Clone the repo and apply the default Kustomize overlay: +## Install with Helm ```bash -git clone https://github.com/schedkit/sked.git -cd sked -kubectl apply -k config/default +helm repo add schedkit https://charts.schedkit.io +helm repo update +helm install sked schedkit/sked \ + --namespace sked-system --create-namespace \ + --wait --timeout 5m ``` -This installs: +Add `--version ` to pin a release. Prereleases require an explicit version or `--devel`. Chart archives are also attached to [GitHub Releases](https://github.com/schedkit/sked/releases). From a sked checkout, use `./charts/sked` instead of `schedkit/sked` for a local chart install. -- The `sked-system` namespace (labelled `control-plane: controller-manager`). -- The `schedexts.sked.schedkit.io` CRD. -- A `sked-controller-manager` Deployment with one replica running the operator binary. -- A `ServiceAccount`, `ClusterRole`, and `ClusterRoleBinding` for the controller (`sked-controller-manager` / `sked-manager-role`). -- Leader-election RBAC (Role + RoleBinding) inside `sked-system`. -- A metrics `Service` (`sked-controller-manager-metrics-service`) on port 8443 (HTTPS, with authn/authz filtering enabled). -- Liveness and readiness probes on port 8081. +With the release name and namespace above, the chart installs: -By default the controller image is `controller:latest` — a placeholder. To install with a real image, override `IMG` and use `make deploy` (which patches the image reference via `kustomize edit set image`): +- `schedulers.sked.schedkit.io`, the cluster-scoped Scheduler CRD; +- `sked-controller-manager`, a two-replica Deployment with leader election; +- a PodDisruptionBudget with `minAvailable: 1` and soft spreading across nodes; +- controller and leader-election RBAC, including DaemonSet access; +- a validating webhook, its Service, and cert-manager certificate resources; +- a default image trust-policy ConfigMap; +- an authenticated HTTPS metrics Service on port 8443, plus health probes on port 8081. -```bash -make deploy IMG=ghcr.io/schedkit/sked:v0.x.x +Scheduler DaemonSets run in `sked-system` by default. They are created by the controller, not by Helm. + +### Configure the release + +Keep overrides in a values file and pass it with `-f my-values.yaml` on install and upgrade. For example, to separate scheduler workloads from the controller: + +```yaml +controller: + workloadNamespace: schedulers ``` -`make deploy` does the same `kubectl apply -k config/default` after rewriting the image, and is what most users should run. +Create `schedulers` first and configure its admission policy to permit privileged workloads. Helm does not create a custom workload namespace. Secrets and ConfigMaps referenced by Scheduler pods must exist there too. Choose the namespace before creating Schedulers; changing it is not a workload migration mechanism. -## Generating a single install YAML +Common values: -If you'd rather have a single file to apply (or to ship to an offline cluster), the repo's `make build-installer` target produces one: +| Value | Default / use | +| --- | --- | +| `image.repository` | `ghcr.io/schedkit/sked` | +| `image.tag` | Chart `appVersion` when unset. | +| `image.digest` | Optional digest; takes precedence over the tag. | +| `imagePullSecrets` | Pull secrets for **controller pods**, in the release namespace. | +| `replicaCount` | `2` | +| `controller.leaderElection` | `true`; required by the chart with multiple replicas. | +| `podDisruptionBudget.enabled` / `minAvailable` | `true` / `1` | +| `topologySpreadConstraints` | `null` uses the default soft spread; `[]` disables it. | +| `nodeSelector`, `tolerations`, `affinity`, `resources` | Configure the **controller**, not scheduler pods. | +| `controller.extraArgs` | Additional manager flags, including logging. | +| `trustPolicy.policy` | Override the [trust policy](./trust-policy). | +| `trustPolicy.existingConfigMap` | Use a separately managed policy ConfigMap. | +| `metrics.serviceMonitor.enabled` | `false`; enable when Prometheus Operator is installed. | +| `networkPolicy.enabled` | `false`; when enabled, restricts metrics ingress to namespaces labelled `metrics: enabled` by default. | -```bash -make build-installer IMG=ghcr.io/schedkit/sked:v0.x.x +The [chart values](https://github.com/schedkit/sked/blob/main/charts/sked/values.yaml) list all settings. + +Two schedulable nodes are needed for protection against a node failure. The default spreading rule allows both replicas on one node. The PDB protects against voluntary evictions, not node failures or direct deletion. More replicas add webhook capacity and failover options, not reconciliation throughput. For a single-replica development install, set `replicaCount: 1` and consider disabling the PDB to allow drains. + +The published `latest` and `main` image tags follow the main branch, not the latest release. Release charts select the matching release image. Prefer a release or digest over a moving tag. + +### Webhook certificates + +The webhook is always enabled. By default the chart creates a self-signed cert-manager Issuer. To use your own issuer: + +```yaml +webhook: + certManager: + issuerRef: + name: platform-ca + kind: ClusterIssuer ``` -This writes `dist/install.yaml`, which is the kustomize-rendered output of `config/default`. Apply it with `kubectl apply -f dist/install.yaml`. +Without cert-manager, set all of: + +- `webhook.certManager.enabled: false` +- `webhook.secretName`: a Secret in the release namespace containing `tls.crt` and `tls.key` +- `webhook.caBundle`: the base64-encoded PEM CA certificate -## Verifying the install +For the default release, the certificate must cover `sked-webhook-service.sked-system.svc` and `sked-webhook-service.sked-system.svc.cluster.local`. Adjust both names if you change the release's fullname or namespace. You are responsible for rotating a supplied certificate. -A few seconds after applying: +### Metrics + +To enable a ServiceMonitor and authorize a Prometheus service account: + +```yaml +metrics: + serviceMonitor: + enabled: true + readerSubjects: + - kind: ServiceAccount + name: prometheus + namespace: monitoring +``` + +Set `metrics.serviceMonitor.labels` if Prometheus selects monitors by label. The default monitor skips TLS certificate verification because the metrics server uses a self-signed certificate; the webhook certificate does not configure metrics TLS. Setting `metrics.secure: false` removes both TLS and metrics authentication. + +## Verify the install ```bash +kubectl rollout status deployment/sked-controller-manager -n sked-system --timeout=5m kubectl get pods -n sked-system -kubectl get crd schedexts.sked.schedkit.io +kubectl get crd schedulers.sked.schedkit.io +kubectl get certificate -n sked-system ``` -You should see a single `sked-controller-manager-*` pod running, and the CRD listed. If the pod is in `CrashLoopBackOff`, the most common causes are: +The certificate command applies to cert-manager installs. If pods cannot start, inspect pod events for image-pull or missing certificate Secret errors. If Scheduler admission fails, check the webhook Service, certificate readiness, and controller logs. See [Usage](./usage) to create the first Scheduler and verify admission as well as reconciliation. + +## Install with Kustomize + +The default overlay also enables two replicas, leader election, a PDB, trust policy, and cert-manager-backed webhooks. Install cert-manager first, then choose a published controller tag: + +```bash +git clone https://github.com/schedkit/sked.git +cd sked +make deploy IMG=ghcr.io/schedkit/sked: +``` -- The image reference is the placeholder `controller:latest` and your cluster can't pull it. Override `IMG` and re-deploy. -- The cluster's Pod Security admission is rejecting the manager pod's security context. The default manifest runs the manager as non-root with all capabilities dropped, but very strict policies still need explicit exceptions. -- RBAC for the leader-election lease can't be acquired — usually a sign that the operator was previously installed in a different namespace and is fighting with a stale lease. +`make deploy` updates the image in `config/manager` and applies `config/default`. Do not apply the unmodified overlay: its `controller:latest` image is a placeholder. -Once the controller is healthy you can apply a `SchedExt`. See [Usage](./usage) for the walkthrough. +For a single rendered file: -## A note on RBAC +```bash +make build-installer IMG=ghcr.io/schedkit/sked: +kubectl apply -f dist/install.yaml +``` -The generated `ClusterRole` (`sked-manager-role`) currently grants verbs on `schedexts`, `schedexts/status`, and `schedexts/finalizers`. It does **not** explicitly grant verbs on `apps/daemonsets`, even though the controller creates and patches DaemonSets. In practice this means a strict cluster will refuse those writes — if you see `forbidden: User "system:serviceaccount:sked-system:sked-controller-manager" cannot create resource "daemonsets"`, that's the cause. The fix is to extend the ClusterRole with the missing rule. There's a tracked issue for adding this to the generated manifests; until it lands, a small kustomize patch under your install does the job. +Use a source revision and image from the same release. -## Air-gapped installs +## Upgrade and rollback -The cluster path needs: +**Helm does not upgrade CRDs.** Apply the CRD from the chart version you intend to install, then upgrade with the same version and your saved values. Replace the example version below with your chosen release: -- The controller image (whatever you pass as `IMG`). -- Every scheduler image you intend to reference from `SchedExt.spec.sched`. +```bash +version=1.2.3 +helm repo update +helm show crds schedkit/sked --version "$version" | kubectl apply -f - +helm upgrade sked schedkit/sked --version "$version" \ + --namespace sked-system -f my-values.yaml \ + --wait --timeout 5m +``` -Mirror both into an internal registry, override `IMG` for the controller deploy, and use the mirrored references in your `SchedExt` resources. The controller does not pull scheduler images itself — the kubelet does, on each node — so your nodes need credentialed access to wherever the scheduler images live. +For a local checkout, apply `charts/sked/crds/` before upgrading from `./charts/sked`. Do not change the release name, namespace, `nameOverride`, or `fullnameOverride` during an upgrade; they determine resource names and Deployment selectors. -## Uninstalling +Remove all `--zap-*` flags from custom Deployments and `controller.extraArgs`, including saved values, before upgrading from a Zap-based controller. They now prevent startup. Use `--log-format=text` and `--log-verbosity=1` to retain the old development output and verbosity, or omit logging flags for JSON/info defaults. ```bash -make undeploy +helm history sked --namespace sked-system +helm rollback sked --namespace sked-system --wait --timeout 5m ``` -`make undeploy` runs `kustomize build config/default | kubectl delete -f -`, which removes the controller, its RBAC, and the CRD. **Deleting the CRD also deletes every `SchedExt` of that kind**, and via owner references that cascades to deleting the corresponding DaemonSets and pods. Schedulers on the nodes will detach as the pods exit; the kernel reverts to its default policy. +Rollback does not restore CRDs or Scheduler objects. Check schema compatibility and restore flags supported by the old binary. Moving from the old `SchedExt` API requires a [separate migration](./crds#migrating-from-schedext), not just a Helm upgrade. -If you only want to remove the operator and keep the existing `SchedExt` resources (they'll have no controller and won't be reconciled, but they won't be deleted), use: +## Uninstall + +Helm uninstall removes the operator but **leaves the CRD, Scheduler objects, and scheduler DaemonSets running**. To stop schedulers, delete their Scheduler objects and wait for workload deletion first. For example: ```bash -kubectl delete -k config/manager -kubectl delete -k config/rbac +kubectl delete scheduler scx-rusty +kubectl wait --for=delete daemonset/scx-rusty -n sked-system --timeout=2m +helm uninstall sked --namespace sked-system ``` -This leaves the CRD and any `SchedExt` resources intact. Reinstalling the controller later will pick them back up. +Repeat the deletion for each Scheduler you intend to stop, using the configured workload namespace. Check that the pods have terminated before treating the nodes as returned to their default CPU scheduler. + +For a Kustomize installation, `make undeploy` deletes the default overlay, **including the CRD and namespace**. Unlike Helm uninstall, it removes all Scheduler objects and their owned workloads. Deleting `schedulers.sked.schedkit.io` manually has the same cluster-wide effect on Scheduler objects. Export anything you need to retain before deleting the CRD. diff --git a/docs/sked/overview.md b/docs/sked/overview.md index 0a4cbf5..acef6df 100644 --- a/docs/sked/overview.md +++ b/docs/sked/overview.md @@ -7,62 +7,34 @@ slug: /sked/overview # sked -:::info -sked is at an early, scaffolded stage. Today it's a thin Kubernetes operator that runs `sched_ext` scheduler images as privileged DaemonSets, driven by a single `SchedExt` CRD. It does not yet do node selection, image signature verification, status reporting, or coordinated rollouts. These pages document what exists in the operator code as of `v1` of the API; they will grow as the operator does. -::: +sked runs OCI-packaged `sched_ext` CPU schedulers on Kubernetes nodes. You declare an image and a set of node labels in a cluster-scoped `Scheduler` resource. The operator creates a privileged DaemonSet for it, verifies the image, and reports rollout status. -sked is the Kubernetes operator side of schedkit. It watches `SchedExt` resources in the API server and, for each one, creates a `DaemonSet` in the same namespace that runs the referenced OCI scheduler image as a privileged container on every node the DaemonSet's pod scheduling rules let it run on. +sked does not replace the Kubernetes scheduler. Kubernetes places pods on nodes; a `sched_ext` scheduler controls how a node's Linux kernel allocates CPU time to threads. -That's the whole feature set today. There is no separate per-node agent, no `schedctl` invocation on the node, no rolling status, no kernel preflight from the operator. The kernel preflight, signature verification, and lifecycle management story you get from [schedctl](../schedctl/overview) on a single host is *not* what sked currently provides on a cluster — those are independent code paths. +## What it manages -## What sked does +- **Node placement:** `spec.nodeSelector` selects a node pool; `spec.tolerations` permits placement on tainted nodes. +- **Container configuration:** arguments, environment variables, image pull secrets, and resource requests and limits. +- **Image trust:** an admission webhook checks the caller, image allowlist, and Cosign signature against a ConfigMap policy. Signature verification is enabled by default. The reconciler verifies again and uses the verified digest in the DaemonSet. +- **Conflicts:** when selectors overlap, the oldest Scheduler wins each node, with name order breaking timestamp ties. Other Schedulers exclude those nodes and report a conflict. +- **Status:** node counts, the resolved image, conditions, and Kubernetes events derived from the DaemonSet's rollout. -In one sentence: applying a `SchedExt` causes a `DaemonSet` to come into existence whose pods run the referenced scheduler image with `securityContext.privileged: true`. +The [Helm chart](./installation) installs two controller replicas, leader election, admission webhooks, and a PodDisruptionBudget by default. Only the elected leader reconciles; both replicas serve webhooks. -Slightly less compressed: +## Boundaries -- The operator watches `SchedExt` resources in the `sked.schedkit.io/v1` API group. -- For each one, it creates or patches a `DaemonSet` with the same name and namespace as the `SchedExt`. -- That DaemonSet has one container, named `scx`, with the image taken verbatim from `spec.sched`. -- The container runs privileged. Privileged containers get all Linux capabilities, including `CAP_BPF`, `CAP_SYS_ADMIN`, and `CAP_PERFMON`, which is what loading a `sched_ext` scheduler needs. -- The DaemonSet is owned by the `SchedExt`; deleting the `SchedExt` deletes the DaemonSet and its pods via the standard owner-reference cascade. -- Pods inherit the DaemonSet's default scheduling: by default that means every node. There is no built-in node selector, toleration, or priority field on the CRD today. +There is no per-node sked agent, and sked does not invoke [schedctl](../schedctl/overview). The kubelet runs the scheduler image directly. schedctl and sked implement their own signature verification and use different trust-policy formats. -## What sked does *not* do (yet) +sked does not inspect node kernels or confirm that a scheduler has attached. `Ready=True` means the DaemonSet rollout is complete, not that `/sys/kernel/sched_ext/state` is `enabled`. Check the kernel when validating a deployment. -It's important to be precise about this. The CRD has a single field. The status field exists but is never written to. There is no: +Rollouts use Kubernetes' default DaemonSet update strategy. The CRD does not expose rollout tuning, scheduler priority, custom probes, or automatic rollback. Conflict handling covers Scheduler objects, not schedulers started outside sked. -- node selector, toleration, or priority field on the CRD; -- mechanism to run different schedulers on different node pools from a single `SchedExt`; -- conditions or status reporting back to the resource; -- coordination with `schedctl` on the node; -- image signature verification (cosign) on the cluster path; -- rolling rollout strategy beyond Kubernetes' default DaemonSet rolling update; -- finalizer or cleanup logic beyond owner references; -- validation on `spec.sched` (any string is accepted; the kubelet will fail to pull a malformed reference at pod creation time). +Scheduler containers run privileged and change node-wide kernel behaviour. Restrict who can manage Scheduler resources and the trust policy, and use node pools whose kernels support the images you deploy. -Some of this is on the roadmap, some isn't. If a feature you need isn't listed, file an issue. +## Start here -## When sked is the right tool +1. [Install sked](./installation) with Helm or Kustomize. +2. [Run a scheduler](./usage) on a labelled node pool. +3. Consult the [CRD reference](./crds) and [trust policy](./trust-policy) for configuration. -For now, sked is the right tool when: - -- You're already running Kubernetes and you want to use the cluster as the deployment substrate for `sched_ext` schedulers. -- You're comfortable defining your own scheduling rules — taints, node labels, the DaemonSet's pod-spec — at the cluster level rather than expecting the operator to provide them on the CRD. -- You're OK with the privileged DaemonSet model and the security implications that come with it. - -If you want fine-grained, declarative control over which scheduler runs on which node pool, with rollouts and status, you're either ahead of the operator or ahead of the project. The schedctl path on individual hosts may serve you better today, possibly driven by your existing configuration management. - -## How it relates to schedctl - -They don't currently share code or a runtime contract. schedctl is the host CLI; sked is a Kubernetes operator that runs scheduler images directly via the kubelet's container runtime. A future iteration could plausibly use schedctl as the in-pod runtime to pick up its kernel preflight, signature verification, and discrepancy reporting — but that integration does not exist today. The two projects are aligned in design (both run OCI-packaged sched_ext schedulers) but independent in implementation. - -If a scheduler refuses to load on a particular node, the debugging path is the same as if you were running it directly with the container runtime — `kubectl logs` on the DaemonSet pod, `dmesg` on the node, `cat /sys/kernel/sched_ext/state` to confirm what (if anything) the kernel has attached. The [schedctl troubleshooting guide](../schedctl/troubleshooting) covers the kernel-side failure modes that show up the same way under sked. - -## Where to start - -- [Installation](./installation) walks through deploying the operator into a cluster. -- [CRD reference](./crds) documents the `SchedExt` resource as it exists today. -- [Usage](./usage) is the practical "how do I apply one" walkthrough. -- [Architecture](./architecture) covers how the controller actually works internally. -- [Development](./development) is for contributors. +[Architecture](./architecture) explains reconciliation and failure handling. [Development](./development) covers builds, tests, and generated manifests. diff --git a/docs/sked/trust-policy.md b/docs/sked/trust-policy.md new file mode 100644 index 0000000..5fcd7e3 --- /dev/null +++ b/docs/sked/trust-policy.md @@ -0,0 +1,88 @@ +--- +id: trust-policy +title: Trust policy +--- + +# Trust policy + +Scheduler pods run privileged. sked checks which images and callers are allowed before accepting Scheduler creates and updates, then verifies the image again before writing the DaemonSet. + +The policy lives in a ConfigMap, normally `sked-scheduler-trust-policy` in `sked-system`, under the key `policy.yaml`. Helm exposes it through `trustPolicy.policy` or `trustPolicy.existingConfigMap`. It is not a CRD and is not interchangeable with a schedctl trust-policy file. + +## Default policy + +The shipped policy enables signature verification and allows the Schedkit repositories listed in [`config/trustpolicy/trust_policy.yaml`](https://github.com/schedkit/sked/blob/main/config/trustpolicy/trust_policy.yaml), including `ghcr.io/schedkit/scx_rusty` and `ghcr.io/schedkit/scx_lavd`. + +Signatures must use: + +- Issuer: `https://token.actions.githubusercontent.com` +- Identity: `https://github.com/schedkit/plumbing/.github/workflows/oci-build-publish.yml@refs/heads/main` + +There is no extra caller restriction by default; Kubernetes RBAC still applies. Restrict Scheduler writes to administrators trusted to change node-wide kernel behaviour. Admission restrictions are additional checks, not replacements for RBAC. + +## Custom policy + +This Helm values example trusts one repository and signing workflow, and limits Scheduler management to a group or service account: + +```yaml +trustPolicy: + policy: + allowedImages: + - ghcr.io/my-org/my-scheduler + verifySignatures: true + allowedIdentities: + groups: + - scheduler-admins + serviceAccounts: + - platform:scheduler-deployer + cosign: + issuer: https://token.actions.githubusercontent.com + identity: https://github.com/my-org/my-scheduler/.github/workflows/release.yml@refs/heads/main +``` + +Replace the repository, workflow identity, and callers with your own. For a separately managed ConfigMap, put the contents of `policy` under `data.policy.yaml` as a YAML string, then set: + +```yaml +trustPolicy: + existingConfigMap: scheduler-trust-policy + namespace: sked-system + key: policy.yaml +``` + +### Policy fields + +| Field | Behaviour | +| --- | --- | +| `allowedImages` | Repository patterns using Go `path.Match` syntax. Matching excludes the tag and digest; `*` does not cross `/`. Omitted uses the built-in list. **An explicit empty list allows all repositories.** | +| `verifySignatures` | Defaults to `true`. Setting it to `false` skips signature verification and digest pinning; caller and image-allowlist checks still apply at admission. | +| `allowedIdentities.users` | Exact Kubernetes usernames. | +| `allowedIdentities.groups` | Exact Kubernetes group names. | +| `allowedIdentities.serviceAccounts` | `namespace:name` or `system:serviceaccount:namespace:name`. | +| `cosign.issuer` | Expected certificate OIDC issuer. Defaults to GitHub Actions. | +| `cosign.identity` | Exact signing certificate identity. Defaults to the Schedkit publishing workflow if neither identity field is set. | +| `cosign.identityRegexp` | Signing certificate identity regular expression, as an alternative to an exact identity. Anchor patterns to avoid unintended matches. | +| `cosign.trustedRootPath` | Optional path **inside the controller container** to a Sigstore trusted-root file. Otherwise the verifier fetches the trusted root. | + +A caller matching any configured user, group, or service account is allowed. If all three lists are empty, the policy adds no caller restriction. Unknown policy fields and invalid patterns are rejected when parsing the ConfigMap. + +The verifier uses Cosign v3 signature bundles through OCI referrers. It resolves the image to a digest, verifies that content, and returns the digest-pinned reference. The reconciler writes that reference to the DaemonSet and `status.resolvedImage`; it does not rewrite `spec.sched`. A tag can resolve differently on a later reconcile, so use an explicit digest when the deployment must stay on fixed content. + +## Policy updates and failure behaviour + +Each controller replica reloads the ConfigMap every 30 seconds. Allowlist, caller, and verification-enable changes are read from the refreshed policy. The Cosign verifier's issuer, identity, and trusted-root path are configured at process startup: **restart every controller replica after changing `cosign` settings**. Helm changes to a chart-managed policy trigger that restart through a pod-template checksum. Changes to an existing external ConfigMap do not. + +A malformed or unreadable policy leaves the last successfully loaded policy in place. At startup, load failures fall back to built-in defaults. A missing ConfigMap or an empty policy document also selects built-in defaults; it is not a deny-all configuration. Check controller logs after policy changes. + +Policy refresh does not itself enqueue every Scheduler. Caller and allowlist checks run at admission, not as continuous enforcement against existing resources. If verification fails during reconciliation, sked reports `ImageVerificationFailed` and retries without replacing an existing DaemonSet. Tightening policy does not automatically stop workloads already running; delete their Scheduler objects if they must be removed. + +## Private and mirrored registries + +There are three separate credential paths: + +1. Helm `imagePullSecrets` lets the kubelet pull the **controller image**. +2. `Scheduler.spec.imagePullSecrets` lets the kubelet pull the **scheduler image**, using secrets in the workload namespace. +3. The **controller's verifier** accesses registries with `authn.DefaultKeychain`. It does not load either list of Kubernetes pull secrets. Supply registry credentials separately inside the controller, for example through a mounted Docker configuration and `DOCKER_CONFIG`, using the chart's `extraVolumes`, `extraVolumeMounts`, and `extraEnv`. + +For a mirror, copy the scheduler image and its signature/referrer artifacts, allow the mirrored repository in the policy, and keep the expected signing identity checks. Nodes and controllers both need access to the mirror. + +For an offline environment, also provision a trusted-root file and set `cosign.trustedRootPath`. sked caches trusted material for 24 hours. Mirroring only the image layers is insufficient for signature verification; test admission and reconciliation with external network access blocked before relying on an offline installation. diff --git a/docs/sked/usage.md b/docs/sked/usage.md index a365007..6a6e437 100644 --- a/docs/sked/usage.md +++ b/docs/sked/usage.md @@ -5,133 +5,164 @@ title: Usage # Usage -This page is a practical walkthrough: from a freshly-installed sked to a running scheduler on every node in the cluster. +This walkthrough assumes sked is [installed](./installation) as release `sked` in `sked-system`, using the default trust policy and workload namespace. Adjust namespace arguments if your scheduler DaemonSets run elsewhere. Scheduler resources themselves are cluster-scoped. -It assumes you've already installed the operator ([Installation](./installation)) and have `kubectl` access to the cluster. It also assumes you're comfortable with the operator's current scope: a single `SchedExt` resource creates a privileged DaemonSet that runs a `sched_ext` scheduler everywhere the DaemonSet's default pod-scheduling rules let it run. There is no node selector or priority on the CRD today — see [CRD reference](./crds) for the precise shape. +## Check the controller and nodes -## Step 1: confirm the operator is healthy +```bash +kubectl rollout status deployment/sked-controller-manager -n sked-system --timeout=5m +kubectl get crd schedulers.sked.schedkit.io +kubectl get schedulers +``` + +Check existing Schedulers before adding one: an empty selector claims every node for conflict arbitration. -Before applying any `SchedExt`, make sure the controller is up: +On each intended node, confirm kernel support and check whether another scheduler is already attached. From a shell on the node: ```bash -kubectl get pods -n sked-system -kubectl get crd schedexts.sked.schedkit.io +ls /sys/kernel/sched_ext +cat /sys/kernel/sched_ext/state ``` -Look for one `sked-controller-manager-*` pod in `Running` and the CRD listed. If the controller pod is in `CrashLoopBackOff`, fix that before applying anything — without a controller, an applied `SchedExt` won't create its DaemonSet. +The directory's presence is a first check, not a guarantee that a particular scheduler is compatible. Stop schedulers managed outside sked before targeting those nodes. sked cannot arbitrate against schedctl or a manually started scheduler. -## Step 2: confirm the nodes can run a sched_ext scheduler - -The operator does not preflight the kernel for you. The scheduler container will simply fail to attach if the kernel can't host it. A quick spot check on a node before rolling sked out widely: +Label the nodes you have checked, replacing the example node name: ```bash -kubectl debug node/ --image=alpine -- ls /host/sys/kernel/sched_ext +kubectl label node worker-1 schedkit.io/pool=batch ``` -(or SSH into the node and `ls /sys/kernel/sched_ext`). On every node where you'd run a scheduler, you want that directory to exist. If your fleet is heterogeneous, you'll want a node selector at the cluster level keeping the DaemonSet pods away from kernels that don't support sched_ext — see *Constraining where the scheduler runs* below. +Repeat for the other nodes in the pool. A label does not prove kernel support; it is your declaration that those nodes are suitable. -## Step 3: apply your first `SchedExt` +## Create a Scheduler -Create a file called `scx-rusty.yaml`: +Save as `scx-rusty.yaml`: ```yaml apiVersion: sked.schedkit.io/v1 -kind: SchedExt +kind: Scheduler metadata: name: scx-rusty - namespace: default spec: - sched: ghcr.io/schedkit/scheds/scx_rusty:latest + sched: ghcr.io/schedkit/scx_rusty:latest + nodeSelector: + schedkit.io/pool: batch ``` -Apply it: +The default policy allows this repository and requires a trusted signature. The reconciler resolves the tag to a verified digest. Use an explicit `@sha256:...` digest instead of `:latest` for a fixed deployment. ```bash kubectl apply -f scx-rusty.yaml +kubectl get scheduler scx-rusty +kubectl get daemonset scx-rusty -n sked-system +kubectl get pods -n sked-system -l name=scx-rusty -o wide +kubectl wait --for=condition=Ready scheduler/scx-rusty --timeout=5m ``` -Within a few seconds the controller will reconcile and create the DaemonSet: +`kubectl get scheduler` shows Ready, Desired, Available, and Age. The DaemonSet should have a pod on each eligible node, excluding nodes assigned to an older Scheduler. Read all conditions even if the wait succeeds: a partially conflicting Scheduler can be ready on its remaining nodes while also degraded. ```bash -kubectl get schedexts -A -kubectl get daemonset scx-rusty -n default -kubectl get pods -l name=scx-rusty -n default +kubectl describe scheduler scx-rusty +kubectl get scheduler scx-rusty -o jsonpath='{.status.resolvedImage}{"\n"}' ``` -You should see one DaemonSet, and one pod per node where the DaemonSet was scheduled. The pods are owned by the DaemonSet, which is owned by the `SchedExt`. +## Verify the kernel state -Important: `kubectl get schedexts` will not report any status today. The CRD has a status subresource declared but the controller never writes to it. To gauge health, look at the DaemonSet's `numberReady` / `desiredNumberScheduled` and the pods' state directly. +DaemonSet readiness does not prove that the BPF scheduler attached. On each target node, inspect: -## Step 4: verify on a node +```bash +cat /sys/kernel/sched_ext/state +cat /sys/kernel/sched_ext/root/ops +``` -To confirm the scheduler is actually loaded on a node: +Expect `state` to be `enabled` and `ops` to name the scheduler you deployed. If the scheduler image contains `cat` and exposes those sysfs files, you can also read them from its pod: ```bash -kubectl exec -n default -- cat /sys/kernel/sched_ext/state -kubectl exec -n default -- cat /sys/kernel/sched_ext/root/ops +kubectl exec -n sked-system -c scx -- cat /sys/kernel/sched_ext/state +kubectl exec -n sked-system -c scx -- cat /sys/kernel/sched_ext/root/ops ``` -Because the pod runs privileged, it has access to the host's `/sys` and the values you read are the kernel's view. `state` should be `enabled`, and `ops` should be the scheduler's struct_ops name. - -If `state` is `disabled`, look at the pod's logs: +If not, use a node shell. A privileged container is not a promise that the image includes debugging tools. For attachment failures, check the scheduler logs and node kernel logs: ```bash -kubectl logs -n default +kubectl logs -n sked-system -c scx ``` -A scheduler container can fail to attach for the same reasons a host-side `schedctl run` can fail: BPF verifier rejecting the program, missing CONFIG_*, missing BTF, or LSM denials. The [schedctl troubleshooting guide](../schedctl/troubleshooting) covers the kernel-side modes; they apply identically here. +The kernel checks in [schedctl troubleshooting](../schedctl/troubleshooting) also apply to scheduler images run by sked. -## Common operations +## Change configuration -### Update a scheduler image - -Edit the resource and re-apply: +Edit `scx-rusty.yaml` and reapply it, or run: ```bash -kubectl edit schedext scx-rusty +kubectl edit scheduler scx-rusty ``` -Change `spec.sched`. The controller patches the DaemonSet, which triggers a rolling pod update. The default DaemonSet update strategy applies — that's what governs how the rollout proceeds. The operator does not currently expose any roll-out tuning of its own. +Changing the image, arguments, environment, or other pod-template fields causes a DaemonSet rolling update. sked does not expose rollout tuning or automatic rollback. To revert, restore the previous Scheduler spec, preferably with its recorded image digest. -### Stop a scheduler +After an update, compare `status.observedGeneration` and each condition's `observedGeneration` with `metadata.generation`. A condition from an earlier generation may still be visible immediately after applying a change. -Delete the `SchedExt`: +### Tainted nodes -```bash -kubectl delete schedext scx-rusty -``` +Add a toleration to the Scheduler when the selected nodes have a matching taint: -The owner reference cascade deletes the DaemonSet and its pods. As pods exit, `sched_ext` ops detach in the kernel and the affected nodes revert to their default scheduling policy. +```yaml +spec: + sched: ghcr.io/schedkit/scx_rusty:latest + nodeSelector: + schedkit.io/pool: batch + tolerations: + - key: dedicated + operator: Equal + value: batch + effect: NoSchedule +``` -### Constraining where the scheduler runs +Keep the selector: a toleration permits placement but does not restrict it. Set these fields on the Scheduler, not on the generated DaemonSet or the chart's controller placement values. -Because the CRD has no `nodeSelector` field, you can't tell sked "only run on these nodes" via the resource itself. The current options are: +### Different schedulers on different pools -1. **Taint the nodes you don't want the DaemonSet on.** Without a corresponding toleration on the pod template, the DaemonSet won't schedule there. The operator has no way to add tolerations today, so this is one-directional — you exclude nodes, you can't include otherwise-tainted nodes. -2. **Run only one scheduler cluster-wide.** This is the simplest path and works fine when your cluster is homogeneous. -3. **Wait for the CRD to grow these fields.** They're on the to-do list. Pinning yourself to clever workarounds is probably not worth it unless you have to ship today. +Create one Scheduler per pool, with non-overlapping selectors. For example, keep `scx-rusty` on `schedkit.io/pool=batch` and use a separate Scheduler for nodes labelled `schedkit.io/pool=latency`. -### Multiple scheduler images on different node sets +If selectors overlap, the oldest Scheduler wins each node. Losers exclude conflict nodes or lose their entire DaemonSet if all selected nodes conflict. Conflicts appear in `Degraded` with reason `SchedulerConflict`. Tolerations and pod health do not affect the winner. See the [CRD reference](./crds#overlapping-selectors) before changing labels on a live pool. -Today, ill-advised. The kernel attaches one `sched_ext` scheduler at a time per node, and applying two `SchedExt` resources without keeping their DaemonSet pods apart will land both on the same nodes — only one will attach, the other will fail. If you really need this before the CRD grows node selection, your options are: +## Diagnose a failed deployment -- Use node taints/labels and the cluster's own DaemonSet scheduling rules (which sked can't influence per-resource today) to ensure the two `SchedExt` resources' pods don't overlap. In practice that means you need to *patch* the generated DaemonSet to add a node selector after the fact, which is fragile. -- Run only one `SchedExt` and accept a single cluster-wide scheduler. +Start with the resource, then the DaemonSet and pods: -## Things to know +```bash +kubectl describe scheduler scx-rusty +kubectl describe daemonset scx-rusty -n sked-system +kubectl get pods -n sked-system -l name=scx-rusty -o wide +kubectl get events -n default \ + --field-selector involvedObject.kind=Scheduler,involvedObject.name=scx-rusty \ + --sort-by=.metadata.creationTimestamp +kubectl logs -n sked-system -l control-plane=controller-manager -c manager --tail=100 +``` -A few practical notes that aren't quite "troubleshooting" but are worth absorbing early. +Scheduler events live in `default` because the resource is cluster-scoped. Pod and DaemonSet events live in the workload namespace. -**There is no status to look at on the resource.** Until the controller writes status, `kubectl describe schedext` is mostly an echo of `spec`. Look at the DaemonSet and its pods for actual health. +| Symptom | Check | +| --- | --- | +| `kubectl apply` is rejected | Webhook availability, spec validation, caller authorization, allowlist, registry access, and signature identity. Admission failure may mean no Scheduler exists yet. | +| `ImageVerificationFailed` | Controller logs and [trust policy](./trust-policy). Existing workloads are not replaced on verification failure. | +| `SchedulerConflict` | Other Schedulers' selectors and creation times; avoid overlap. | +| `NoNodesScheduled` | Node labels, taints, tolerations, and DaemonSet events. | +| Pods rejected by admission | The **workload namespace** must allow privileged containers. | +| `ErrImagePull` / `ImagePullBackOff` | Pod events, the resolved image, and `spec.imagePullSecrets` in the workload namespace. | +| `NodesUnavailable` or crashing pods | `kubectl describe pod`, container logs, node kernel support, and any competing scheduler outside sked. | +| Ready pods but no attached scheduler | Kernel sysfs and logs. There is no kernel-aware readiness probe. | -**There is no image signature verification on the cluster path.** schedctl verifies cosign signatures on the host; sked currently does not. If supply-chain integrity matters for your cluster, you'll want admission-controller-level image policy (e.g. `policy-controller`, `cosign-controller`) until the operator gains its own verification. +Do not patch the generated DaemonSet to fix its configuration; the controller will replace that spec. Use the Scheduler fields listed in the [CRD reference](./crds). -**Privileged pods are required.** The DaemonSet creates pods with `securityContext.privileged: true`. If your cluster's Pod Security admission level is set to `baseline` or `restricted`, the pods will be rejected. The `sked-system` namespace itself is *not* labelled to bypass admission, so the *namespace where you create `SchedExt` resources* needs to permit privileged workloads. +## Stop a scheduler -**`kubectl describe daemonset `** is the most useful command when something looks wrong. The events show pull errors, admission rejections, and node-selection issues in plain language. +```bash +kubectl delete scheduler scx-rusty +kubectl wait --for=delete daemonset/scx-rusty -n sked-system --timeout=2m +kubectl get pods -n sked-system -l name=scx-rusty +``` -## Where to next +Kubernetes garbage-collects the owned DaemonSet and pods. Wait for the pods to terminate and check kernel state. The kernel returns to its default scheduling policy when the scheduler detaches, unless another scheduler attaches in its place. Deleting the winner can let a previously conflicting Scheduler claim those nodes. -- [CRD reference](./crds) for the full field-by-field schema. -- [Architecture](./architecture) if you want to understand what the controller is actually doing. -- [schedctl troubleshooting](../schedctl/troubleshooting) if a node's scheduler refuses to attach — those are kernel-layer problems, not sked-layer ones. +Deleting the operator's Helm release alone does **not** stop scheduler workloads. See [Uninstall](./installation#uninstall). diff --git a/sidebars.ts b/sidebars.ts index 54d3a5a..ca36909 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -36,6 +36,7 @@ const sidebars: SidebarsConfig = { 'sked/overview', 'sked/installation', 'sked/crds', + 'sked/trust-policy', 'sked/usage', 'sked/architecture', 'sked/development',