diff --git a/README.md b/README.md index 27b55eedf..0564004c8 100644 --- a/README.md +++ b/README.md @@ -165,6 +165,7 @@ Apps communicate with the guest agent via HTTP over `/var/run/dstack.sock`. Use - [Intel TDX Attestation](./docs/attestation-tdx.md) - Measurement and runtime-event verification - [Native TEE Interfaces](./docs/native-tee-interfaces.md) - Advanced compatibility with Linux TEE devices and configfs-tsm - [VMM CLI Guide](./docs/vmm-cli-user-guide.md) - Command-line reference +- [VMM Top](./docs/vmm-top.md) - Terminal dashboard for one or many VMM nodes - [Design Decisions](./docs/design-and-hardening-decisions.md) - Architecture rationale - [FAQ](./docs/faq.md) - Frequently asked questions diff --git a/docs/vmm-top.md b/docs/vmm-top.md new file mode 100644 index 000000000..aed354194 --- /dev/null +++ b/docs/vmm-top.md @@ -0,0 +1,127 @@ +# Watching VMM nodes from a terminal + +`dstack-vmm-top` is a live terminal dashboard for one or many `dstack-vmm` nodes. It shows every CVM's state, boot progress and resource allocation, each node's capacity, and a running journal of what changed. From the same screen you can tail a VM's serial, stdout and stderr logs and start, shut down, restart or stop it. + +It is only ever a client. It reads each node's existing VMM API, or reads a hub that polls many nodes on its behalf, so watching a whole fleet needs no change to the VMMs. + +## Quick start + +Run it on a host with a VMM and it finds the local instances on its own, from the registration files every `dstack-vmm` writes under `$XDG_RUNTIME_DIR/dstack-vmm/`: + +```bash +cargo build --release -p dstack-vmm-top +./target/release/dstack-vmm-top +``` + +Watch other nodes by naming them. `--node` takes a VMM address, `unix:/path/vmm.sock` or `http(s)://host:port`, optionally prefixed with a display name: + +```bash +export DSTACK_VMM_TOKEN=... # only if the VMMs have [auth] enabled +dstack-vmm-top -n tdx-01=http://10.0.0.11:9080 -n tdx-02=http://10.0.0.12:9080 +``` + +Piped, or with `--once`, it prints one plain-text snapshot and exits. `--json` prints the same data as JSON: + +```text +$ dstack-vmm-top -n tdx-01=http://10.0.0.11:9080 -n tdx-04=http://10.0.0.14:9080 --once +NODE STATUS VMS VCPU MEM GPU RTT ENDPOINT +tdx-01 online 4/6 28/96 112G/384G 1/4 15ms http://10.0.0.11:9080 +tdx-04 down - - - - - http://10.0.0.14:9080 + error: error sending request for url (http://10.0.0.14:9080/prpc/Status?json) + +VM NODE STATUS UPTIME VCPU MEM DISK ID IMAGE +postgres-primary tdx-01 exited - 8 32G 40G b8b6d8fe dstack-0.5.4 +api-gateway tdx-01 booting 0s 8 32G 20G 1e2feb89 dstack-0.5.4 +kms-auth tdx-01 running 3d17h 8 32G 200G 1622bd79 dstack-0.5.4 +``` + +## Watching a fleet through a hub + +A VMM usually listens on a unix socket only, each node has its own token, and every operator polling every node multiplies the load. A hub solves all three. Run `dstack-vmm-top serve` next to the VMMs: it reaches their sockets, holds their tokens, polls each node once, and serves the result to any number of terminals under one URL and one token of its own. + +```mermaid +flowchart LR + A[dstack-vmm-top] -- "--hub, hub token" --> H[dstack-vmm-top serve] + A -- "--node, VMM token" --> V3[VMM tdx-03] + H -- "unix socket" --> V1[VMM tdx-01] + H -- "http + token" --> V2[VMM tdx-02] +``` + +On the hub host, with nothing configured, `serve` watches the local VMMs: + +```bash +export DSTACK_VMM_TOP_TOKEN=$(openssl rand -hex 32) +dstack-vmm-top serve --listen 0.0.0.0:9970 --name dc-east +``` + +Then from anywhere, using the same token: + +```bash +export DSTACK_VMM_TOP_TOKEN=... +dstack-vmm-top --hub https://vmm-hub.example:9970 +``` + +`--hub` and `--node` mix freely, so one terminal can watch several hubs plus a few nodes polled directly. Nodes seen through a hub are labeled `via `. + +The hub is read-only by construction. It serves snapshots and log tails and has no route that changes a VM, so its token lets someone watch but never stop anything. Start, shutdown, restart and stop are offered only for nodes polled directly with their own VMM token. The hub never sends VM secrets: no encrypted env, user config or compose file is in a snapshot. It refuses to listen on a non-loopback address without a token unless given `--insecure`, and it serves plain HTTP, so put it behind a TLS-terminating proxy, or an SSH tunnel, when it crosses an untrusted network. + +| Route | Answers | +|---|---| +| `GET /healthz` | `ok`, no token needed | +| `GET /api/v1/nodes` | every node's latest snapshot as JSON | +| `GET /api/v1/logs?node=&id=&ch=serial\|stdout\|stderr&lines=` | the last lines of one VM's log, for VMs the node listed | + +## Config file + +Long node lists belong in `~/.config/dstack/vmm-top.toml` (or pass `-c FILE`). Both the TUI and the hub read it: + +```toml +interval = "2s" +read_only = false +# Also watch this host's VMMs. Defaults to on only when nothing else is listed. +# discover = true + +[[node]] +name = "tdx-01" +url = "http://10.0.0.11:9080" +token_env = "TDX01_VMM_TOKEN" # or token = "..." + +[[hub]] +name = "dc-east" +url = "https://vmm-hub.east.example:9970" +token_env = "DC_EAST_HUB_TOKEN" + +# Used by `dstack-vmm-top serve`. +[serve] +name = "dc-west" +listen = "0.0.0.0:9970" +token_env = "DSTACK_VMM_TOP_TOKEN" +``` + +A node without its own token gets `--token` or `DSTACK_VMM_TOKEN`, and a hub without one gets `--hub-token` or `DSTACK_VMM_TOP_TOKEN`. A VMM token is never sent to a hub. + +## Reading the screen + +The **Fleet** tab is a tree of nodes and their VMs. A node row rolls up its VMs: allocated vCPU and memory against the node's limits, provisioned disk, attached GPUs, and the round-trip time of the last poll. The header totals the same across all nodes, and on wide terminals a board lists every node's load side by side. **VMs** is one flat, sortable list across all nodes. **Events** is the journal: VM state changes (`running → exited`), nodes going down and coming back, actions taken from this terminal, and the guest's own boot events. + +A VM's status folds the VMM's runtime status together with what the guest reported, the way the web console does. `booting` shows the current boot step (`requesting app keys`), `failed` shows the boot error inline, and anything in transition spins. Allocation counts only VMs holding host resources, so an exited or stopped VM does not use up a node's vCPU. + +When a node stops answering, its last good data stays on screen, dimmed and marked `?`, with how long ago it was fresh. A dead node never makes its VMs vanish from view. + +## Keys + +| Key | Action | +|---|---| +| `1` `2` `3`, `tab` | Fleet · VMs · Events | +| `↑` `↓`, `j` `k`, `pgup` `pgdn`, `g` `G` | move | +| `←` `→`, `h` `l`, `z` | fold or unfold a node, all nodes | +| `enter` | full-screen detail: Info, Serial, Stdout, Stderr, Events | +| `v` | preview pane under the list | +| `x` | act on the VM: `u` start, `d` shutdown, `r` restart (force stop, then start), `k` force stop, then `y` to confirm | +| `/` | filter by name, id, app id, image, status or node | +| `a` | scope: all · live (holding resources) · attention (failed, exited, in transition, unreachable) | +| `s` `S`, `I` | sort by status, name, node, uptime, vcpu or memory; invert | +| `r`, `p`, `+` `-` | refresh now, pause, change the poll interval | +| `?`, `q` | help, quit | + +In the detail view, logs follow the tail until you scroll up, `f` toggles following, and `←` `→` pan long lines. The mouse works too: click to select, click a selected node to fold it, double-click to open, and scroll with the wheel. diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index 31a97f389..b0b286fc1 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -138,7 +138,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -149,7 +149,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -806,6 +806,21 @@ dependencies = [ "tokio", ] +[[package]] +name = "cassowary" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df8670b8c7b9dae1793364eafadf7239c40d669904660c5960d74cfd80b46a53" + +[[package]] +name = "castaway" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a" +dependencies = [ + "rustversion", +] + [[package]] name = "cc" version = "1.2.63" @@ -1094,6 +1109,20 @@ dependencies = [ "memchr", ] +[[package]] +name = "compact_str" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fd622ebbb56a5b2ccb651b32b911cdeb2a9b4b11776b2473bf26a26a286244e" +dependencies = [ + "castaway", + "cfg-if", + "itoa", + "rustversion", + "ryu", + "static_assertions", +] + [[package]] name = "console" version = "0.16.3" @@ -1287,7 +1316,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c36c10130df424b2f3552fcc2ddcd9b28a27b1e54b358b45874f88d1ca6888c" dependencies = [ "bitflags 1.3.2", - "crossterm_winapi", + "crossterm_winapi 0.7.0", "lazy_static", "libc", "mio 0.7.14", @@ -1296,6 +1325,22 @@ dependencies = [ "winapi", ] +[[package]] +name = "crossterm" +version = "0.28.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "829d955a0bb380ef178a640b91779e3987da38c9aea133b20614cfed8cdea9c6" +dependencies = [ + "bitflags 2.11.1", + "crossterm_winapi 0.9.1", + "mio 1.2.1", + "parking_lot 0.12.5", + "rustix 0.38.44", + "signal-hook 0.3.18", + "signal-hook-mio", + "winapi", +] + [[package]] name = "crossterm_winapi" version = "0.7.0" @@ -1305,6 +1350,15 @@ dependencies = [ "winapi", ] +[[package]] +name = "crossterm_winapi" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "acdd7c62a3665c7f6830a51635d9ac9b23ed385797f70a83bb8bafe9c572ab2b" +dependencies = [ + "winapi", +] + [[package]] name = "crunchy" version = "0.2.4" @@ -1420,6 +1474,16 @@ dependencies = [ "darling_macro 0.23.0", ] +[[package]] +name = "darling" +version = "0.24.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed17f5901b6630b993ca003def43f2f8ef4014fc13b047b57aad617ff32bc2ec" +dependencies = [ + "darling_core 0.24.1", + "darling_macro 0.24.1", +] + [[package]] name = "darling_core" version = "0.20.11" @@ -1447,6 +1511,19 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling_core" +version = "0.24.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6837e2cf7485aaae18f86181d2f0e9a7ed297a025e220aeabf63fdebd3a2ddff" +dependencies = [ + "ident_case", + "proc-macro2", + "quote", + "strsim", + "syn 3.0.9", +] + [[package]] name = "darling_macro" version = "0.20.11" @@ -1469,6 +1546,17 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "darling_macro" +version = "0.24.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785" +dependencies = [ + "darling_core 0.24.1", + "quote", + "syn 3.0.9", +] + [[package]] name = "dashmap" version = "6.2.1" @@ -1761,7 +1849,7 @@ dependencies = [ "libc", "option-ext", "redox_users 0.5.2", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1814,7 +1902,7 @@ dependencies = [ "optfield", "proc-macro2", "quote", - "strum", + "strum 0.27.2", "syn 2.0.117", ] @@ -2474,6 +2562,34 @@ dependencies = [ "serde_json", ] +[[package]] +name = "dstack-vmm-top" +version = "0.6.1" +dependencies = [ + "anyhow", + "chrono", + "clap", + "crossterm 0.28.1", + "dirs 6.0.0", + "dstack-vmm-rpc", + "fs-err", + "futures", + "http-body-util", + "http-client", + "humantime", + "hyper", + "hyper-util", + "libc", + "ratatui", + "serde", + "serde_json", + "tokio", + "toml", + "tracing", + "tracing-subscriber", + "unicode-width 0.2.0", +] + [[package]] name = "dstack-volume" version = "0.6.1" @@ -2680,7 +2796,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -3252,6 +3368,8 @@ version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ + "allocator-api2", + "equivalent", "foldhash 0.1.5", ] @@ -3596,7 +3714,7 @@ dependencies = [ "libc", "percent-encoding", "pin-project-lite", - "socket2 0.5.10", + "socket2 0.6.4", "tokio", "tower-service", "tracing", @@ -3790,6 +3908,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "indoc" +version = "2.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a37b2691796cffeb8a8cd305ac66e65841559f147f4e63231d0eafa4db5384d1" +dependencies = [ + "rustversion", +] + [[package]] name = "inlinable_string" version = "0.1.15" @@ -3837,6 +3964,19 @@ dependencies = [ "tempfile", ] +[[package]] +name = "instability" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c3b5acc1e2fd9375041a388da33d1eb8aed5f7a8c0dd3543e3ea2805adfbe20" +dependencies = [ + "darling 0.24.1", + "indoc", + "proc-macro2", + "quote", + "syn 3.0.9", +] + [[package]] name = "instant" version = "0.1.13" @@ -3930,7 +4070,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46" dependencies = [ "hermit-abi", "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -4328,6 +4468,15 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "lru" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" +dependencies = [ + "hashbrown 0.15.5", +] + [[package]] name = "lru" version = "0.18.1" @@ -4358,7 +4507,7 @@ dependencies = [ "aes 0.7.5", "base64 0.13.1", "bincode 1.3.3", - "crossterm", + "crossterm 0.19.0", "hmac 0.11.0", "pbkdf2", "rust-argon2", @@ -4795,7 +4944,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -5073,7 +5222,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d8fae84b431384b68627d0f9b3b1245fcf9f46f6c0e3dc902e9dce64edd1967" dependencies = [ "libc", - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -5777,7 +5926,7 @@ dependencies = [ "quinn-udp", "rustc-hash", "rustls", - "socket2 0.5.10", + "socket2 0.6.4", "thiserror 2.0.18", "tokio", "tracing", @@ -5816,7 +5965,7 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.5.10", + "socket2 0.6.4", "tracing", "windows-sys 0.60.2", ] @@ -5998,6 +6147,27 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "ratatui" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eabd94c2f37801c20583fc49dd5cd6b0ba68c716787c2dd6ed18571e1e63117b" +dependencies = [ + "bitflags 2.11.1", + "cassowary", + "compact_str", + "crossterm 0.28.1", + "indoc", + "instability", + "itertools 0.13.0", + "lru 0.12.5", + "paste", + "strum 0.26.3", + "unicode-segmentation", + "unicode-truncate", + "unicode-width 0.2.0", +] + [[package]] name = "rcgen" version = "0.13.2" @@ -6153,7 +6323,7 @@ dependencies = [ "ipnet", "jsonschema", "lazy_static", - "lru", + "lru 0.18.1", "msvc_spectre_libs", "num-bigint", "num-traits", @@ -6499,7 +6669,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.12.1", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -6567,7 +6737,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -7301,6 +7471,17 @@ dependencies = [ "signal-hook-registry", ] +[[package]] +name = "signal-hook-mio" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b75a19a7a740b25bc7944bdee6172368f988763b744e3d4dfe753f6b4ece40cc" +dependencies = [ + "libc", + "mio 1.2.1", + "signal-hook 0.3.18", +] + [[package]] name = "signal-hook-registry" version = "1.4.8" @@ -7421,7 +7602,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" dependencies = [ "libc", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -7503,13 +7684,35 @@ version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" +dependencies = [ + "strum_macros 0.26.4", +] + [[package]] name = "strum" version = "0.27.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf" dependencies = [ - "strum_macros", + "strum_macros 0.27.2", +] + +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.117", ] [[package]] @@ -7597,6 +7800,17 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "syn" +version = "3.0.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + [[package]] name = "sync_wrapper" version = "1.0.2" @@ -7738,7 +7952,7 @@ dependencies = [ "getrandom 0.4.2", "once_cell", "rustix 1.1.4", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -8282,6 +8496,29 @@ version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" +[[package]] +name = "unicode-truncate" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3644627a5af5fa321c95b9b235a72fd24cd29c648c2c379431e6628655627bf" +dependencies = [ + "itertools 0.13.0", + "unicode-segmentation", + "unicode-width 0.1.14", +] + +[[package]] +name = "unicode-width" +version = "0.1.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af" + +[[package]] +name = "unicode-width" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fc81956842c57dac11422a97c3b8195a1ff727f06e85c84ed2e8aa277c9a0fd" + [[package]] name = "unicode-xid" version = "0.2.6" @@ -8732,7 +8969,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index 8877f6296..456dd40e1 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -37,6 +37,7 @@ members = [ "vmm", "vmm/rpc", "vmm/netd-rpc", + "vmm-top", "gateway", "gateway/rpc", "certbot", @@ -289,9 +290,11 @@ derive_more = "2.1.1" tempfile = "3.18.0" # Utilities +crossterm = "0.28" dirs = "6.0.0" humantime = "2.2.0" parcelona = "0.4.3" +ratatui = "0.29" pin-project = "1.1.10" regex = "1.11.1" rinja = "0.3.5" @@ -299,6 +302,7 @@ shared_child = "1.0.1" strip-ansi-escapes = "0.2.1" tailf = "0.1.2" time = "0.3.47" +unicode-width = "0.2" uuid = { version = "1.15.1", features = ["v4"] } wait-timeout = "0.2" which = "7.0.2" diff --git a/dstack/vmm-top/Cargo.toml b/dstack/vmm-top/Cargo.toml new file mode 100644 index 000000000..2d6297825 --- /dev/null +++ b/dstack/vmm-top/Cargo.toml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: © 2026 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "dstack-vmm-top" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true +description = "terminal dashboard for one or many dstack-vmm nodes, plus a hub that aggregates them" + +[dependencies] +anyhow.workspace = true +chrono.workspace = true +clap = { workspace = true, features = ["derive", "env"] } +crossterm.workspace = true +dirs.workspace = true +dstack-vmm-rpc.workspace = true +fs-err.workspace = true +futures.workspace = true +http-body-util.workspace = true +http-client = { workspace = true, features = ["prpc"] } +humantime.workspace = true +hyper = { workspace = true, features = ["server", "http1"] } +hyper-util = { workspace = true, features = ["tokio"] } +libc.workspace = true +ratatui.workspace = true +serde = { workspace = true, features = ["derive", "std"] } +serde_json = { workspace = true, features = ["std"] } +tokio = { workspace = true, features = ["full"] } +toml.workspace = true +tracing.workspace = true +tracing-subscriber = { workspace = true, features = ["env-filter"] } +unicode-width.workspace = true diff --git a/dstack/vmm-top/src/collector.rs b/dstack/vmm-top/src/collector.rs new file mode 100644 index 000000000..3a3edc391 --- /dev/null +++ b/dstack/vmm-top/src/collector.rs @@ -0,0 +1,254 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Background polling, shared by the TUI and the hub. +//! +//! One task per source polls on the shared interval and sends what it got. +//! A failed poll keeps the last good data and marks it with the error, so a +//! node that drops off still shows what was on it. Log fetches and actions +//! run on their own tasks and report back the same way; nothing here ever +//! blocks the caller's loop. + +use crate::fmt::now_ms; +use crate::snapshot::NodeSnapshot; +use crate::source::{Action, LogChannel, Source, SourceKind}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; +use std::time::Duration; +use tokio::sync::{mpsc, Notify}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LogRequest { + pub source: usize, + pub node: String, + pub vm_id: String, + pub channel: LogChannel, + pub lines: u32, +} + +pub enum Command { + RefreshNow, + Logs(LogRequest), + Act { + source: usize, + vm_id: String, + what: String, + action: Action, + }, +} + +pub enum Update { + Nodes { + source: usize, + nodes: Vec, + }, + Logs { + req: LogRequest, + result: Result, + }, + Acted { + what: String, + action: Action, + result: Result<(), String>, + }, +} + +pub struct Handle { + pub commands: mpsc::UnboundedSender, + pub updates: mpsc::UnboundedReceiver, + /// Poll interval in ms; pollers read it before every sleep. + pub interval_ms: Arc, +} + +pub fn spawn(sources: Vec>, interval_ms: u64) -> Handle { + let (cmd_tx, mut cmd_rx) = mpsc::unbounded_channel::(); + let (up_tx, up_rx) = mpsc::unbounded_channel(); + let interval = Arc::new(AtomicU64::new(interval_ms)); + let wake = Arc::new(Notify::new()); + + for (idx, source) in sources.iter().enumerate() { + tokio::spawn(poll_loop( + idx, + source.clone(), + interval.clone(), + wake.clone(), + up_tx.clone(), + )); + } + + tokio::spawn(async move { + while let Some(cmd) = cmd_rx.recv().await { + match cmd { + Command::RefreshNow => wake.notify_waiters(), + Command::Logs(req) => { + let Some(source) = sources.get(req.source).cloned() else { + continue; + }; + let tx = up_tx.clone(); + tokio::spawn(async move { + let result = source + .logs(&req.node, &req.vm_id, req.channel, req.lines) + .await + .map_err(|err| format!("{err:#}")); + let _ = tx.send(Update::Logs { req, result }); + }); + } + Command::Act { + source, + vm_id, + what, + action, + } => { + let Some(source) = sources.get(source).cloned() else { + continue; + }; + let tx = up_tx.clone(); + let wake = wake.clone(); + tokio::spawn(async move { + let result = source + .act(&vm_id, action) + .await + .map_err(|err| format!("{err:#}")); + let _ = tx.send(Update::Acted { + what, + action, + result, + }); + wake.notify_waiters(); + }); + } + } + } + }); + + Handle { + commands: cmd_tx, + updates: up_rx, + interval_ms: interval, + } +} + +async fn poll_loop( + idx: usize, + source: Arc, + interval: Arc, + wake: Arc, + tx: mpsc::UnboundedSender, +) { + let mut last: Vec = Vec::new(); + loop { + let result = source.poll().await; + let now = now_ms(); + last = merge_poll(&source, last, result, now); + if tx + .send(Update::Nodes { + source: idx, + nodes: last.clone(), + }) + .is_err() + { + return; + } + let wait = Duration::from_millis(interval.load(Ordering::Relaxed).max(200)); + tokio::select! { + _ = tokio::time::sleep(wait) => {} + _ = wake.notified() => {} + } + } +} + +/// The nodes to show after a poll: fresh ones on success, the previous ones +/// marked with the error on failure. +fn merge_poll( + source: &Source, + last: Vec, + result: anyhow::Result>, + now: u64, +) -> Vec { + match result { + Ok(mut nodes) => { + if source.spec.kind == SourceKind::Vmm { + for n in &mut nodes { + n.polled_at_ms = now; + n.ok_at_ms = Some(now); + } + } + nodes + } + Err(err) => { + let msg = match source.spec.kind { + SourceKind::Vmm => format!("{err:#}"), + SourceKind::Hub => format!("hub {} unreachable: {err:#}", source.spec.name), + }; + if last.is_empty() { + return vec![NodeSnapshot { + name: source.spec.name.clone(), + endpoint: source.endpoint(), + via: (source.spec.kind == SourceKind::Hub).then(|| source.spec.name.clone()), + polled_at_ms: now, + error: Some(msg), + ..Default::default() + }]; + } + last.into_iter() + .map(|mut n| { + n.polled_at_ms = now; + n.error = Some(msg.clone()); + n + }) + .collect() + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::source::SourceSpec; + + fn source(kind: SourceKind) -> Source { + Source::new( + SourceSpec { + name: "n1".into(), + kind, + url: "http://h:1".into(), + token: None, + }, + false, + ) + .unwrap() + } + + #[test] + fn failure_keeps_last_good_data() { + let s = source(SourceKind::Vmm); + let good = merge_poll( + &s, + vec![], + Ok(vec![NodeSnapshot { + name: "n1".into(), + vms: vec![Default::default()], + ..Default::default() + }]), + 1000, + ); + assert_eq!(good[0].ok_at_ms, Some(1000)); + let bad = merge_poll(&s, good, Err(anyhow::anyhow!("refused")), 2000); + assert_eq!(bad.len(), 1); + assert_eq!(bad[0].vms.len(), 1); + assert_eq!(bad[0].ok_at_ms, Some(1000)); + assert_eq!(bad[0].polled_at_ms, 2000); + assert_eq!(bad[0].error.as_deref(), Some("refused")); + assert!(!bad[0].online()); + } + + #[test] + fn first_failure_makes_placeholder() { + let s = source(SourceKind::Hub); + let bad = merge_poll(&s, vec![], Err(anyhow::anyhow!("refused")), 5); + assert_eq!(bad[0].name, "n1"); + assert_eq!(bad[0].via.as_deref(), Some("n1")); + assert!(bad[0].error.as_deref().unwrap().contains("hub n1")); + } +} diff --git a/dstack/vmm-top/src/config.rs b/dstack/vmm-top/src/config.rs new file mode 100644 index 000000000..3d30e1205 --- /dev/null +++ b/dstack/vmm-top/src/config.rs @@ -0,0 +1,400 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Which sources to watch: a config file, `--node` / `--hub` flags, and, when +//! neither names any, the VMMs running on this host. +//! +//! ```toml +//! # ~/.config/dstack/vmm-top.toml +//! interval = "2s" +//! +//! [[node]] +//! name = "tdx-01" +//! url = "http://10.0.0.11:9080" +//! token_env = "TDX01_VMM_TOKEN" +//! +//! [[hub]] +//! name = "dc-east" +//! url = "https://vmm-hub.east.example:9970" +//! token = "…" +//! +//! [serve] +//! listen = "0.0.0.0:9970" +//! token_env = "DSTACK_VMM_TOP_TOKEN" +//! ``` + +use crate::source::{SourceKind, SourceSpec}; +use anyhow::{bail, Context, Result}; +use fs_err as fs; +use serde::Deserialize; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +/// Token used for any VMM that names none, as with `dstack` and `vmm-cli.py`. +pub const VMM_TOKEN_ENV: &str = "DSTACK_VMM_TOKEN"; +/// Token a hub requires, and a TUI sends to a hub that names none. +pub const HUB_TOKEN_ENV: &str = "DSTACK_VMM_TOP_TOKEN"; +/// The control socket `dstackup install` creates. +pub const DEFAULT_SOCKET: &str = "/var/run/dstack/vmm.sock"; + +#[derive(Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct FileConfig { + #[serde(default)] + pub interval: Option, + #[serde(default)] + pub read_only: bool, + /// Also watch the VMMs registered on this host. Defaults to on only when + /// nothing else is configured. + #[serde(default)] + pub discover: Option, + #[serde(default, rename = "node")] + pub nodes: Vec, + #[serde(default, rename = "hub")] + pub hubs: Vec, + #[serde(default)] + pub serve: ServeConfig, +} + +#[derive(Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Entry { + #[serde(default)] + pub name: Option, + pub url: String, + #[serde(default)] + pub token: Option, + /// Name of an environment variable holding the token; keeps it out of + /// the file. + #[serde(default)] + pub token_env: Option, +} + +#[derive(Debug, Default, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ServeConfig { + #[serde(default)] + pub name: Option, + #[serde(default)] + pub listen: Option, + #[serde(default)] + pub token: Option, + #[serde(default)] + pub token_env: Option, +} + +pub fn default_path() -> Option { + dirs::config_dir().map(|d| d.join("dstack").join("vmm-top.toml")) +} + +/// Read `path`, or the default file when `path` is `None`. A missing default +/// file is an empty config; a missing named one is an error. +pub fn load(path: Option<&Path>) -> Result { + let (path, required) = match path { + Some(p) => (p.to_path_buf(), true), + None => match default_path() { + Some(p) => (p, false), + None => return Ok(FileConfig::default()), + }, + }; + let text = match fs::read_to_string(&path) { + Ok(text) => text, + Err(err) if !required && err.kind() == std::io::ErrorKind::NotFound => { + return Ok(FileConfig::default()) + } + Err(err) => return Err(err.into()), + }; + toml::from_str(&text).with_context(|| format!("failed to parse {}", path.display())) +} + +pub fn resolve_token(token: Option<&str>, token_env: Option<&str>) -> Result> { + if let Some(var) = token_env { + return match std::env::var(var) { + Ok(v) if !v.trim().is_empty() => Ok(Some(v.trim().to_string())), + _ => bail!("environment variable {var} is not set"), + }; + } + Ok(token + .map(str::trim) + .filter(|t| !t.is_empty()) + .map(str::to_string)) +} + +fn env_token() -> Option { + std::env::var(VMM_TOKEN_ENV) + .ok() + .map(|t| t.trim().to_string()) + .filter(|t| !t.is_empty()) +} + +/// `NAME=URL` or a bare `URL`. +pub fn parse_flag(flag: &str) -> (Option, String) { + if let Some((name, url)) = flag.split_once('=') { + if !name.is_empty() && !name.contains(['/', ':']) { + return (Some(name.to_string()), url.to_string()); + } + } + (None, flag.to_string()) +} + +/// A name for a source given only its address: the host of a URL, `local` +/// for a socket. +pub fn name_from_url(url: &str) -> String { + if url.starts_with("unix:") { + return "local".into(); + } + let rest = url.split_once("://").map_or(url, |(_, r)| r); + let authority = rest.split('/').next().unwrap_or(rest); + let host = authority.rsplit_once('@').map_or(authority, |(_, h)| h); + let host = if host.starts_with('[') { + host.split_once(']') + .map_or(host, |(h, _)| h.trim_start_matches('[')) + } else { + host.split(':').next().unwrap_or(host) + }; + if host.is_empty() { + url.to_string() + } else { + host.to_string() + } +} + +pub struct Flags<'a> { + pub nodes: &'a [String], + pub hubs: &'a [String], + /// For VMMs that name no token of their own. + pub token: Option<&'a str>, + /// For hubs that name no token of their own. + pub hub_token: Option<&'a str>, +} + +/// Every source to watch, with unique names. +pub fn sources(file: &FileConfig, flags: Flags) -> Result> { + let fallback = flags.token.map(str::to_string).or_else(env_token); + // A hub has its own token; a VMM token is never sent to one. + let hub_fallback = flags.hub_token.map(str::to_string); + let mut out = Vec::new(); + for e in &file.nodes { + out.push(entry_spec(e, SourceKind::Vmm, fallback.clone())?); + } + for e in &file.hubs { + out.push(entry_spec(e, SourceKind::Hub, hub_fallback.clone())?); + } + for flag in flags.nodes { + let (name, url) = parse_flag(flag); + out.push(SourceSpec { + name: name.unwrap_or_else(|| name_from_url(&url)), + kind: SourceKind::Vmm, + url, + token: fallback.clone(), + }); + } + for flag in flags.hubs { + let (name, url) = parse_flag(flag); + out.push(SourceSpec { + name: name.unwrap_or_else(|| name_from_url(&url)), + kind: SourceKind::Hub, + url, + token: hub_fallback.clone(), + }); + } + if file.discover.unwrap_or(out.is_empty()) { + out.extend(discover(fallback.as_deref())); + } + if out.is_empty() && Path::new(DEFAULT_SOCKET).exists() { + out.push(SourceSpec { + name: "local".into(), + kind: SourceKind::Vmm, + url: format!("unix:{DEFAULT_SOCKET}"), + token: fallback, + }); + } + dedupe_names(&mut out); + Ok(out) +} + +fn entry_spec(e: &Entry, kind: SourceKind, fallback: Option) -> Result { + let token = resolve_token(e.token.as_deref(), e.token_env.as_deref()) + .with_context(|| format!("token for {}", e.url))? + .or(fallback); + Ok(SourceSpec { + name: e.name.clone().unwrap_or_else(|| name_from_url(&e.url)), + kind, + url: e.url.clone(), + token, + }) +} + +/// `name`, `name-2`, `name-3`, … so every node key is distinct. +pub fn dedupe_names(specs: &mut [SourceSpec]) { + let mut seen: HashMap = HashMap::new(); + for s in specs.iter_mut() { + let n = seen.entry(s.name.clone()).or_insert(0); + *n += 1; + if *n > 1 { + s.name = format!("{}-{n}", s.name); + } + } +} + +#[derive(Debug, Deserialize)] +struct Registration { + pid: u32, + address: String, + #[serde(default)] + working_dir: String, + #[serde(default)] + node_name: String, + #[serde(default)] + started_at: u64, +} + +/// The VMMs this user runs on this host, from the registration files each +/// `dstack-vmm` writes at startup (see `vmm/src/discovery.rs`). +pub fn discover(token: Option<&str>) -> Vec { + let dir = match std::env::var("XDG_RUNTIME_DIR") { + Ok(x) if !x.is_empty() => PathBuf::from(x).join("dstack-vmm"), + // SAFETY: getuid has no preconditions and cannot fail. + _ => PathBuf::from(format!("/run/user/{}/dstack-vmm", unsafe { + libc::getuid() + })), + }; + let Ok(entries) = fs::read_dir(&dir) else { + return Vec::new(); + }; + let mut regs: Vec = entries + .flatten() + .filter(|e| e.path().extension().is_some_and(|x| x == "json")) + .filter_map(|e| fs::read(e.path()).ok()) + .filter_map(|b| serde_json::from_slice::(&b).ok()) + .filter(|r| Path::new(&format!("/proc/{}", r.pid)).exists()) + .collect(); + regs.sort_by_key(|r| r.started_at); + regs.into_iter() + .filter_map(|r| { + let url = registration_url(&r.address, &r.working_dir)?; + Some(SourceSpec { + name: if r.node_name.is_empty() { + "local".into() + } else { + r.node_name + }, + kind: SourceKind::Vmm, + url, + token: token.map(str::to_string), + }) + }) + .collect() +} + +/// The address a VMM registered, as something to connect to. +pub fn registration_url(address: &str, working_dir: &str) -> Option { + if let Some(sock) = address.strip_prefix("unix:") { + let path = Path::new(sock); + let path = if path.is_absolute() { + path.to_path_buf() + } else { + Path::new(working_dir).join(path) + }; + return Some(format!("unix:{}", path.display())); + } + if address.starts_with("http://") || address.starts_with("https://") { + return Some(address.to_string()); + } + let (host, port) = address.rsplit_once(':')?; + let host = match host { + "0.0.0.0" | "" => "127.0.0.1", + "[::]" | "::" => "[::1]", + h => h, + }; + Some(format!("http://{host}:{port}")) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn flags_take_optional_name() { + assert_eq!( + parse_flag("tdx=http://h:1"), + (Some("tdx".into()), "http://h:1".into()) + ); + assert_eq!( + parse_flag("http://h:1/?a=b"), + (None, "http://h:1/?a=b".into()) + ); + assert_eq!(parse_flag("unix:/x=y"), (None, "unix:/x=y".into())); + } + + #[test] + fn names_come_from_hosts() { + assert_eq!(name_from_url("http://10.0.0.1:9080"), "10.0.0.1"); + assert_eq!(name_from_url("https://u:p@vmm.example/x"), "vmm.example"); + assert_eq!(name_from_url("http://[fe80::1]:80"), "fe80::1"); + assert_eq!(name_from_url("unix:/run/vmm.sock"), "local"); + } + + #[test] + fn registration_addresses() { + assert_eq!( + registration_url("unix:./vmm.sock", "/srv/vmm").as_deref(), + Some("unix:/srv/vmm/./vmm.sock") + ); + assert_eq!( + registration_url("0.0.0.0:9080", "/").as_deref(), + Some("http://127.0.0.1:9080") + ); + assert_eq!( + registration_url("http://h:1", "/").as_deref(), + Some("http://h:1") + ); + } + + #[test] + fn duplicate_names_get_suffixes() { + let spec = |n: &str| SourceSpec { + name: n.into(), + kind: SourceKind::Vmm, + url: "unix:/x".into(), + token: None, + }; + let mut v = vec![spec("a"), spec("a"), spec("b"), spec("a")]; + dedupe_names(&mut v); + let names: Vec<_> = v.iter().map(|s| s.name.as_str()).collect(); + assert_eq!(names, ["a", "a-2", "b", "a-3"]); + } + + #[test] + fn config_file_parses() { + let cfg: FileConfig = toml::from_str( + r#" + interval = "5s" + [[node]] + name = "a" + url = "http://a:1" + token = "t" + [[hub]] + url = "https://hub:9970" + [serve] + listen = "127.0.0.1:9970" + "#, + ) + .unwrap(); + let specs = sources( + &cfg, + Flags { + nodes: &[], + hubs: &[], + token: None, + hub_token: None, + }, + ) + .unwrap(); + assert_eq!(specs.len(), 2); + assert_eq!(specs[0].token.as_deref(), Some("t")); + assert_eq!(specs[1].kind, SourceKind::Hub); + assert_eq!(specs[1].name, "hub"); + } +} diff --git a/dstack/vmm-top/src/fmt.rs b/dstack/vmm-top/src/fmt.rs new file mode 100644 index 000000000..67fd051d2 --- /dev/null +++ b/dstack/vmm-top/src/fmt.rs @@ -0,0 +1,212 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Text helpers that measure by display width, so CJK names and emoji keep +//! columns aligned. + +use unicode_width::{UnicodeWidthChar, UnicodeWidthStr}; + +pub fn width(s: &str) -> usize { + UnicodeWidthStr::width(s) +} + +/// At most `max` columns of `s`, ending in "…" when cut. +pub fn truncate(s: &str, max: usize) -> String { + if width(s) <= max { + return s.to_string(); + } + if max == 0 { + return String::new(); + } + let mut out = String::new(); + let mut used = 0; + for c in s.chars() { + let w = c.width().unwrap_or(0); + if used + w > max - 1 { + break; + } + used += w; + out.push(c); + } + out.push('…'); + out +} + +/// `s` cut or padded with spaces to exactly `w` columns. +pub fn pad(s: &str, w: usize) -> String { + let t = truncate(s, w); + let tw = width(&t); + format!("{t}{}", " ".repeat(w.saturating_sub(tw))) +} + +/// A log line made safe to draw: ANSI escape sequences removed (they would +/// move the cursor or recolor the screen), tabs expanded, other control +/// characters dropped. +pub fn sanitize(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut chars = s.chars().peekable(); + while let Some(c) = chars.next() { + match c { + '\x1b' => match chars.peek() { + // CSI: parameters up to a final byte in @..~. + Some('[') => { + chars.next(); + for c in chars.by_ref() { + if ('@'..='~').contains(&c) { + break; + } + } + } + // OSC: up to BEL or ST. + Some(']') => { + chars.next(); + while let Some(c) = chars.next() { + if c == '\x07' || (c == '\x1b' && chars.next_if_eq(&'\\').is_some()) { + break; + } + } + } + _ => { + chars.next(); + } + }, + '\t' => out.push_str(" "), + c if c.is_control() => {} + c => out.push(c), + } + } + out +} + +/// Two most significant units: `42s`, `5m09s`, `3h12m`, `2d04h`. +pub fn duration(secs: u64) -> String { + let (d, h, m, s) = ( + secs / 86400, + secs % 86400 / 3600, + secs % 3600 / 60, + secs % 60, + ); + if d > 0 { + format!("{d}d{h:02}h") + } else if h > 0 { + format!("{h}h{m:02}m") + } else if m > 0 { + format!("{m}m{s:02}s") + } else { + format!("{s}s") + } +} + +/// `512M`, `8G`, `1.5T` from megabytes. +pub fn mem_mb(mb: u64) -> String { + if mb >= 1024 * 1024 { + trim_float(mb as f64 / (1024.0 * 1024.0), "T") + } else if mb >= 1024 { + trim_float(mb as f64 / 1024.0, "G") + } else { + format!("{mb}M") + } +} + +/// `40G`, `1.2T` from gigabytes. +pub fn disk_gb(gb: u64) -> String { + if gb >= 1024 { + trim_float(gb as f64 / 1024.0, "T") + } else { + format!("{gb}G") + } +} + +fn trim_float(v: f64, unit: &str) -> String { + if v >= 100.0 || (v - v.round()).abs() < 0.05 { + format!("{:.0}{unit}", v) + } else { + format!("{:.1}{unit}", v) + } +} + +/// `3s ago`, `5m ago`, from two unix-ms instants. +pub fn ago(now_ms: u64, then_ms: u64) -> String { + let secs = now_ms.saturating_sub(then_ms) / 1000; + let d = duration(secs); + // Only the leading unit reads well in a sentence. + let lead: String = d + .chars() + .scan(false, |seen_unit, c| { + if *seen_unit && c.is_ascii_digit() { + return None; + } + if c.is_ascii_alphabetic() { + *seen_unit = true; + } + Some(c) + }) + .collect(); + format!("{lead} ago") +} + +/// Local wall clock `HH:MM:SS` of a unix-ms instant. +pub fn clock(ms: u64) -> String { + use chrono::{Local, TimeZone}; + match Local.timestamp_millis_opt(ms as i64) { + chrono::LocalResult::Single(t) => t.format("%H:%M:%S").to_string(), + _ => "--:--:--".into(), + } +} + +/// Local `MM-DD HH:MM:SS` of a unix-ms instant. +pub fn stamp(ms: u64) -> String { + use chrono::{Local, TimeZone}; + match Local.timestamp_millis_opt(ms as i64) { + chrono::LocalResult::Single(t) => t.format("%m-%d %H:%M:%S").to_string(), + _ => "--".into(), + } +} + +pub fn now_ms() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn truncate_by_width() { + assert_eq!(truncate("hello", 5), "hello"); + assert_eq!(truncate("hello world", 6), "hello…"); + assert_eq!(truncate("名字很长的", 5), "名字…"); + assert_eq!(width(&pad("名字", 6)), 6); + } + + #[test] + fn durations_keep_two_units() { + assert_eq!(duration(42), "42s"); + assert_eq!(duration(309), "5m09s"); + assert_eq!(duration(3 * 3600 + 12 * 60 + 5), "3h12m"); + assert_eq!(duration(2 * 86400 + 4 * 3600), "2d04h"); + assert_eq!(ago(10_000, 5_000), "5s ago"); + assert_eq!(ago(400_000, 0), "6m ago"); + } + + #[test] + fn sizes_are_compact() { + assert_eq!(mem_mb(512), "512M"); + assert_eq!(mem_mb(2048), "2G"); + assert_eq!(mem_mb(1536), "1.5G"); + assert_eq!(mem_mb(1024 * 1024 * 3 / 2), "1.5T"); + assert_eq!(disk_gb(40), "40G"); + assert_eq!(disk_gb(2048), "2T"); + } + + #[test] + fn sanitize_strips_cursor_movement() { + assert_eq!(sanitize("a\x1b[2Kb\tc\r"), "ab c"); + assert_eq!(sanitize("\x1b[1;32mok\x1b[0m done"), "ok done"); + assert_eq!(sanitize("\x1b]0;title\x07x\x1b]8;;u\x1b\\y"), "xy"); + } +} diff --git a/dstack/vmm-top/src/main.rs b/dstack/vmm-top/src/main.rs new file mode 100644 index 000000000..df8179b5a --- /dev/null +++ b/dstack/vmm-top/src/main.rs @@ -0,0 +1,192 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstack-vmm-top`: watch one or many dstack-vmm nodes from one terminal. +//! +//! ```text +//! dstack-vmm-top this host's VMMs (auto-discovered) +//! dstack-vmm-top -n tdx-01=http://10.0.0.11:9080 -n tdx-02=http://10.0.0.12:9080 +//! dstack-vmm-top --hub https://vmm-hub.example:9970 +//! dstack-vmm-top serve --listen 0.0.0.0:9970 run a hub for others to watch +//! ``` +//! +//! The TUI is only ever a client: it reads each node's VMM API directly, or +//! reads a hub (`serve`) that polls many nodes on its behalf. See `source.rs` +//! and `serve.rs`. + +mod collector; +mod config; +mod fmt; +mod once; +mod serve; +mod snapshot; +mod source; +mod tui; + +use anyhow::{bail, Context, Result}; +use clap::{Parser, Subcommand}; +use source::Source; +use std::io::IsTerminal; +use std::path::PathBuf; +use std::time::Duration; + +const DEFAULT_INTERVAL: Duration = Duration::from_secs(2); + +#[derive(Parser)] +#[command( + name = "dstack-vmm-top", + version, + about = "watch one or many dstack-vmm nodes from one terminal", + after_help = "With no --node, --hub or config file, watches the VMMs running on this host.\n\ + Config file: ~/.config/dstack/vmm-top.toml (see --help of `serve` for the hub)." +)] +struct Cli { + /// Config file listing nodes and hubs. Default: ~/.config/dstack/vmm-top.toml. + #[arg(long, short = 'c', value_name = "FILE")] + config: Option, + + /// A VMM to watch: `unix:/path/vmm.sock` or `http(s)://host:port`, optionally + /// named as `NAME=URL`. Repeatable. + #[arg(long = "node", short = 'n', value_name = "[NAME=]URL")] + nodes: Vec, + + /// A hub (`dstack-vmm-top serve`) to watch every node behind. Repeatable. + #[arg(long = "hub", value_name = "[NAME=]URL")] + hubs: Vec, + + /// Bearer token for VMMs that name none (`[auth]` enabled). + #[arg(long, env = "DSTACK_VMM_TOKEN", hide_env_values = true)] + token: Option, + + /// Bearer token for hubs that name none. + #[arg(long, env = "DSTACK_VMM_TOP_TOKEN", hide_env_values = true)] + hub_token: Option, + + /// Poll interval, e.g. `1s`, `5s`. `+` / `-` change it live. + #[arg(long, short = 'i', value_parser = humantime::parse_duration)] + interval: Option, + + /// Print one plain-text snapshot and exit. Implied when stdout is not a terminal. + #[arg(long)] + once: bool, + + /// Print one JSON snapshot and exit. + #[arg(long, conflicts_with = "once")] + json: bool, + + /// Never start, shut down or stop a VM from this terminal. + #[arg(long)] + read_only: bool, + + #[command(subcommand)] + command: Option, +} + +#[derive(Subcommand)] +enum Cmd { + /// Run a hub: poll the configured VMMs and serve them, read-only, to TUIs. + /// + /// Sources are chosen as for the TUI (config file, --node, discovery). + /// Watch it with `dstack-vmm-top --hub http://HOST:PORT`. + Serve { + /// `HOST:PORT` or `unix:/path`. Default: [serve] listen, else 127.0.0.1:9970. + #[arg(long)] + listen: Option, + /// Name the hub reports. Default: [serve] name, else the host name. + #[arg(long)] + name: Option, + /// Token clients must send. Falls back to [serve] token / token_env, + /// then DSTACK_VMM_TOP_TOKEN. + #[arg(long, hide_env_values = true)] + auth_token: Option, + /// Allow a non-loopback listen address without a token. + #[arg(long)] + insecure: bool, + }, +} + +#[tokio::main] +async fn main() -> Result<()> { + let cli = Cli::parse(); + let file = config::load(cli.config.as_deref())?; + let interval = match (cli.interval, &file.interval) { + (Some(d), _) => d, + (None, Some(s)) => humantime::parse_duration(s) + .with_context(|| format!("invalid interval '{s}' in config"))?, + (None, None) => DEFAULT_INTERVAL, + }; + let interval_ms = (interval.as_millis() as u64).max(200); + let read_only = cli.read_only || file.read_only; + let specs = config::sources( + &file, + config::Flags { + nodes: &cli.nodes, + hubs: &cli.hubs, + token: cli.token.as_deref(), + hub_token: cli.hub_token.as_deref(), + }, + )?; + let sources = specs + .into_iter() + .map(|s| Source::new(s, read_only)) + .collect::>>()?; + + if let Some(Cmd::Serve { + listen, + name, + auth_token, + insecure, + }) = cli.command + { + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| "info".into()), + ) + .with_writer(std::io::stderr) + .init(); + let token = match auth_token { + Some(t) => Some(t), + None => { + config::resolve_token(file.serve.token.as_deref(), file.serve.token_env.as_deref())? + .or_else(|| std::env::var(config::HUB_TOKEN_ENV).ok()) + .filter(|t| !t.trim().is_empty()) + } + }; + let opts = serve::Options { + name: name.or(file.serve.name.clone()).unwrap_or_else(hostname), + listen: listen + .or(file.serve.listen.clone()) + .unwrap_or_else(|| "127.0.0.1:9970".into()), + token, + interval_ms, + insecure, + }; + return serve::run(sources, opts).await; + } + + if sources.is_empty() { + bail!( + "no VMM to watch: none is registered on this host and no {} exists.\n\ + Give one with --node URL, a hub with --hub URL, or list them in {}", + config::DEFAULT_SOCKET, + config::default_path() + .map(|p| p.display().to_string()) + .unwrap_or_else(|| "a config file".into()) + ); + } + let interactive = std::io::stdout().is_terminal() && std::io::stdin().is_terminal(); + if cli.json || cli.once || !interactive { + return once::run(sources, cli.json).await; + } + tui::run(sources, interval_ms, read_only).await +} + +fn hostname() -> String { + fs_err::read_to_string("/proc/sys/kernel/hostname") + .ok() + .map(|h| h.trim().to_string()) + .filter(|h| !h.is_empty()) + .unwrap_or_else(|| "vmm-hub".into()) +} diff --git a/dstack/vmm-top/src/once.rs b/dstack/vmm-top/src/once.rs new file mode 100644 index 000000000..55cce264d --- /dev/null +++ b/dstack/vmm-top/src/once.rs @@ -0,0 +1,154 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! One snapshot to stdout, for pipes, scripts and `--once` / `--json`. + +use crate::fmt; +use crate::snapshot::{NodeSnapshot, Report}; +use crate::source::Source; +use anyhow::Result; +use std::fmt::Write as _; + +pub async fn run(sources: Vec, json: bool) -> Result<()> { + let now = fmt::now_ms(); + let polls = futures::future::join_all(sources.iter().map(|s| s.poll())).await; + let mut nodes: Vec = Vec::new(); + for (src, result) in sources.iter().zip(polls) { + match result { + Ok(ns) => nodes.extend(ns.into_iter().map(|mut n| { + if n.ok_at_ms.is_none() && n.error.is_none() { + n.ok_at_ms = Some(now); + n.polled_at_ms = now; + } + n + })), + Err(err) => nodes.push(NodeSnapshot { + name: src.spec.name.clone(), + endpoint: src.endpoint(), + polled_at_ms: now, + error: Some(format!("{err:#}")), + ..Default::default() + }), + } + } + if json { + let report = Report { + hub: String::new(), + version: env!("CARGO_PKG_VERSION").to_string(), + interval_ms: 0, + nodes, + }; + println!("{}", serde_json::to_string_pretty(&report)?); + return Ok(()); + } + print!("{}", render(&nodes)); + Ok(()) +} + +pub fn render(nodes: &[NodeSnapshot]) -> String { + let mut out = String::new(); + let _ = writeln!( + out, + "{:<20} {:<8} {:>7} {:>9} {:>11} {:>5} {:>6} ENDPOINT", + "NODE", "STATUS", "VMS", "VCPU", "MEM", "GPU", "RTT" + ); + for n in nodes { + let status = if n.error.is_some() { "down" } else { "online" }; + let res = n.resources.clone().unwrap_or_default(); + let known = n.ok_at_ms.is_some(); + let cell = |s: String| if known { s } else { "-".to_string() }; + let _ = writeln!( + out, + "{} {:<8} {:>7} {:>9} {:>11} {:>5} {:>6} {}", + fmt::pad(&n.name, 20), + status, + cell(format!("{}/{}", n.live_vms().count(), n.vms.len())), + cell(format!("{}/{}", n.vcpu_used(), res.max_vcpu)), + cell(format!( + "{}/{}", + fmt::mem_mb(n.memory_used_mb()), + fmt::mem_mb(res.max_memory_mb as u64) + )), + if n.gpus.is_empty() { + "-".to_string() + } else { + format!("{}/{}", n.gpus_used(), n.gpus.len()) + }, + n.latency_ms + .map(|l| format!("{l}ms")) + .unwrap_or_else(|| "-".into()), + n.endpoint, + ); + if let Some(e) = &n.error { + let _ = writeln!(out, " error: {e}"); + } + } + let vms: Vec<_> = nodes + .iter() + .flat_map(|n| n.vms.iter().map(move |v| (n, v))) + .collect(); + if vms.is_empty() { + return out; + } + let _ = writeln!(out); + let _ = writeln!( + out, + "{:<28} {:<16} {:<14} {:>7} {:>5} {:>6} {:>6} {:<8} IMAGE", + "VM", "NODE", "STATUS", "UPTIME", "VCPU", "MEM", "DISK", "ID" + ); + let mut vms = vms; + vms.sort_by(|(na, a), (nb, b)| { + a.phase() + .cmp(&b.phase()) + .then_with(|| na.name.cmp(&nb.name)) + .then_with(|| a.name.cmp(&b.name)) + }); + for (n, v) in vms { + let _ = writeln!( + out, + "{} {} {} {:>7} {:>5} {:>6} {:>6} {:<8} {}", + fmt::pad(&v.name, 28), + fmt::pad(&n.name, 16), + fmt::pad(v.phase().label(), 14), + v.uptime_secs + .map(fmt::duration) + .unwrap_or_else(|| "-".into()), + v.vcpu, + fmt::mem_mb(v.memory_mb as u64), + fmt::disk_gb(v.disk_gb as u64), + v.short_id(), + v.image, + ); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::snapshot::Vm; + + #[test] + fn plain_text_lists_nodes_then_vms() { + let nodes = vec![NodeSnapshot { + name: "tdx-01".into(), + endpoint: "http://h:1".into(), + ok_at_ms: Some(1), + vms: vec![Vm { + id: "abcdef0123".into(), + name: "web".into(), + status: "running".into(), + vcpu: 2, + memory_mb: 2048, + ..Default::default() + }], + ..Default::default() + }]; + let text = render(&nodes); + assert!(text.lines().nth(1).unwrap().starts_with("tdx-01")); + assert!(text.contains("web")); + assert!(text.contains("abcdef01")); + assert!(text.contains("2G")); + } +} diff --git a/dstack/vmm-top/src/serve.rs b/dstack/vmm-top/src/serve.rs new file mode 100644 index 000000000..f2bcb9d99 --- /dev/null +++ b/dstack/vmm-top/src/serve.rs @@ -0,0 +1,346 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! `dstack-vmm-top serve`: a hub that polls many VMMs and serves what it sees +//! to any number of TUIs. +//! +//! Why a hub and not just more `--node` flags: a VMM usually listens on a +//! unix socket only, its token is per node, and every watcher polling every +//! node multiplies load. A hub next to the VMMs reaches their sockets, holds +//! their tokens, polls each once, and hands out one URL with one token of its +//! own. +//! +//! It is read-only by construction: it serves snapshots and log tails, and +//! has no route that changes a VM. Handing someone the hub token lets them +//! watch, never stop. +//! +//! Routes (all `GET`): +//! - `/healthz`: `ok`, no auth. +//! - `/api/v1/nodes`: a [`Report`]. +//! - `/api/v1/logs?node=&id=&ch=serial|stdout|stderr&lines=`: a log tail. + +use crate::collector::{self, Update}; +use crate::snapshot::{NodeSnapshot, Report}; +use crate::source::{LogChannel, Source}; +use anyhow::{bail, Context, Result}; +use fs_err as fs; +use http_body_util::Full; +use hyper::body::{Bytes, Incoming}; +use hyper::header::{HeaderValue, CACHE_CONTROL, CONTENT_TYPE}; +use hyper::server::conn::http1; +use hyper::service::service_fn; +use hyper::{Request, Response, StatusCode}; +use hyper_util::rt::TokioIo; +use std::collections::{HashMap, HashSet}; +use std::net::SocketAddr; +use std::sync::{Arc, RwLock}; +use tracing::{info, warn}; + +const MAX_LOG_LINES: u32 = 20_000; + +pub struct Options { + pub name: String, + pub listen: String, + pub token: Option, + pub interval_ms: u64, + /// Serve without a token on a non-loopback address. + pub insecure: bool, +} + +struct Hub { + name: String, + interval_ms: u64, + token: Option, + sources: Vec>, + latest: RwLock>>, +} + +pub async fn run(sources: Vec, opts: Options) -> Result<()> { + if sources.is_empty() { + bail!("nothing to serve: give --node, a config file, or run next to a VMM"); + } + let listen = Listen::parse(&opts.listen)?; + if opts.token.is_none() && !listen.is_local() && !opts.insecure { + bail!( + "refusing to serve {} without a token: set --auth-token (or DSTACK_VMM_TOP_TOKEN), \ + listen on loopback or a unix socket, or pass --insecure", + opts.listen + ); + } + let sources: Vec> = sources.into_iter().map(Arc::new).collect(); + for s in &sources { + info!("watching {} at {}", s.spec.name, s.endpoint()); + } + let hub = Arc::new(Hub { + name: opts.name, + interval_ms: opts.interval_ms, + token: opts.token, + latest: RwLock::new(vec![Vec::new(); sources.len()]), + sources: sources.clone(), + }); + let mut handle = collector::spawn(sources, opts.interval_ms); + let h = hub.clone(); + tokio::spawn(async move { + while let Some(up) = handle.updates.recv().await { + if let Update::Nodes { source, nodes } = up { + for n in &nodes { + if let Some(e) = &n.error { + warn!("{}: {e}", n.name); + } + } + if let Ok(mut latest) = h.latest.write() { + latest[source] = nodes; + } + } + } + }); + + match listen { + Listen::Tcp(addr) => { + let listener = tokio::net::TcpListener::bind(addr) + .await + .with_context(|| format!("failed to listen on {addr}"))?; + info!("serving on http://{addr}"); + loop { + let (stream, _) = listener.accept().await?; + spawn_conn(TokioIo::new(stream), hub.clone()); + } + } + Listen::Unix(path) => { + let _ = fs::remove_file(&path); + let listener = tokio::net::UnixListener::bind(&path) + .with_context(|| format!("failed to listen on {path}"))?; + info!("serving on unix:{path}"); + loop { + let (stream, _) = listener.accept().await?; + spawn_conn(TokioIo::new(stream), hub.clone()); + } + } + } +} + +fn spawn_conn(io: I, hub: Arc) +where + I: hyper::rt::Read + hyper::rt::Write + Unpin + Send + 'static, +{ + tokio::spawn(async move { + let svc = service_fn(move |req| { + let hub = hub.clone(); + async move { Ok::<_, std::convert::Infallible>(route(&hub, req).await) } + }); + if let Err(err) = http1::Builder::new().serve_connection(io, svc).await { + tracing::debug!("connection ended: {err}"); + } + }); +} + +enum Listen { + Tcp(SocketAddr), + Unix(String), +} + +impl Listen { + fn parse(s: &str) -> Result { + if let Some(path) = s.strip_prefix("unix:") { + return Ok(Listen::Unix(path.to_string())); + } + Ok(Listen::Tcp(s.parse().with_context(|| { + format!("invalid listen address '{s}': expected HOST:PORT or unix:PATH") + })?)) + } + + fn is_local(&self) -> bool { + match self { + Listen::Tcp(a) => a.ip().is_loopback(), + Listen::Unix(_) => true, + } + } +} + +fn reply( + status: StatusCode, + content_type: &'static str, + body: impl Into, +) -> Response> { + let mut resp = Response::new(Full::new(body.into())); + *resp.status_mut() = status; + let headers = resp.headers_mut(); + headers.insert(CONTENT_TYPE, HeaderValue::from_static(content_type)); + headers.insert(CACHE_CONTROL, HeaderValue::from_static("no-store")); + resp +} + +fn text(status: StatusCode, body: impl Into) -> Response> { + reply(status, "text/plain; charset=utf-8", body.into()) +} + +/// Equal without leaking where the first difference is. +fn ct_eq(a: &[u8], b: &[u8]) -> bool { + if a.len() != b.len() { + return false; + } + a.iter().zip(b).fold(0u8, |acc, (x, y)| acc | (x ^ y)) == 0 +} + +fn authorized(hub: &Hub, req: &Request) -> bool { + let Some(token) = &hub.token else { + return true; + }; + let Some(header) = req + .headers() + .get(hyper::header::AUTHORIZATION) + .and_then(|v| v.to_str().ok()) + else { + return false; + }; + match header.strip_prefix("Bearer ") { + Some(given) => ct_eq(given.trim().as_bytes(), token.as_bytes()), + None => false, + } +} + +async fn route(hub: &Hub, req: Request) -> Response> { + if req.method() != hyper::Method::GET { + return text(StatusCode::METHOD_NOT_ALLOWED, "read-only hub: GET only"); + } + let path = req.uri().path().to_string(); + if path == "/healthz" { + return text(StatusCode::OK, "ok"); + } + if !authorized(hub, &req) { + return text(StatusCode::UNAUTHORIZED, "missing or wrong bearer token"); + } + let query = parse_query(req.uri().query().unwrap_or("")); + match path.as_str() { + "/api/v1/nodes" => { + let report = Report { + hub: hub.name.clone(), + version: env!("CARGO_PKG_VERSION").to_string(), + interval_ms: hub.interval_ms, + nodes: served(hub).into_iter().map(|(n, _, _)| n).collect(), + }; + match serde_json::to_vec(&report) { + Ok(body) => reply(StatusCode::OK, "application/json", body), + Err(err) => text(StatusCode::INTERNAL_SERVER_ERROR, err.to_string()), + } + } + "/api/v1/logs" => logs(hub, &query).await, + _ => text(StatusCode::NOT_FOUND, "not found"), + } +} + +/// Every node with the name it is served under, its source, and the name the +/// source knows it by. Names stay as the source gave them unless two sources +/// collide, which happens when hubs are chained. +fn served(hub: &Hub) -> Vec<(NodeSnapshot, usize, String)> { + let latest = hub.latest.read().map(|l| l.clone()).unwrap_or_default(); + let mut seen = HashSet::new(); + let mut out = Vec::new(); + for (si, nodes) in latest.into_iter().enumerate() { + for mut n in nodes { + let orig = n.name.clone(); + if !seen.insert(n.name.clone()) { + n.name = format!("{}/{orig}", hub.sources[si].spec.name); + seen.insert(n.name.clone()); + } + out.push((n, si, orig)); + } + } + out +} + +async fn logs(hub: &Hub, q: &HashMap) -> Response> { + let (Some(node), Some(id)) = (q.get("node"), q.get("id")) else { + return text(StatusCode::BAD_REQUEST, "node and id are required"); + }; + let c = q.get("ch").map(String::as_str).unwrap_or("serial"); + let Some(ch) = LogChannel::parse(c) else { + return text(StatusCode::BAD_REQUEST, format!("unknown channel {c}")); + }; + let lines = q + .get("lines") + .and_then(|l| l.parse::().ok()) + .unwrap_or(1000) + .min(MAX_LOG_LINES); + let Some((snap, si, orig)) = served(hub).into_iter().find(|(n, _, _)| &n.name == node) else { + return text(StatusCode::NOT_FOUND, format!("no node {node}")); + }; + // Only VMs the node listed: an id is never passed through unchecked. + if !snap.vms.iter().any(|v| &v.id == id) { + return text(StatusCode::NOT_FOUND, format!("no VM {id} on {node}")); + } + match hub.sources[si].logs(&orig, id, ch, lines).await { + Ok(body) => text(StatusCode::OK, body), + Err(err) => text(StatusCode::BAD_GATEWAY, format!("{err:#}")), + } +} + +fn parse_query(q: &str) -> HashMap { + q.split('&') + .filter(|kv| !kv.is_empty()) + .map(|kv| { + let (k, v) = kv.split_once('=').unwrap_or((kv, "")); + (percent_decode(k), percent_decode(v)) + }) + .collect() +} + +fn percent_decode(s: &str) -> String { + let bytes = s.as_bytes(); + let mut out = Vec::with_capacity(bytes.len()); + let mut i = 0; + while i < bytes.len() { + match bytes[i] { + b'%' if i + 2 < bytes.len() => { + let hex = std::str::from_utf8(&bytes[i + 1..i + 3]).ok(); + match hex.and_then(|h| u8::from_str_radix(h, 16).ok()) { + Some(b) => { + out.push(b); + i += 3; + continue; + } + None => out.push(b'%'), + } + } + b'+' => out.push(b' '), + b => out.push(b), + } + i += 1; + } + String::from_utf8_lossy(&out).into_owned() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::source::query_escape; + + #[test] + fn query_round_trips() { + let q = parse_query(&format!( + "node={}&id={}&lines=5", + query_escape("dc/tdx 01"), + query_escape("a&b") + )); + assert_eq!(q["node"], "dc/tdx 01"); + assert_eq!(q["id"], "a&b"); + assert_eq!(q["lines"], "5"); + assert_eq!(percent_decode("%zz%4"), "%zz%4"); + } + + #[test] + fn token_compare() { + assert!(ct_eq(b"abc", b"abc")); + assert!(!ct_eq(b"abc", b"abd")); + assert!(!ct_eq(b"abc", b"ab")); + } + + #[test] + fn listen_locality() { + assert!(Listen::parse("127.0.0.1:9970").unwrap().is_local()); + assert!(Listen::parse("unix:/tmp/x.sock").unwrap().is_local()); + assert!(!Listen::parse("0.0.0.0:9970").unwrap().is_local()); + assert!(Listen::parse("nope").is_err()); + } +} diff --git a/dstack/vmm-top/src/snapshot.rs b/dstack/vmm-top/src/snapshot.rs new file mode 100644 index 000000000..230244a6c --- /dev/null +++ b/dstack/vmm-top/src/snapshot.rs @@ -0,0 +1,413 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! What one poll of one node returns, in the shape both the TUI and the hub +//! speak. +//! +//! The hub serves these as JSON, so this is a wire format: fields are only +//! added, each new one with `#[serde(default)]`, so a newer TUI reads an older +//! hub and the other way round. It deliberately carries no secrets: no +//! encrypted env, no user config, no compose file. + +use dstack_vmm_rpc as rpc; +use serde::{Deserialize, Serialize}; +use std::time::Duration; + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)] +pub struct NodeSnapshot { + /// Display name, unique within one source. + pub name: String, + /// Where the node is reached, without credentials. + pub endpoint: String, + /// The hub this node was read through, when not polled directly. Set by + /// the TUI, not by the hub. + #[serde(default)] + pub via: Option, + /// Unix ms of the latest poll attempt. + pub polled_at_ms: u64, + /// Unix ms of the latest successful poll. Data below is from then. + #[serde(default)] + pub ok_at_ms: Option, + #[serde(default)] + pub latency_ms: Option, + /// Why the latest poll failed. The data below is then the last good copy. + #[serde(default)] + pub error: Option, + #[serde(default)] + pub version: Option, + #[serde(default)] + pub rev: Option, + #[serde(default)] + pub resources: Option, + #[serde(default)] + pub kms_urls: Vec, + #[serde(default)] + pub gateway_urls: Vec, + #[serde(default)] + pub base_domain: Option, + #[serde(default)] + pub vms: Vec, + #[serde(default)] + pub gpus: Vec, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +pub struct Resources { + pub max_cvms: u32, + pub max_vcpu: u32, + pub max_memory_mb: u32, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)] +pub struct Vm { + pub id: String, + pub name: String, + /// VMM runtime status: running, exited, stopping, stopped, removing. + pub status: String, + #[serde(default)] + pub running: bool, + /// Seconds since QEMU started, when it has. + #[serde(default)] + pub uptime_secs: Option, + #[serde(default)] + pub app_id: String, + #[serde(default)] + pub instance_id: Option, + #[serde(default)] + pub app_url: Option, + #[serde(default)] + pub image: String, + #[serde(default)] + pub image_version: String, + #[serde(default)] + pub vcpu: u32, + #[serde(default)] + pub memory_mb: u32, + #[serde(default)] + pub disk_gb: u32, + #[serde(default)] + pub gpus: Vec, + #[serde(default)] + pub ports: Vec, + #[serde(default)] + pub networks: Vec, + #[serde(default)] + pub no_tee: bool, + #[serde(default)] + pub boot_progress: String, + #[serde(default)] + pub boot_error: String, + #[serde(default)] + pub shutdown_progress: String, + #[serde(default)] + pub events: Vec, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +pub struct Port { + pub protocol: String, + pub host_address: String, + pub host_port: u32, + pub vm_port: u32, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +pub struct Gpu { + pub slot: String, + pub product_id: String, + pub description: String, + pub is_free: bool, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +pub struct GuestEvent { + pub event: String, + pub body: String, + pub timestamp_ms: u64, +} + +/// Every node at once: what `--json` prints and a hub serves on +/// `GET /api/v1/nodes`. +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +pub struct Report { + /// Name of the hub that served this; empty from `--json`. + #[serde(default)] + pub hub: String, + pub version: String, + /// The hub's poll interval; 0 from `--json`. + #[serde(default)] + pub interval_ms: u64, + pub nodes: Vec, +} + +/// Where a VM is in its life, as an operator reads it. Folds `status`, +/// `boot_progress`, `boot_error` and `shutdown_progress` into one value, the +/// same way the web console does. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)] +pub enum Phase { + // Declaration order is the default sort: trouble first, idle last. + Failed, + Exited, + Booting, + ShuttingDown, + Stopping, + Removing, + Running, + Stopped, + Unknown, +} + +impl Phase { + pub fn label(self) -> &'static str { + match self { + Phase::Failed => "failed", + Phase::Booting => "booting", + Phase::ShuttingDown => "shutting down", + Phase::Stopping => "stopping", + Phase::Removing => "removing", + Phase::Running => "running", + Phase::Exited => "exited", + Phase::Stopped => "stopped", + Phase::Unknown => "unknown", + } + } + + /// Changing on its own right now; the UI animates these. + pub fn is_transient(self) -> bool { + matches!( + self, + Phase::Booting | Phase::ShuttingDown | Phase::Stopping | Phase::Removing + ) + } + + /// Holds host resources. + pub fn is_live(self) -> bool { + matches!( + self, + Phase::Running | Phase::Booting | Phase::ShuttingDown | Phase::Stopping | Phase::Failed + ) + } + + /// Worth an operator's look. + pub fn needs_attention(self) -> bool { + matches!(self, Phase::Failed | Phase::Exited) || self.is_transient() + } +} + +fn unset(s: &str) -> bool { + s.is_empty() || s == "N/A" +} + +impl Vm { + pub fn phase(&self) -> Phase { + match self.status.as_str() { + "running" => { + if !self.boot_error.is_empty() { + Phase::Failed + } else if !unset(&self.shutdown_progress) { + Phase::ShuttingDown + } else if unset(&self.boot_progress) + || self.boot_progress == "done" + || self.boot_progress == "running" + { + Phase::Running + } else { + Phase::Booting + } + } + "exited" => Phase::Exited, + "stopping" => Phase::Stopping, + "stopped" => Phase::Stopped, + "removing" => Phase::Removing, + _ => Phase::Unknown, + } + } + + /// One short phrase on what the VM is doing beyond its phase, if any. + pub fn activity(&self) -> Option<&str> { + match self.phase() { + Phase::Failed => Some(&self.boot_error), + Phase::Booting if self.boot_progress != "booting" => Some(&self.boot_progress), + Phase::ShuttingDown => Some(&self.shutdown_progress), + _ => None, + } + } + + pub fn short_id(&self) -> &str { + short(&self.id, 8) + } +} + +pub fn short(s: &str, n: usize) -> &str { + match s.char_indices().nth(n) { + Some((i, _)) => &s[..i], + None => s, + } +} + +impl NodeSnapshot { + pub fn online(&self) -> bool { + self.error.is_none() && self.ok_at_ms.is_some() + } + + /// vCPUs held by VMs that hold host resources. + pub fn vcpu_used(&self) -> u32 { + self.live_vms().map(|vm| vm.vcpu).sum() + } + + pub fn memory_used_mb(&self) -> u64 { + self.live_vms().map(|vm| vm.memory_mb as u64).sum() + } + + pub fn disk_gb(&self) -> u64 { + self.vms.iter().map(|vm| vm.disk_gb as u64).sum() + } + + pub fn gpus_used(&self) -> usize { + self.gpus.iter().filter(|g| !g.is_free).count() + } + + pub fn live_vms(&self) -> impl Iterator { + self.vms.iter().filter(|vm| vm.phase().is_live()) + } +} + +/// Seconds out of the VMM's `humantime` uptime ("2days 3h 4m 5s"), or +/// `None` for "never" and anything unparseable. +pub fn parse_uptime(s: &str) -> Option { + if s.is_empty() || s == "never" { + return None; + } + humantime::parse_duration(s) + .ok() + .map(|d: Duration| d.as_secs()) +} + +impl From for Vm { + fn from(info: rpc::VmInfo) -> Self { + let cfg = info.configuration.unwrap_or_default(); + let uptime_secs = if info.running { + parse_uptime(&info.uptime) + } else { + None + }; + let networks = if info.interfaces.is_empty() { + cfg.networks.iter().map(|n| n.mode.clone()).collect() + } else { + info.interfaces + .iter() + .map(|i| match &i.bridge_name { + Some(b) if !b.is_empty() => format!("{}:{b}", i.mode), + _ => i.mode.clone(), + }) + .collect() + }; + Vm { + id: info.id, + name: info.name, + status: info.status, + running: info.running, + uptime_secs, + app_id: info.app_id, + instance_id: info.instance_id.filter(|s| !s.is_empty()), + app_url: info.app_url.filter(|s| !s.is_empty()), + image: cfg.image, + image_version: info.image_version, + vcpu: cfg.vcpu, + memory_mb: cfg.memory, + disk_gb: cfg.disk_size, + gpus: cfg + .gpus + .map(|g| g.gpus.into_iter().map(|s| s.slot).collect()) + .unwrap_or_default(), + ports: cfg + .ports + .into_iter() + .map(|p| Port { + protocol: p.protocol, + host_address: p.host_address, + host_port: p.host_port, + vm_port: p.vm_port, + }) + .collect(), + networks, + no_tee: cfg.no_tee, + boot_progress: info.boot_progress, + boot_error: info.boot_error, + shutdown_progress: info.shutdown_progress, + events: info + .events + .into_iter() + .map(|e| GuestEvent { + event: e.event, + body: e.body, + timestamp_ms: e.timestamp, + }) + .collect(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn vm(status: &str, boot: &str, err: &str, shutdown: &str) -> Vm { + Vm { + status: status.into(), + boot_progress: boot.into(), + boot_error: err.into(), + shutdown_progress: shutdown.into(), + ..Default::default() + } + } + + #[test] + fn phase_folds_progress_into_status() { + assert_eq!(vm("running", "done", "", "").phase(), Phase::Running); + assert_eq!(vm("running", "running", "", "").phase(), Phase::Running); + assert_eq!(vm("running", "N/A", "", "N/A").phase(), Phase::Running); + assert_eq!(vm("running", "booting", "", "").phase(), Phase::Booting); + assert_eq!( + vm("running", "requesting app keys", "", "").phase(), + Phase::Booting + ); + assert_eq!( + vm("running", "done", "", "powering off").phase(), + Phase::ShuttingDown + ); + assert_eq!( + vm("running", "mounting data disk", "kms refused", "").phase(), + Phase::Failed + ); + assert_eq!(vm("exited", "", "", "").phase(), Phase::Exited); + assert_eq!(vm("stopped", "", "", "").phase(), Phase::Stopped); + assert_eq!(vm("what", "", "", "").phase(), Phase::Unknown); + } + + #[test] + fn activity_names_the_step() { + assert_eq!(vm("running", "booting", "", "").activity(), None); + assert_eq!( + vm("running", "requesting app keys", "", "").activity(), + Some("requesting app keys") + ); + assert_eq!(vm("running", "done", "", "").activity(), None); + } + + #[test] + fn uptime_parses_vmm_format() { + assert_eq!(parse_uptime("never"), None); + assert_eq!(parse_uptime("5s"), Some(5)); + assert_eq!(parse_uptime("1h 2m 3s"), Some(3723)); + assert_eq!(parse_uptime("2days 1h"), Some(2 * 86400 + 3600)); + } + + #[test] + fn short_is_char_safe() { + assert_eq!(short("abcdef", 3), "abc"); + assert_eq!(short("ab", 3), "ab"); + assert_eq!(short("名字很长", 2), "名字"); + } +} diff --git a/dstack/vmm-top/src/source.rs b/dstack/vmm-top/src/source.rs new file mode 100644 index 000000000..21fd8e94e --- /dev/null +++ b/dstack/vmm-top/src/source.rs @@ -0,0 +1,376 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Where snapshots come from: a VMM polled directly, or a hub that polls many. +//! +//! Both speak HTTP (a VMM also over its unix socket), so a source is a base +//! address plus an optional bearer token. A VMM is read through its pRPC API +//! and `/logs`; a hub through `/api/v1/nodes` and `/api/v1/logs`. Only a VMM +//! polled directly takes lifecycle actions: a hub is read-only, so handing +//! someone its URL never hands them a stop button. + +use crate::snapshot::{Gpu, NodeSnapshot, Report, Resources, Vm}; +use anyhow::{anyhow, bail, Context, Result}; +use dstack_vmm_rpc::vmm_client::VmmClient; +use dstack_vmm_rpc::{Id, StatusRequest}; +use http_client::http_request_with_headers; +use http_client::prpc::PrpcClient; +use serde::{Deserialize, Serialize}; +use std::time::{Duration, Instant}; + +/// Bound on one poll or one request, so a hung node shows as an error instead +/// of freezing its row. +pub const REQUEST_TIMEOUT: Duration = Duration::from_secs(8); + +/// Bound on one lifecycle action. The VMM answers a stop only once QEMU has +/// exited (up to 15s before it kills), a shutdown once the guest agent has +/// (up to 30s), and a start after waiting out a VM still stopping (up to 60s), +/// so the poll bound would report slow successes as failures. +pub const ACTION_TIMEOUT: Duration = Duration::from_secs(120); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum SourceKind { + Vmm, + Hub, +} + +#[derive(Clone, Debug)] +pub struct SourceSpec { + pub name: String, + pub kind: SourceKind, + /// `unix:/path/to/vmm.sock` or `http(s)://host:port`. + pub url: String, + pub token: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum LogChannel { + Serial, + Stdout, + Stderr, +} + +impl LogChannel { + pub fn as_str(self) -> &'static str { + match self { + LogChannel::Serial => "serial", + LogChannel::Stdout => "stdout", + LogChannel::Stderr => "stderr", + } + } + + pub fn parse(s: &str) -> Option { + match s { + "serial" => Some(LogChannel::Serial), + "stdout" => Some(LogChannel::Stdout), + "stderr" => Some(LogChannel::Stderr), + _ => None, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Action { + Start, + /// Ask the guest to power off. + Shutdown, + /// Kill QEMU. + Stop, + /// Kill QEMU, then start again. The VMM has no restart call; its start + /// waits for a VM told to stop to finish exiting, so the two in sequence + /// are safe. + Restart, +} + +impl Action { + pub fn label(self) -> &'static str { + match self { + Action::Start => "start", + Action::Shutdown => "shutdown", + Action::Stop => "force stop", + Action::Restart => "restart", + } + } + + /// Kills QEMU without asking the guest. + pub fn is_forceful(self) -> bool { + matches!(self, Action::Stop | Action::Restart) + } +} + +pub struct Source { + pub spec: SourceSpec, + /// Base for plain-HTTP requests: `unix:/path` or `http(s)://host:port`. + base: String, + rpc: Option>, + /// Same endpoint with the longer [`ACTION_TIMEOUT`]. + act_rpc: Option>, + read_only: bool, +} + +impl Source { + pub fn new(spec: SourceSpec, read_only: bool) -> Result { + let url = spec.url.trim(); + let base = if let Some(sock) = url.strip_prefix("unix:") { + format!("unix:{sock}") + } else if url.starts_with("http://") || url.starts_with("https://") { + url.trim_end_matches('/').to_string() + } else { + bail!("unsupported address '{url}': expected unix:/path/to/vmm.sock or http(s)://host:port"); + }; + let vmm_client = |timeout: Duration| { + let client = match base.strip_prefix("unix:") { + Some(sock) => PrpcClient::new_unix(sock.to_string(), "/prpc".to_string()), + None => PrpcClient::new(format!("{base}/prpc")), + }; + let client = client.with_request_timeout(timeout); + let client = match &spec.token { + Some(token) => client.with_bearer_token(token.clone()), + None => client, + }; + VmmClient::new(client) + }; + let (rpc, act_rpc) = match spec.kind { + SourceKind::Hub => (None, None), + SourceKind::Vmm => ( + Some(vmm_client(REQUEST_TIMEOUT)), + Some(vmm_client(ACTION_TIMEOUT)), + ), + }; + Ok(Self { + spec, + base, + rpc, + act_rpc, + read_only, + }) + } + + /// Whether lifecycle actions may go through this source. + pub fn writable(&self) -> bool { + self.rpc.is_some() && !self.read_only + } + + /// The address shown on screen: never carries credentials. + pub fn endpoint(&self) -> String { + display_endpoint(&self.base) + } + + /// Poll every node behind this source once. + pub async fn poll(&self) -> Result> { + tokio::time::timeout(REQUEST_TIMEOUT, async { + match &self.rpc { + Some(rpc) => Ok(vec![self.poll_vmm(rpc).await?]), + None => self.poll_hub().await, + } + }) + .await + .map_err(|_| anyhow!("timed out after {}s", REQUEST_TIMEOUT.as_secs()))? + } + + async fn poll_vmm(&self, rpc: &VmmClient) -> Result { + let started = Instant::now(); + let status = rpc + .status(StatusRequest::default()) + .await + .map_err(|err| anyhow!("{err}"))?; + let latency = started.elapsed(); + // Everything past the VM list is context: a VMM too old to answer + // one of these still shows its VMs. + let (version, meta, gpus) = tokio::join!(rpc.version(), rpc.get_meta(), rpc.list_gpus()); + let mut node = NodeSnapshot { + name: self.spec.name.clone(), + endpoint: self.endpoint(), + latency_ms: Some(latency.as_millis() as u64), + vms: status.vms.into_iter().map(Vm::from).collect(), + ..Default::default() + }; + if let Ok(v) = version { + node.version = Some(v.version); + node.rev = Some(v.rev).filter(|r| !r.is_empty()); + } + if let Ok(meta) = meta { + node.resources = meta.resources.map(|r| Resources { + max_cvms: r.max_cvm_number, + max_vcpu: r.max_allocable_vcpu, + max_memory_mb: r.max_allocable_memory_in_mb, + }); + if let Some(kms) = meta.kms { + node.kms_urls = if kms.urls.is_empty() && !kms.url.is_empty() { + vec![kms.url] + } else { + kms.urls + }; + } + if let Some(gw) = meta.gateway { + node.gateway_urls = if gw.urls.is_empty() && !gw.url.is_empty() { + vec![gw.url] + } else { + gw.urls + }; + node.base_domain = Some(gw.base_domain).filter(|d| !d.is_empty()); + } + } + if let Ok(gpus) = gpus { + node.gpus = gpus + .gpus + .into_iter() + .map(|g| Gpu { + slot: g.slot, + product_id: g.product_id, + description: g.description, + is_free: g.is_free, + }) + .collect(); + } + Ok(node) + } + + async fn poll_hub(&self) -> Result> { + let body = self.get("/api/v1/nodes").await?; + let report: Report = + serde_json::from_slice(&body).context("hub answered something that is not a report")?; + Ok(report + .nodes + .into_iter() + .map(|mut n| { + n.via = Some(self.spec.name.clone()); + n + }) + .collect()) + } + + /// The last `lines` lines of one VM log channel. + pub async fn logs( + &self, + node: &str, + vm_id: &str, + ch: LogChannel, + lines: u32, + ) -> Result { + let path = match self.spec.kind { + SourceKind::Vmm => format!( + "/logs?id={}&follow=false&ansi=false&lines={lines}&ch={}", + query_escape(vm_id), + ch.as_str() + ), + SourceKind::Hub => format!( + "/api/v1/logs?node={}&id={}&lines={lines}&ch={}", + query_escape(node), + query_escape(vm_id), + ch.as_str() + ), + }; + let body = self.get(&path).await?; + Ok(String::from_utf8_lossy(&body).into_owned()) + } + + pub async fn act(&self, vm_id: &str, action: Action) -> Result<()> { + let Some(rpc) = self.act_rpc.as_ref().filter(|_| !self.read_only) else { + bail!("{} is read-only", self.spec.name); + }; + let id = Id { + id: vm_id.to_string(), + }; + let result = match action { + Action::Start => rpc.start_vm(id).await, + Action::Shutdown => rpc.shutdown_vm(id).await, + Action::Stop => rpc.stop_vm(id).await, + Action::Restart => { + rpc.stop_vm(id.clone()) + .await + .map_err(|err| anyhow!("stop failed: {err}"))?; + return rpc + .start_vm(id) + .await + .map_err(|err| anyhow!("stopped, but start failed: {err}")); + } + }; + result.map_err(|err| anyhow!("{err}"))?; + Ok(()) + } + + async fn get(&self, path: &str) -> Result> { + let auth = self.spec.token.as_ref().map(|t| format!("Bearer {t}")); + let mut headers: Vec<(&str, &str)> = Vec::new(); + if let Some(auth) = &auth { + headers.push(("Authorization", auth)); + } + let (status, body) = tokio::time::timeout( + REQUEST_TIMEOUT, + http_request_with_headers("GET", &self.base, path, b"", &headers), + ) + .await + .map_err(|_| anyhow!("timed out after {}s", REQUEST_TIMEOUT.as_secs()))??; + if status != 200 { + let text = String::from_utf8_lossy(&body); + let text = text.trim(); + if text.is_empty() { + bail!("HTTP {status}"); + } + bail!("HTTP {status}: {}", crate::fmt::truncate(text, 160)); + } + Ok(body) + } +} + +/// `base` with any `user:pass@` removed. +pub fn display_endpoint(base: &str) -> String { + if let Some((scheme, rest)) = base.split_once("://") { + let (authority, path) = match rest.find('/') { + Some(i) => rest.split_at(i), + None => (rest, ""), + }; + let host = authority.rsplit_once('@').map_or(authority, |(_, h)| h); + format!("{scheme}://{host}{path}") + } else { + base.to_string() + } +} + +/// Percent-encode everything outside RFC 3986 unreserved characters. +pub fn query_escape(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + for b in s.bytes() { + if b.is_ascii_alphanumeric() || matches!(b, b'-' | b'_' | b'.' | b'~') { + out.push(b as char); + } else { + out.push_str(&format!("%{b:02X}")); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn endpoint_hides_credentials() { + assert_eq!( + display_endpoint("https://u:p@host:9080/x"), + "https://host:9080/x" + ); + assert_eq!(display_endpoint("http://host:9080"), "http://host:9080"); + assert_eq!(display_endpoint("unix:/run/vmm.sock"), "unix:/run/vmm.sock"); + } + + #[test] + fn query_escape_is_strict() { + assert_eq!(query_escape("a b/c&d"), "a%20b%2Fc%26d"); + assert_eq!(query_escape("tdx-01_x.y~"), "tdx-01_x.y~"); + } + + #[test] + fn rejects_unknown_scheme() { + let spec = SourceSpec { + name: "x".into(), + kind: SourceKind::Vmm, + url: "ftp://x".into(), + token: None, + }; + assert!(Source::new(spec, false).is_err()); + } +} diff --git a/dstack/vmm-top/src/tui/mod.rs b/dstack/vmm-top/src/tui/mod.rs new file mode 100644 index 000000000..6772101f6 --- /dev/null +++ b/dstack/vmm-top/src/tui/mod.rs @@ -0,0 +1,449 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Interactive dashboard: terminal setup, the event loop, and key and mouse +//! handling. State lives in [`model`], drawing in [`ui`]. + +pub mod model; +pub mod ui; + +use crate::collector::{self, Command, LogRequest, Update}; +use crate::fmt::now_ms; +use crate::source::{Action, Source}; +use anyhow::Result; +use crossterm::event::{ + self, DisableMouseCapture, EnableMouseCapture, Event, KeyCode, KeyEvent, KeyEventKind, + KeyModifiers, MouseButton, MouseEvent, MouseEventKind, +}; +use crossterm::execute; +use crossterm::terminal::{ + disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen, +}; +use model::{DetailTab, Effect, Input, RowKey, State, Tab}; +use ratatui::backend::CrosstermBackend; +use ratatui::Terminal; +use std::io::{self, Stdout}; +use std::sync::atomic::Ordering; +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tokio::sync::mpsc; + +/// Restores the terminal on drop and on panic, so a crash never leaves the +/// shell in raw mode on the alternate screen. +struct TerminalGuard; + +impl TerminalGuard { + fn enter() -> Result { + enable_raw_mode()?; + execute!(io::stdout(), EnterAlternateScreen, EnableMouseCapture)?; + let hook = std::panic::take_hook(); + std::panic::set_hook(Box::new(move |info| { + restore(); + hook(info); + })); + Ok(Self) + } +} + +fn restore() { + let _ = disable_raw_mode(); + let _ = execute!(io::stdout(), DisableMouseCapture, LeaveAlternateScreen); +} + +impl Drop for TerminalGuard { + fn drop(&mut self) { + restore(); + } +} + +pub async fn run(sources: Vec, interval_ms: u64, read_only: bool) -> Result<()> { + let names = sources.iter().map(|s| s.spec.name.clone()).collect(); + let writable: Vec = sources.iter().map(|s| s.writable()).collect(); + let sources: Vec> = sources.into_iter().map(Arc::new).collect(); + let mut handle = collector::spawn(sources, interval_ms); + let mut state = State::new(names, interval_ms, read_only); + state.tick(now_ms()); + + let _guard = TerminalGuard::enter()?; + let mut term: Terminal> = + Terminal::new(CrosstermBackend::new(io::stdout()))?; + term.clear()?; + + // crossterm's reader blocks, so it gets a thread of its own. + let (ev_tx, mut ev_rx) = mpsc::unbounded_channel::(); + std::thread::spawn(move || loop { + match event::read() { + Ok(ev) => { + if ev_tx.send(ev).is_err() { + return; + } + } + Err(_) => return, + } + }); + + let mut clicks = ClickTracker::default(); + let mut ticker = tokio::time::interval(Duration::from_millis(100)); + ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + let mut last_draw = Instant::now() - Duration::from_secs(1); + let mut dirty = true; + + loop { + if dirty { + term.draw(|f| ui::draw(f, &mut state))?; + last_draw = Instant::now(); + dirty = false; + } + let effects = tokio::select! { + Some(ev) = ev_rx.recv() => { + dirty = true; + state.tick(now_ms()); + handle_event(&mut state, ev, &mut clicks) + } + Some(up) = handle.updates.recv() => { + dirty = true; + state.tick(now_ms()); + apply_update(&mut state, up, &writable); + Vec::new() + } + _ = ticker.tick() => { + state.tick(now_ms()); + // Spinners and toasts animate at 10 fps; otherwise redraw + // once a second so relative times stay true. + if state.animating() || state.toast.is_some() || last_draw.elapsed() >= Duration::from_secs(1) { + dirty = true; + } + if state.paused { Vec::new() } else { state.detail_fetch() } + } + }; + for e in effects { + match e { + Effect::Quit => return Ok(()), + Effect::Refresh => { + let _ = handle.commands.send(Command::RefreshNow); + } + Effect::SetInterval(ms) => { + handle.interval_ms.store(ms, Ordering::Relaxed); + let _ = handle.commands.send(Command::RefreshNow); + } + Effect::Logs { + source, + node, + vm_id, + channel, + } => { + let _ = handle.commands.send(Command::Logs(LogRequest { + source, + node, + vm_id, + channel, + lines: model::LOG_LINES, + })); + } + Effect::Act { + source, + vm_id, + what, + action, + } => { + let _ = handle.commands.send(Command::Act { + source, + vm_id, + what, + action, + }); + } + } + } + } +} + +fn apply_update(state: &mut State, up: Update, writable: &[bool]) { + match up { + Update::Nodes { source, nodes } => { + let w = writable.get(source).copied().unwrap_or(false); + state.apply_nodes(source, nodes, w); + } + Update::Logs { req, result } => { + let Some(node_key) = state + .nodes + .iter() + .find(|n| n.source == req.source && n.snap.name == req.node) + .map(|n| n.key.clone()) + else { + return; + }; + state.apply_logs(&RowKey::Vm(node_key, req.vm_id), req.channel, result); + } + Update::Acted { + what, + action, + result, + } => state.acted(&what, action, result), + } +} + +fn handle_event(state: &mut State, ev: Event, clicks: &mut ClickTracker) -> Vec { + match ev { + Event::Key(k) if k.kind != KeyEventKind::Release => handle_key(state, k), + Event::Mouse(m) => handle_mouse(state, m, clicks), + _ => Vec::new(), + } +} + +fn handle_key(state: &mut State, k: KeyEvent) -> Vec { + let ctrl = k.modifiers.contains(KeyModifiers::CONTROL); + if ctrl && matches!(k.code, KeyCode::Char('c')) { + return vec![Effect::Quit]; + } + if state.help { + state.help = false; + return Vec::new(); + } + // Prompts take every key while open. + match state.input.clone() { + Some(Input::Filter(mut buf)) => { + match k.code { + KeyCode::Esc => { + state.input = None; + state.set_filter(String::new()); + } + KeyCode::Enter => state.input = None, + KeyCode::Backspace => { + buf.pop(); + state.set_filter(buf.clone()); + state.input = Some(Input::Filter(buf)); + } + KeyCode::Char(c) if !ctrl => { + buf.push(c); + state.set_filter(buf.clone()); + state.input = Some(Input::Filter(buf)); + } + KeyCode::Char('u') if ctrl => { + state.set_filter(String::new()); + state.input = Some(Input::Filter(String::new())); + } + _ => {} + } + return Vec::new(); + } + Some(Input::Choose { .. }) => { + match k.code { + KeyCode::Char('u') => state.choose_action(Action::Start), + KeyCode::Char('d') => state.choose_action(Action::Shutdown), + KeyCode::Char('k') => state.choose_action(Action::Stop), + KeyCode::Char('r') => state.choose_action(Action::Restart), + _ => state.input = None, + } + return Vec::new(); + } + Some(Input::Confirm { .. }) => { + return match k.code { + KeyCode::Char('y') | KeyCode::Enter => state.confirm_action().into_iter().collect(), + _ => { + state.input = None; + Vec::new() + } + }; + } + None => {} + } + if state.detail.is_some() { + return handle_detail_key(state, k); + } + let page = state.list_height.max(1) as isize; + match k.code { + KeyCode::Char('q') => return vec![Effect::Quit], + KeyCode::Esc => { + if !state.filter.is_empty() { + state.set_filter(String::new()); + } else if state.preview { + state.preview = false; + } + } + KeyCode::Char('1') => state.set_tab(Tab::Fleet), + KeyCode::Char('2') => state.set_tab(Tab::Vms), + KeyCode::Char('3') => state.set_tab(Tab::Events), + KeyCode::Tab | KeyCode::BackTab => { + let i = Tab::ALL.iter().position(|t| *t == state.tab).unwrap_or(0); + let n = Tab::ALL.len(); + let next = if k.code == KeyCode::Tab { + (i + 1) % n + } else { + (i + n - 1) % n + }; + state.set_tab(Tab::ALL[next]); + } + KeyCode::Up | KeyCode::Char('k') => state.move_by(-1), + KeyCode::Down | KeyCode::Char('j') => state.move_by(1), + KeyCode::PageUp => state.move_by(-page), + KeyCode::PageDown => state.move_by(page), + KeyCode::Home | KeyCode::Char('g') => state.move_to(0), + KeyCode::End | KeyCode::Char('G') => state.move_to(usize::MAX), + KeyCode::Left | KeyCode::Char('h') => state.fold(false), + KeyCode::Right | KeyCode::Char('l') => state.fold(true), + KeyCode::Char('z') => state.toggle_fold_all(), + KeyCode::Enter => return state.open_detail(), + KeyCode::Char('v') => state.preview = !state.preview, + KeyCode::Char('a') => state.cycle_scope(), + KeyCode::Char('s') | KeyCode::Char('>') => state.cycle_sort(true), + KeyCode::Char('S') | KeyCode::Char('<') => state.cycle_sort(false), + KeyCode::Char('I') => state.invert_sort(), + KeyCode::Char('/') => state.input = Some(Input::Filter(state.filter.clone())), + KeyCode::Char('x') => state.begin_action(), + KeyCode::Char('r') => { + state.toast("refreshing…", false); + return vec![Effect::Refresh]; + } + KeyCode::Char('p') | KeyCode::Char(' ') => state.set_paused(!state.paused), + KeyCode::Char('+') | KeyCode::Char('=') => { + return vec![Effect::SetInterval(state.step_interval(true))] + } + KeyCode::Char('-') | KeyCode::Char('_') => { + return vec![Effect::SetInterval(state.step_interval(false))] + } + KeyCode::Char('?') => state.help = true, + _ => {} + } + Vec::new() +} + +fn handle_detail_key(state: &mut State, k: KeyEvent) -> Vec { + let page = state.detail.as_ref().map_or(10, |d| d.body_height.max(1)) as isize; + match k.code { + KeyCode::Esc | KeyCode::Char('q') => state.detail = None, + KeyCode::Tab | KeyCode::Char(']') => return state.step_detail_tab(true), + KeyCode::BackTab | KeyCode::Char('[') => return state.step_detail_tab(false), + KeyCode::Char(c @ '1'..='9') => { + let i = c as usize - '1' as usize; + let tab = state.detail.as_ref().and_then(|d| d.tabs().get(i).copied()); + if let Some(tab) = tab { + return state.set_detail_tab(tab); + } + } + KeyCode::Up | KeyCode::Char('k') => state.scroll_detail(-1), + KeyCode::Down | KeyCode::Char('j') => state.scroll_detail(1), + KeyCode::PageUp => state.scroll_detail(-page), + KeyCode::PageDown | KeyCode::Char(' ') => state.scroll_detail(page), + KeyCode::Home | KeyCode::Char('g') => state.scroll_detail(isize::MIN / 2), + KeyCode::End | KeyCode::Char('G') => state.scroll_detail(isize::MAX / 2), + KeyCode::Left | KeyCode::Char('h') => { + if let Some(d) = state.detail.as_mut() { + d.pan = d.pan.saturating_sub(8); + } + } + KeyCode::Right | KeyCode::Char('l') => { + if let Some(d) = state.detail.as_mut() { + if matches!(d.tab, DetailTab::Log(_)) { + d.pan += 8; + } + } + } + KeyCode::Char('f') => { + if let Some(d) = state.detail.as_mut() { + if matches!(d.tab, DetailTab::Log(_)) { + d.follow = !d.follow; + } + } + if state.detail.as_ref().is_some_and(|d| d.follow) { + state.scroll_detail(isize::MAX / 2); + } + } + KeyCode::Char('x') => { + if let Some(RowKey::Vm(n, v)) = state.detail.as_ref().map(|d| d.target.clone()) { + let prev = state.sel.replace(RowKey::Vm(n, v)); + state.begin_action(); + if state.input.is_none() { + state.sel = prev; + } + } + } + KeyCode::Char('r') => return vec![Effect::Refresh], + KeyCode::Char('?') => state.help = true, + _ => {} + } + Vec::new() +} + +#[derive(Default)] +struct ClickTracker { + last: Option<(Instant, u16, u16)>, +} + +impl ClickTracker { + fn double(&mut self, x: u16, y: u16) -> bool { + let now = Instant::now(); + let double = self.last.is_some_and(|(t, lx, ly)| { + now.duration_since(t) < Duration::from_millis(400) && lx == x && ly == y + }); + self.last = if double { None } else { Some((now, x, y)) }; + double + } +} + +fn handle_mouse(state: &mut State, m: MouseEvent, clicks: &mut ClickTracker) -> Vec { + if state.detail.is_some() { + return match m.kind { + MouseEventKind::ScrollUp => { + state.scroll_detail(-3); + Vec::new() + } + MouseEventKind::ScrollDown => { + state.scroll_detail(3); + Vec::new() + } + MouseEventKind::Down(MouseButton::Left) => { + let hit = state + .hits + .detail_tabs + .iter() + .find(|(x0, x1, y, _)| m.row == *y && m.column >= *x0 && m.column < *x1) + .map(|h| h.3); + match hit { + Some(tab) => state.set_detail_tab(tab), + None => Vec::new(), + } + } + _ => Vec::new(), + }; + } + match m.kind { + MouseEventKind::ScrollUp => state.move_by(-3), + MouseEventKind::ScrollDown => state.move_by(3), + MouseEventKind::Down(MouseButton::Left) => { + if let Some(tab) = state + .hits + .tabs + .iter() + .find(|(x0, x1, y, _)| m.row == *y && m.column >= *x0 && m.column < *x1) + .map(|h| h.3) + { + state.set_tab(tab); + return Vec::new(); + } + let (y0, h) = (state.hits.list_y, state.hits.list_h); + if m.row >= y0 && m.row < y0 + h { + let idx = state.offset + (m.row - y0) as usize; + let was = state.selected_index(); + let double = clicks.double(m.column, m.row); + state.move_to(idx); + if state.selected_index() != idx { + return Vec::new(); + } + // Clicking the selected node folds it; double-click opens. + if double { + return state.open_detail(); + } + if was == idx { + if let Some(RowKey::Node(k)) = state.sel.clone() { + let open = state.folded.contains(&k); + state.fold(open); + } + } + } + } + _ => {} + } + Vec::new() +} diff --git a/dstack/vmm-top/src/tui/model.rs b/dstack/vmm-top/src/tui/model.rs new file mode 100644 index 000000000..173bbb826 --- /dev/null +++ b/dstack/vmm-top/src/tui/model.rs @@ -0,0 +1,1496 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! TUI state and every decision about what is on screen: which rows, in what +//! order, which one is selected, what changed since the last poll. No I/O and +//! no drawing, so all of it is unit-tested. + +use crate::fmt; +use crate::snapshot::{NodeSnapshot, Phase, Vm}; +use crate::source::{Action, LogChannel}; +use std::cmp::Ordering; +use std::collections::{HashMap, HashSet, VecDeque}; + +pub const INTERVAL_STEPS_MS: [u64; 5] = [1000, 2000, 5000, 10_000, 30_000]; +const JOURNAL_CAP: usize = 2000; +pub const LOG_LINES: u32 = 2000; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Tab { + Fleet, + Vms, + Events, +} + +impl Tab { + pub const ALL: [Tab; 3] = [Tab::Fleet, Tab::Vms, Tab::Events]; + + pub fn label(self) -> &'static str { + match self { + Tab::Fleet => "Fleet", + Tab::Vms => "VMs", + Tab::Events => "Events", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Scope { + All, + Live, + Attention, +} + +impl Scope { + pub fn label(self) -> &'static str { + match self { + Scope::All => "all", + Scope::Live => "live", + Scope::Attention => "attention", + } + } + + fn next(self) -> Self { + match self { + Scope::All => Scope::Live, + Scope::Live => Scope::Attention, + Scope::Attention => Scope::All, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SortKey { + Status, + Name, + Node, + Uptime, + Vcpu, + Memory, +} + +impl SortKey { + const ORDER: [SortKey; 6] = [ + SortKey::Status, + SortKey::Name, + SortKey::Node, + SortKey::Uptime, + SortKey::Vcpu, + SortKey::Memory, + ]; + + pub fn label(self) -> &'static str { + match self { + SortKey::Status => "status", + SortKey::Name => "name", + SortKey::Node => "node", + SortKey::Uptime => "uptime", + SortKey::Vcpu => "vcpu", + SortKey::Memory => "memory", + } + } + + fn step(self, forward: bool) -> Self { + let i = Self::ORDER.iter().position(|k| *k == self).unwrap_or(0); + let n = Self::ORDER.len(); + Self::ORDER[if forward { + (i + 1) % n + } else { + (i + n - 1) % n + }] + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum RowKey { + Node(String), + Vm(String, String), + Event(u64), +} + +/// One line of the main list. +#[derive(Clone, Debug)] +pub struct Row { + pub key: RowKey, + pub node: usize, + pub vm: Option, + pub event: Option, + /// Last child under its node, for the tree guide. + pub last: bool, +} + +#[derive(Clone, Debug)] +pub struct NodeEntry { + /// `/`; stable across polls. + pub key: String, + pub source: usize, + pub writable: bool, + pub snap: NodeSnapshot, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EntryKind { + Guest, + Change, + NodeDown, + NodeUp, + Action, +} + +#[derive(Clone, Debug)] +pub struct JournalEntry { + pub seq: u64, + pub at_ms: u64, + pub kind: EntryKind, + pub node_key: String, + pub node: String, + pub vm_id: Option, + pub vm_name: Option, + pub title: String, + pub body: String, + /// Bad news, drawn in the error color. + pub bad: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum DetailTab { + Info, + Log(LogChannel), + Events, +} + +impl DetailTab { + pub const VM: [DetailTab; 5] = [ + DetailTab::Info, + DetailTab::Log(LogChannel::Serial), + DetailTab::Log(LogChannel::Stdout), + DetailTab::Log(LogChannel::Stderr), + DetailTab::Events, + ]; + pub const NODE: [DetailTab; 2] = [DetailTab::Info, DetailTab::Events]; + + pub fn label(self) -> &'static str { + match self { + DetailTab::Info => "Info", + DetailTab::Log(LogChannel::Serial) => "Serial", + DetailTab::Log(LogChannel::Stdout) => "Stdout", + DetailTab::Log(LogChannel::Stderr) => "Stderr", + DetailTab::Events => "Events", + } + } +} + +#[derive(Clone, Debug, Default)] +pub struct LogBuf { + pub lines: Vec, + pub fetched_at_ms: Option, + pub error: Option, + pub loading: bool, +} + +#[derive(Clone, Debug)] +pub struct Detail { + pub target: RowKey, + pub tab: DetailTab, + /// First visible body line. + pub scroll: usize, + /// Columns panned right in log views. + pub pan: usize, + /// Stick to the bottom as logs grow. + pub follow: bool, + pub logs: HashMap<&'static str, LogBuf>, + /// Body height at the last draw, for paging and follow. + pub body_height: usize, +} + +impl Detail { + pub fn new(target: RowKey) -> Self { + Self { + target, + tab: DetailTab::Info, + scroll: 0, + pan: 0, + follow: true, + logs: HashMap::new(), + body_height: 0, + } + } + + pub fn tabs(&self) -> &'static [DetailTab] { + match self.target { + RowKey::Node(_) => &DetailTab::NODE, + _ => &DetailTab::VM, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Input { + Filter(String), + Choose { + node_key: String, + vm_id: String, + vm_name: String, + }, + Confirm { + node_key: String, + vm_id: String, + vm_name: String, + action: Action, + }, +} + +#[derive(Clone, Debug)] +pub struct Toast { + pub text: String, + pub bad: bool, + pub until_ms: u64, +} + +/// Screen regions the last draw put where, for mouse hits. +#[derive(Clone, Debug, Default)] +pub struct Hits { + pub tabs: Vec<(u16, u16, u16, Tab)>, + pub list_y: u16, + pub list_h: u16, + pub detail_tabs: Vec<(u16, u16, u16, DetailTab)>, +} + +/// What the UI asks the collector for after handling input. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Effect { + Refresh, + Logs { + source: usize, + node: String, + vm_id: String, + channel: LogChannel, + }, + Act { + source: usize, + vm_id: String, + what: String, + action: Action, + }, + SetInterval(u64), + Quit, +} + +pub struct State { + pub nodes: Vec, + pub sources: Vec, + pub tab: Tab, + pub scope: Scope, + pub sort: SortKey, + pub sort_rev: bool, + pub filter: String, + pub input: Option, + pub folded: HashSet, + pub sel: Option, + sel_idx: usize, + /// The user picked the selection; until then it stays on the first row + /// while nodes arrive in whatever order they answer. + sel_pinned: bool, + pub offset: usize, + pub list_height: usize, + pub preview: bool, + pub help: bool, + pub detail: Option, + pub journal: VecDeque, + seen_guest: HashSet<(String, String, u64, String)>, + seq: u64, + pub toast: Option, + pub interval_ms: u64, + pub paused: bool, + pub now_ms: u64, + pub updated_ms: u64, + pub frame: u64, + pub read_only: bool, + pub hits: Hits, + /// Polls received while paused, applied on resume. + pending: Vec<(usize, Vec, bool)>, +} + +impl State { + pub fn new(sources: Vec, interval_ms: u64, read_only: bool) -> Self { + Self { + nodes: Vec::new(), + sources, + tab: Tab::Fleet, + scope: Scope::All, + sort: SortKey::Status, + sort_rev: false, + filter: String::new(), + input: None, + folded: HashSet::new(), + sel: None, + sel_idx: 0, + sel_pinned: false, + offset: 0, + list_height: 20, + preview: false, + help: false, + detail: None, + journal: VecDeque::new(), + seen_guest: HashSet::new(), + seq: 0, + toast: None, + interval_ms, + paused: false, + now_ms: 0, + updated_ms: 0, + frame: 0, + read_only, + hits: Hits::default(), + pending: Vec::new(), + } + } + + // ── data in ─────────────────────────────────────────────────────────── + + /// Replace the nodes of one source with a fresh poll, journaling what + /// changed. + pub fn apply_nodes(&mut self, source: usize, nodes: Vec, writable: bool) { + if self.paused { + self.pending.retain(|(s, _, _)| *s != source); + self.pending.push((source, nodes, writable)); + return; + } + let now = self.now_ms; + self.updated_ms = now; + let old: HashMap = self + .nodes + .iter() + .filter(|n| n.source == source) + .map(|n| (n.key.clone(), n.clone())) + .collect(); + let fresh: Vec = nodes + .into_iter() + .map(|snap| NodeEntry { + key: format!("{source}/{}", snap.name), + source, + writable: writable && snap.via.is_none(), + snap, + }) + .collect(); + for n in &fresh { + self.journal_node(old.get(&n.key), n, now); + } + // Keep source order, then the order the source listed its nodes. + let at = self + .nodes + .iter() + .position(|n| n.source > source) + .unwrap_or(self.nodes.len()); + self.nodes.retain(|n| n.source != source); + let at = at.min( + self.nodes + .iter() + .position(|n| n.source > source) + .unwrap_or(self.nodes.len()), + ); + self.nodes.splice(at..at, fresh); + self.clamp_selection(); + } + + fn journal_node(&mut self, old: Option<&NodeEntry>, new: &NodeEntry, now: u64) { + let snap = &new.snap; + let name = display_node(snap); + let entry = |kind, vm: Option<&Vm>, title: String, body: String, bad: bool| JournalEntry { + seq: 0, + at_ms: now, + kind, + node_key: new.key.clone(), + node: name.clone(), + vm_id: vm.map(|v| v.id.clone()), + vm_name: vm.map(|v| v.name.clone()), + title, + body, + bad, + }; + let mut out = Vec::new(); + match (old.map(|o| o.snap.error.is_some()), snap.error.as_ref()) { + (Some(false) | None, Some(err)) => out.push(entry( + EntryKind::NodeDown, + None, + "unreachable".into(), + err.clone(), + true, + )), + (Some(true), None) => out.push(entry( + EntryKind::NodeUp, + None, + "reachable".into(), + String::new(), + false, + )), + _ => {} + } + // Only a poll that succeeded says anything about the VMs. + if snap.error.is_none() { + if let Some(old) = old.filter(|o| o.snap.ok_at_ms.is_some()) { + let before: HashMap<&str, &Vm> = + old.snap.vms.iter().map(|v| (v.id.as_str(), v)).collect(); + for vm in &snap.vms { + match before.get(vm.id.as_str()) { + None => out.push(entry( + EntryKind::Change, + Some(vm), + "created".into(), + vm.phase().label().into(), + false, + )), + Some(prev) if prev.phase() != vm.phase() => { + let to = vm.phase(); + let body = vm.activity().map(str::to_string).unwrap_or_default(); + out.push(entry( + EntryKind::Change, + Some(vm), + format!("{} → {}", prev.phase().label(), to.label()), + body, + matches!(to, Phase::Failed | Phase::Exited), + )); + } + _ => {} + } + } + let now_ids: HashSet<&str> = snap.vms.iter().map(|v| v.id.as_str()).collect(); + for vm in &old.snap.vms { + if !now_ids.contains(vm.id.as_str()) { + out.push(entry( + EntryKind::Change, + Some(vm), + "removed".into(), + String::new(), + false, + )); + } + } + } + for vm in &snap.vms { + for ev in &vm.events { + let key = ( + new.key.clone(), + vm.id.clone(), + ev.timestamp_ms, + format!("{}\u{0}{}", ev.event, ev.body), + ); + if self.seen_guest.insert(key) { + let mut e = entry( + EntryKind::Guest, + Some(vm), + ev.event.clone(), + ev.body.clone(), + ev.event.contains("error"), + ); + e.at_ms = ev.timestamp_ms; + out.push(e); + } + } + } + } + for e in out { + self.push_journal(e); + } + } + + pub fn push_journal(&mut self, mut e: JournalEntry) { + self.seq += 1; + e.seq = self.seq; + // Newest first; guest events can arrive with older timestamps. + let at = self + .journal + .iter() + .position(|j| j.at_ms <= e.at_ms) + .unwrap_or(self.journal.len()); + self.journal.insert(at, e); + while self.journal.len() > JOURNAL_CAP { + self.journal.pop_back(); + } + if self.seen_guest.len() > JOURNAL_CAP * 4 { + self.seen_guest.clear(); + } + } + + pub fn apply_logs( + &mut self, + target: &RowKey, + channel: LogChannel, + result: Result, + ) { + let now = self.now_ms; + let Some(d) = self.detail.as_mut().filter(|d| &d.target == target) else { + return; + }; + let buf = d.logs.entry(channel.as_str()).or_default(); + buf.loading = false; + match result { + Ok(text) => { + buf.lines = text + .lines() + .filter(|l| *l != "[vmm heartbeat]") + .map(fmt::sanitize) + .collect(); + buf.error = None; + buf.fetched_at_ms = Some(now); + } + Err(err) => buf.error = Some(err), + } + if d.follow && d.tab == DetailTab::Log(channel) { + d.scroll = buf.lines.len().saturating_sub(d.body_height.max(1)); + } + } + + pub fn toast(&mut self, text: impl Into, bad: bool) { + self.toast = Some(Toast { + text: text.into(), + bad, + until_ms: self.now_ms + 5000, + }); + } + + pub fn acted(&mut self, what: &str, action: Action, result: Result<(), String>) { + match result { + Ok(()) => self.toast(format!("{} {what}: sent", action.label()), false), + Err(err) => self.toast(format!("{} {what} failed: {err}", action.label()), true), + } + } + + // ── lookups ─────────────────────────────────────────────────────────── + + pub fn node(&self, key: &str) -> Option<&NodeEntry> { + self.nodes.iter().find(|n| n.key == key) + } + + pub fn vm(&self, node_key: &str, vm_id: &str) -> Option<(&NodeEntry, &Vm)> { + let n = self.node(node_key)?; + Some((n, n.snap.vms.iter().find(|v| v.id == vm_id)?)) + } + + pub fn event(&self, seq: u64) -> Option<&JournalEntry> { + self.journal.iter().find(|e| e.seq == seq) + } + + /// Something on screen is changing on its own, so redraw on a timer. + pub fn animating(&self) -> bool { + !self.paused + && self.nodes.iter().any(|n| { + (n.snap.ok_at_ms.is_none() && n.snap.error.is_none()) + || n.snap.vms.iter().any(|v| v.phase().is_transient()) + }) + } + + // ── rows ────────────────────────────────────────────────────────────── + + fn vm_matches(&self, n: &NodeEntry, vm: &Vm) -> bool { + let phase = vm.phase(); + let in_scope = match self.scope { + Scope::All => true, + Scope::Live => phase.is_live(), + Scope::Attention => phase.needs_attention(), + }; + in_scope && (self.filter.is_empty() || self.text_matches(n, Some(vm))) + } + + fn text_matches(&self, n: &NodeEntry, vm: Option<&Vm>) -> bool { + let needle = self.filter.to_lowercase(); + let hit = |s: &str| s.to_lowercase().contains(&needle); + match vm { + None => hit(&n.snap.name) || hit(&n.snap.endpoint), + Some(vm) => { + hit(&vm.name) + || hit(&vm.id) + || hit(&vm.app_id) + || hit(&vm.image) + || hit(vm.phase().label()) + || vm.instance_id.as_deref().is_some_and(hit) + || hit(&n.snap.name) + } + } + } + + fn node_attention(n: &NodeEntry) -> bool { + n.snap.error.is_some() || n.snap.vms.iter().any(|v| v.phase().needs_attention()) + } + + fn cmp_vms(&self, a: (&NodeEntry, &Vm), b: (&NodeEntry, &Vm)) -> Ordering { + let by_name = |a: &Vm, b: &Vm| { + a.name + .to_lowercase() + .cmp(&b.name.to_lowercase()) + .then_with(|| a.id.cmp(&b.id)) + }; + let (na, va) = a; + let (nb, vb) = b; + let ord = match self.sort { + SortKey::Status => va.phase().cmp(&vb.phase()), + SortKey::Name => Ordering::Equal, + SortKey::Node => na.snap.name.cmp(&nb.snap.name), + // Biggest first reads best for quantities. + SortKey::Uptime => vb.uptime_secs.cmp(&va.uptime_secs), + SortKey::Vcpu => vb.vcpu.cmp(&va.vcpu), + SortKey::Memory => vb.memory_mb.cmp(&va.memory_mb), + } + .then_with(|| by_name(va, vb)); + if self.sort_rev { + ord.reverse() + } else { + ord + } + } + + pub fn rows(&self) -> Vec { + match self.tab { + Tab::Fleet => self.fleet_rows(), + Tab::Vms => self.vm_rows(), + Tab::Events => self.event_rows(), + } + } + + fn sorted_vms(&self, node: usize) -> Vec { + let n = &self.nodes[node]; + let mut idx: Vec = (0..n.snap.vms.len()) + .filter(|&i| self.vm_matches(n, &n.snap.vms[i])) + .collect(); + idx.sort_by(|&a, &b| self.cmp_vms((n, &n.snap.vms[a]), (n, &n.snap.vms[b]))); + idx + } + + fn fleet_rows(&self) -> Vec { + let mut rows = Vec::new(); + for (ni, n) in self.nodes.iter().enumerate() { + let vms = self.sorted_vms(ni); + let filtered_out = + !self.filter.is_empty() && vms.is_empty() && !self.text_matches(n, None); + let calm = self.scope == Scope::Attention && !Self::node_attention(n); + if filtered_out || calm { + continue; + } + rows.push(Row { + key: RowKey::Node(n.key.clone()), + node: ni, + vm: None, + event: None, + last: false, + }); + if self.folded.contains(&n.key) { + continue; + } + let count = vms.len(); + for (i, vi) in vms.into_iter().enumerate() { + rows.push(Row { + key: RowKey::Vm(n.key.clone(), n.snap.vms[vi].id.clone()), + node: ni, + vm: Some(vi), + event: None, + last: i + 1 == count, + }); + } + } + rows + } + + fn vm_rows(&self) -> Vec { + let mut all: Vec<(usize, usize)> = Vec::new(); + for (ni, n) in self.nodes.iter().enumerate() { + for (vi, vm) in n.snap.vms.iter().enumerate() { + if self.vm_matches(n, vm) { + all.push((ni, vi)); + } + } + } + all.sort_by(|&(na, va), &(nb, vb)| { + let (a, b) = (&self.nodes[na], &self.nodes[nb]); + self.cmp_vms((a, &a.snap.vms[va]), (b, &b.snap.vms[vb])) + }); + all.into_iter() + .map(|(ni, vi)| Row { + key: RowKey::Vm( + self.nodes[ni].key.clone(), + self.nodes[ni].snap.vms[vi].id.clone(), + ), + node: ni, + vm: Some(vi), + event: None, + last: false, + }) + .collect() + } + + fn event_rows(&self) -> Vec { + let needle = self.filter.to_lowercase(); + self.journal + .iter() + .enumerate() + .filter(|(_, e)| match self.scope { + Scope::Attention => e.bad, + _ => true, + }) + .filter(|(_, e)| { + needle.is_empty() + || [&e.node, &e.title, &e.body] + .into_iter() + .chain(e.vm_name.as_ref()) + .chain(e.vm_id.as_ref()) + .any(|s| s.to_lowercase().contains(&needle)) + }) + .map(|(i, e)| Row { + key: RowKey::Event(e.seq), + node: self + .nodes + .iter() + .position(|n| n.key == e.node_key) + .unwrap_or(usize::MAX), + vm: None, + event: Some(i), + last: false, + }) + .collect() + } + + pub fn count(&self, tab: Tab) -> usize { + match tab { + Tab::Fleet => self.nodes.len(), + Tab::Vms => self + .nodes + .iter() + .map(|n| n.snap.vms.iter().filter(|v| self.vm_matches(n, v)).count()) + .sum(), + Tab::Events => self.journal.len(), + } + } + + // ── selection ───────────────────────────────────────────────────────── + + /// Keep the selection on the same item across polls; when it went away, + /// stay at the same position. + pub fn clamp_selection(&mut self) { + let rows = self.rows(); + if rows.is_empty() { + self.sel = None; + self.sel_idx = 0; + self.offset = 0; + return; + } + let idx = self + .sel + .as_ref() + .filter(|_| self.sel_pinned) + .and_then(|k| rows.iter().position(|r| &r.key == k)) + .unwrap_or(self.sel_idx.min(rows.len() - 1)); + self.select_idx(&rows, idx); + } + + fn select_idx(&mut self, rows: &[Row], idx: usize) { + let idx = idx.min(rows.len().saturating_sub(1)); + self.sel_idx = idx; + self.sel = rows.get(idx).map(|r| r.key.clone()); + let h = self.list_height.max(1); + if idx < self.offset { + self.offset = idx; + } else if idx >= self.offset + h { + self.offset = idx + 1 - h; + } + self.offset = self.offset.min(rows.len().saturating_sub(h)); + } + + pub fn selected_index(&self) -> usize { + self.sel_idx + } + + pub fn move_by(&mut self, delta: isize) { + self.sel_pinned = true; + let rows = self.rows(); + if rows.is_empty() { + return; + } + let idx = (self.sel_idx as isize + delta).clamp(0, rows.len() as isize - 1); + self.select_idx(&rows, idx as usize); + } + + pub fn move_to(&mut self, idx: usize) { + self.sel_pinned = true; + let rows = self.rows(); + self.select_idx(&rows, idx); + } + + pub fn set_tab(&mut self, tab: Tab) { + if self.tab != tab { + self.tab = tab; + self.sel = None; + self.sel_idx = 0; + self.sel_pinned = false; + self.offset = 0; + self.clamp_selection(); + } + } + + // ── input ───────────────────────────────────────────────────────────── + + pub fn fold(&mut self, open: bool) { + self.sel_pinned = true; + match self.sel.clone() { + Some(RowKey::Node(k)) => { + if open { + self.folded.remove(&k); + } else { + self.folded.insert(k); + } + } + Some(RowKey::Vm(k, _)) if !open && self.tab == Tab::Fleet => { + self.sel = Some(RowKey::Node(k)); + } + _ => {} + } + self.clamp_selection(); + } + + pub fn toggle_fold_all(&mut self) { + self.sel_pinned = true; + if self.folded.is_empty() { + self.folded = self.nodes.iter().map(|n| n.key.clone()).collect(); + if let Some(RowKey::Vm(k, _)) = self.sel.clone() { + self.sel = Some(RowKey::Node(k)); + } + } else { + self.folded.clear(); + } + self.clamp_selection(); + } + + pub fn cycle_scope(&mut self) { + self.scope = self.scope.next(); + self.clamp_selection(); + } + + pub fn cycle_sort(&mut self, forward: bool) { + self.sort = self.sort.step(forward); + self.clamp_selection(); + } + + pub fn invert_sort(&mut self) { + self.sort_rev = !self.sort_rev; + self.clamp_selection(); + } + + pub fn set_filter(&mut self, filter: String) { + self.filter = filter; + self.clamp_selection(); + } + + pub fn set_paused(&mut self, paused: bool) { + self.paused = paused; + if !paused { + for (s, nodes, w) in std::mem::take(&mut self.pending) { + self.apply_nodes(s, nodes, w); + } + } + } + + pub fn step_interval(&mut self, faster: bool) -> u64 { + let i = INTERVAL_STEPS_MS + .iter() + .position(|&s| s >= self.interval_ms) + .unwrap_or(INTERVAL_STEPS_MS.len() - 1); + let i = if faster { + i.saturating_sub(1) + } else { + (i + 1).min(INTERVAL_STEPS_MS.len() - 1) + }; + self.interval_ms = INTERVAL_STEPS_MS[i]; + self.interval_ms + } + + /// The VM the selection points at, directly or through an event. + pub fn selected_vm(&self) -> Option<(String, String)> { + match self.sel.as_ref()? { + RowKey::Vm(n, v) => Some((n.clone(), v.clone())), + RowKey::Event(seq) => { + let e = self.event(*seq)?; + Some((e.node_key.clone(), e.vm_id.clone()?)) + } + RowKey::Node(_) => None, + } + } + + /// Start the action prompt for the selected VM. + pub fn begin_action(&mut self) { + if self.read_only { + self.toast("read-only: started with --read-only", true); + return; + } + let Some((node_key, vm_id)) = self.selected_vm() else { + self.toast("select a VM to act on", true); + return; + }; + let Some((n, vm)) = self.vm(&node_key, &vm_id) else { + return; + }; + if !n.writable { + let why = match &n.snap.via { + Some(hub) => format!("read-only: {} is watched through hub {hub}", n.snap.name), + None => format!("read-only: {}", n.snap.name), + }; + self.toast(why, true); + return; + } + self.input = Some(Input::Choose { + node_key, + vm_id, + vm_name: vm.name.clone(), + }); + } + + pub fn choose_action(&mut self, action: Action) { + if let Some(Input::Choose { + node_key, + vm_id, + vm_name, + }) = self.input.take() + { + self.input = Some(Input::Confirm { + node_key, + vm_id, + vm_name, + action, + }); + } + } + + pub fn confirm_action(&mut self) -> Option { + let Some(Input::Confirm { + node_key, + vm_id, + vm_name, + action, + }) = self.input.take() + else { + return None; + }; + let n = self.node(&node_key)?; + let what = format!("{vm_name} on {}", n.snap.name); + let source = n.source; + let node = n.snap.name.clone(); + let now = self.now_ms; + self.push_journal(JournalEntry { + seq: 0, + at_ms: now, + kind: EntryKind::Action, + node_key, + node, + vm_id: Some(vm_id.clone()), + vm_name: Some(vm_name), + title: action.label().into(), + body: "requested from this terminal".into(), + bad: false, + }); + self.toast(format!("{}: {what}…", action.label()), false); + Some(Effect::Act { + source, + vm_id, + what, + action, + }) + } + + /// Open the detail view for the selection. + pub fn open_detail(&mut self) -> Vec { + let target = match self.sel.clone() { + Some(RowKey::Event(seq)) => match self.event(seq) { + Some(e) => match &e.vm_id { + Some(v) if self.vm(&e.node_key, v).is_some() => { + RowKey::Vm(e.node_key.clone(), v.clone()) + } + _ if self.node(&e.node_key).is_some() => RowKey::Node(e.node_key.clone()), + _ => return vec![], + }, + None => return vec![], + }, + Some(k) => k, + None => return vec![], + }; + self.detail = Some(Detail::new(target)); + self.detail_fetch() + } + + pub fn set_detail_tab(&mut self, tab: DetailTab) -> Vec { + let Some(d) = self.detail.as_mut() else { + return vec![]; + }; + if !d.tabs().contains(&tab) { + return vec![]; + } + d.tab = tab; + d.pan = 0; + d.follow = matches!(tab, DetailTab::Log(_)); + d.scroll = match tab { + DetailTab::Log(ch) => d + .logs + .get(ch.as_str()) + .map_or(0, |b| b.lines.len().saturating_sub(d.body_height.max(1))), + _ => 0, + }; + self.detail_fetch() + } + + pub fn step_detail_tab(&mut self, forward: bool) -> Vec { + let Some(d) = self.detail.as_ref() else { + return vec![]; + }; + let tabs = d.tabs(); + let i = tabs.iter().position(|t| *t == d.tab).unwrap_or(0); + let n = tabs.len(); + let next = tabs[if forward { + (i + 1) % n + } else { + (i + n - 1) % n + }]; + self.set_detail_tab(next) + } + + /// Log fetch for the open detail tab, if it shows a log that is due. + pub fn detail_fetch(&mut self) -> Vec { + let Some(d) = self.detail.as_ref() else { + return vec![]; + }; + let (DetailTab::Log(channel), RowKey::Vm(node_key, vm_id)) = (d.tab, &d.target) else { + return vec![]; + }; + let Some(n) = self.node(node_key) else { + return vec![]; + }; + let effect = Effect::Logs { + source: n.source, + node: n.snap.name.clone(), + vm_id: vm_id.clone(), + channel, + }; + let interval = self.interval_ms; + let now = self.now_ms; + let Some(d) = self.detail.as_mut() else { + return vec![]; + }; + let buf = d.logs.entry(channel.as_str()).or_default(); + let due = buf + .fetched_at_ms + .is_none_or(|t| now.saturating_sub(t) >= interval); + if buf.loading || !due { + return vec![]; + } + buf.loading = true; + vec![effect] + } + + pub fn detail_lines_len(&self) -> usize { + let Some(d) = self.detail.as_ref() else { + return 0; + }; + match d.tab { + DetailTab::Log(ch) => d.logs.get(ch.as_str()).map_or(0, |b| b.lines.len()), + _ => usize::MAX, + } + } + + pub fn scroll_detail(&mut self, delta: isize) { + let len = self.detail_lines_len(); + let Some(d) = self.detail.as_mut() else { + return; + }; + let max = if len == usize::MAX { + usize::MAX / 2 + } else { + len.saturating_sub(d.body_height.max(1)) + }; + let next = (d.scroll as isize + delta).clamp(0, max as isize) as usize; + d.scroll = next; + if matches!(d.tab, DetailTab::Log(_)) { + d.follow = next >= max; + } + } + + pub fn tick(&mut self, now_ms: u64) { + self.now_ms = now_ms; + self.frame = self.frame.wrapping_add(1); + if self.toast.as_ref().is_some_and(|t| t.until_ms <= now_ms) { + self.toast = None; + } + } +} + +/// `name` or `hub/name` when read through a hub. +pub fn display_node(s: &NodeSnapshot) -> String { + match &s.via { + Some(hub) if hub != &s.name => format!("{hub}/{}", s.name), + _ => s.name.clone(), + } +} + +// ── fleet totals ──────────────────────────────────────────────────────────── + +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Totals { + pub nodes: usize, + pub nodes_up: usize, + pub nodes_connecting: usize, + pub vms: usize, + pub by_phase: Vec<(Phase, usize)>, + pub vcpu_used: u64, + pub vcpu_max: u64, + pub mem_used_mb: u64, + pub mem_max_mb: u64, + pub gpus_used: usize, + pub gpus: usize, + pub disk_gb: u64, +} + +pub fn totals(nodes: &[NodeEntry]) -> Totals { + let mut t = Totals { + nodes: nodes.len(), + ..Default::default() + }; + let mut phases: HashMap = HashMap::new(); + for n in nodes { + let s = &n.snap; + if s.online() { + t.nodes_up += 1; + } else if s.error.is_none() { + t.nodes_connecting += 1; + } + t.vms += s.vms.len(); + for vm in &s.vms { + *phases.entry(vm.phase()).or_default() += 1; + } + t.vcpu_used += s.vcpu_used() as u64; + t.mem_used_mb += s.memory_used_mb(); + t.disk_gb += s.disk_gb(); + if let Some(r) = &s.resources { + t.vcpu_max += r.max_vcpu as u64; + t.mem_max_mb += r.max_memory_mb as u64; + } + t.gpus += s.gpus.len(); + t.gpus_used += s.gpus_used(); + } + let mut by_phase: Vec<_> = phases.into_iter().collect(); + by_phase.sort(); + t.by_phase = by_phase; + t +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::snapshot::{GuestEvent, Resources}; + + fn vm(id: &str, name: &str, status: &str, vcpu: u32) -> Vm { + Vm { + id: id.into(), + name: name.into(), + status: status.into(), + running: status == "running", + boot_progress: "done".into(), + vcpu, + memory_mb: 1024 * vcpu, + ..Default::default() + } + } + + fn node(name: &str, vms: Vec) -> NodeSnapshot { + NodeSnapshot { + name: name.into(), + endpoint: format!("http://{name}:9080"), + ok_at_ms: Some(1), + resources: Some(Resources { + max_cvms: 10, + max_vcpu: 64, + max_memory_mb: 65536, + }), + vms, + ..Default::default() + } + } + + fn state() -> State { + let mut s = State::new(vec!["a".into(), "b".into()], 2000, false); + s.now_ms = 10_000; + s.apply_nodes( + 0, + vec![node( + "a", + vec![ + vm("1", "web", "running", 4), + vm("2", "db", "exited", 8), + vm("3", "old", "stopped", 2), + ], + )], + true, + ); + s.apply_nodes(1, vec![node("b", vec![vm("4", "api", "running", 2)])], true); + s + } + + fn keys(s: &State) -> Vec { + s.rows() + .iter() + .map(|r| match &r.key { + RowKey::Node(k) => format!("N:{k}"), + RowKey::Vm(_, v) => v.clone(), + RowKey::Event(e) => format!("E{e}"), + }) + .collect() + } + + #[test] + fn fleet_tree_sorts_by_status_then_name() { + let s = state(); + // exited (attention) before running before stopped. + assert_eq!(keys(&s), ["N:0/a", "2", "1", "3", "N:1/b", "4"]); + let mut s = s; + s.sort = SortKey::Name; + s.clamp_selection(); + assert_eq!(keys(&s), ["N:0/a", "2", "3", "1", "N:1/b", "4"]); + s.sort = SortKey::Vcpu; + assert_eq!(keys(&s), ["N:0/a", "2", "1", "3", "N:1/b", "4"]); + s.invert_sort(); + assert_eq!(keys(&s), ["N:0/a", "3", "1", "2", "N:1/b", "4"]); + } + + #[test] + fn scopes_and_filter() { + let mut s = state(); + s.cycle_scope(); // live + assert_eq!(keys(&s), ["N:0/a", "1", "N:1/b", "4"]); + s.cycle_scope(); // attention: calm node b disappears + assert_eq!(keys(&s), ["N:0/a", "2"]); + s.cycle_scope(); + s.set_filter("API".into()); + assert_eq!(keys(&s), ["N:1/b", "4"]); + s.set_filter("b".into()); // "web", "db", and every VM on node "b" + assert_eq!(keys(&s), ["N:0/a", "2", "1", "N:1/b", "4"]); + } + + #[test] + fn vm_tab_is_flat_across_nodes() { + let mut s = state(); + s.set_tab(Tab::Vms); + s.sort = SortKey::Name; + assert_eq!(keys(&s), ["4", "2", "3", "1"]); + assert_eq!(s.count(Tab::Vms), 4); + } + + #[test] + fn selection_follows_item_across_polls() { + let mut s = state(); + s.move_to(3); // "3" old + assert_eq!(s.sel, Some(RowKey::Vm("0/a".into(), "3".into()))); + // A new VM shifts "old" down a row; the selection moves with it. + s.apply_nodes( + 0, + vec![node( + "a", + vec![ + vm("0", "new", "running", 1), + vm("1", "web", "exited", 4), + vm("2", "db", "exited", 8), + vm("3", "old", "stopped", 2), + ], + )], + true, + ); + assert_eq!(s.sel, Some(RowKey::Vm("0/a".into(), "3".into()))); + assert_eq!(s.selected_index(), 4); + // It disappears: selection stays at the same position. + s.apply_nodes(0, vec![node("a", vec![vm("0", "new", "running", 1)])], true); + assert_eq!(s.selected_index(), 3); + assert_eq!(s.sel, Some(RowKey::Vm("1/b".into(), "4".into()))); + } + + #[test] + fn selection_stays_on_top_until_the_user_moves() { + let mut s = State::new(vec!["a".into(), "b".into()], 2000, false); + s.apply_nodes(1, vec![node("b", vec![])], true); + assert_eq!(s.sel, Some(RowKey::Node("1/b".into()))); + s.apply_nodes(0, vec![node("a", vec![])], true); + assert_eq!(s.sel, Some(RowKey::Node("0/a".into()))); + } + + #[test] + fn folding() { + let mut s = state(); + s.move_to(1); + s.fold(false); // on a VM: jump to its node + assert_eq!(s.sel, Some(RowKey::Node("0/a".into()))); + s.fold(false); + assert_eq!(keys(&s), ["N:0/a", "N:1/b", "4"]); + s.fold(true); + assert_eq!(keys(&s).len(), 6); + s.toggle_fold_all(); + assert_eq!(keys(&s), ["N:0/a", "N:1/b"]); + s.toggle_fold_all(); + assert_eq!(keys(&s).len(), 6); + } + + #[test] + fn journal_records_transitions_and_outages() { + let mut s = state(); + let base = s.journal.len(); + s.now_ms = 20_000; + s.apply_nodes( + 0, + vec![node( + "a", + vec![vm("1", "web", "exited", 4), vm("2", "db", "exited", 8)], + )], + true, + ); + let titles: Vec<_> = s.journal.iter().take(2).map(|e| e.title.clone()).collect(); + assert!(titles.contains(&"running → exited".to_string())); + assert!(titles.contains(&"removed".to_string())); + assert!(s + .journal + .iter() + .any(|e| e.bad && e.title == "running → exited")); + + let mut down = node("a", vec![]); + down.error = Some("connection refused".into()); + s.apply_nodes(0, vec![down], true); + assert_eq!(s.journal[0].kind, EntryKind::NodeDown); + // A failed poll says nothing about VMs: no "removed" storm. + assert_eq!(s.journal.iter().filter(|e| e.title == "removed").count(), 1); + s.apply_nodes(0, vec![node("a", vec![])], true); + assert_eq!(s.journal[0].kind, EntryKind::NodeUp); + assert!(s.journal.len() > base); + } + + #[test] + fn guest_events_are_deduplicated_and_ordered() { + let mut s = State::new(vec!["a".into()], 2000, false); + s.now_ms = 50_000; + let mut v = vm("1", "web", "running", 1); + v.events = vec![ + GuestEvent { + event: "boot.progress".into(), + body: "done".into(), + timestamp_ms: 2000, + }, + GuestEvent { + event: "boot.progress".into(), + body: "booting".into(), + timestamp_ms: 1000, + }, + ]; + s.apply_nodes(0, vec![node("a", vec![v.clone()])], true); + s.apply_nodes(0, vec![node("a", vec![v])], true); + let guest: Vec<_> = s + .journal + .iter() + .filter(|e| e.kind == EntryKind::Guest) + .map(|e| e.at_ms) + .collect(); + assert_eq!(guest, [2000, 1000]); + } + + #[test] + fn actions_need_a_writable_direct_node() { + let mut s = state(); + s.move_to(1); + s.begin_action(); + assert!(matches!(s.input, Some(Input::Choose { .. }))); + s.choose_action(Action::Stop); + let eff = s.confirm_action().unwrap(); + assert!(matches!( + eff, + Effect::Act { + source: 0, + action: Action::Stop, + .. + } + )); + assert_eq!(s.journal[0].kind, EntryKind::Action); + + let mut hubbed = node("c", vec![vm("9", "x", "running", 1)]); + hubbed.via = Some("hub".into()); + s.apply_nodes(1, vec![hubbed], true); + s.sel = Some(RowKey::Vm("1/c".into(), "9".into())); + s.begin_action(); + assert!(s.input.is_none()); + assert!(s.toast.as_ref().unwrap().text.contains("hub")); + } + + #[test] + fn restart_goes_through_the_same_confirmation() { + let mut s = state(); + s.move_to(1); + s.begin_action(); + s.choose_action(Action::Restart); + assert!(matches!( + s.input, + Some(Input::Confirm { + action: Action::Restart, + .. + }) + )); + let eff = s.confirm_action().unwrap(); + assert!(matches!( + eff, + Effect::Act { + action: Action::Restart, + .. + } + )); + assert_eq!(s.journal[0].title, "restart"); + } + + #[test] + fn pause_defers_updates() { + let mut s = state(); + s.set_paused(true); + s.apply_nodes(1, vec![node("b", vec![])], true); + assert_eq!(s.nodes[1].snap.vms.len(), 1); + s.set_paused(false); + assert_eq!(s.nodes[1].snap.vms.len(), 0); + } + + #[test] + fn interval_steps() { + let mut s = state(); + assert_eq!(s.step_interval(true), 1000); + assert_eq!(s.step_interval(true), 1000); + assert_eq!(s.step_interval(false), 2000); + assert_eq!(s.step_interval(false), 5000); + } + + #[test] + fn totals_count_live_allocation() { + let s = state(); + let t = totals(&s.nodes); + assert_eq!(t.nodes, 2); + assert_eq!(t.nodes_up, 2); + assert_eq!(t.vms, 4); + // exited and stopped VMs hold nothing. + assert_eq!(t.vcpu_used, 6); + assert_eq!(t.vcpu_max, 128); + } + + #[test] + fn logs_follow_tail() { + let mut s = state(); + s.move_to(1); + s.open_detail(); + let eff = s.set_detail_tab(DetailTab::Log(LogChannel::Serial)); + assert_eq!(eff.len(), 1); + s.detail.as_mut().unwrap().body_height = 10; + let target = s.detail.as_ref().unwrap().target.clone(); + let text: String = (0..100).map(|i| format!("line {i}\n")).collect(); + s.apply_logs(&target, LogChannel::Serial, Ok(text)); + assert_eq!(s.detail.as_ref().unwrap().scroll, 90); + s.scroll_detail(-5); + assert!(!s.detail.as_ref().unwrap().follow); + s.scroll_detail(100); + assert!(s.detail.as_ref().unwrap().follow); + // Not due again until an interval passed. + assert!(s.detail_fetch().is_empty()); + s.now_ms += 5000; + assert_eq!(s.detail_fetch().len(), 1); + } +} diff --git a/dstack/vmm-top/src/tui/ui.rs b/dstack/vmm-top/src/tui/ui.rs new file mode 100644 index 000000000..ae4cde49c --- /dev/null +++ b/dstack/vmm-top/src/tui/ui.rs @@ -0,0 +1,1801 @@ +// SPDX-FileCopyrightText: © 2026 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Ratatui drawing. A function of [`State`] and the frame size; no I/O. +//! +//! Visual language: no boxed panels on the main screen, dim labels beside +//! bright values, color reserved for status glyphs, meters and the accent. The +//! selected row carries a `▌` marker on a slightly lifted background. Every +//! line is clipped to the terminal width; nothing wraps. + +use super::model::{ + display_node, totals, Detail, DetailTab, EntryKind, Input, JournalEntry, NodeEntry, Row, + RowKey, State, Tab, Totals, +}; +use crate::fmt; +use crate::snapshot::{Phase, Vm}; +use crate::source::LogChannel; +use ratatui::layout::{Constraint, Direction, Layout, Rect}; +use ratatui::style::{Color, Modifier, Style}; +use ratatui::text::{Line, Span}; +use ratatui::widgets::{Block, BorderType, Borders, Clear, Paragraph}; +use ratatui::Frame; + +// ── palette ──────────────────────────────────────────────────────────────── + +const ACCENT: Color = Color::Rgb(130, 170, 255); +const OK: Color = Color::Rgb(134, 200, 120); +const ERR: Color = Color::Rgb(240, 110, 120); +const WARN: Color = Color::Rgb(230, 180, 90); +const TEXT: Color = Color::Rgb(222, 225, 232); +const MUTED: Color = Color::Rgb(150, 156, 172); +const DIM: Color = Color::Rgb(100, 106, 124); +const RULE: Color = Color::Rgb(62, 66, 80); +const SEL_BG: Color = Color::Rgb(38, 42, 56); + +const SPINNER: [&str; 10] = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"]; + +fn fg(c: Color) -> Style { + Style::default().fg(c) +} + +fn sp(style: Style, text: impl Into) -> Span<'static> { + Span::styled(text.into(), style) +} + +fn dim(text: impl Into) -> Span<'static> { + sp(fg(DIM), text) +} + +fn val(text: impl Into) -> Span<'static> { + let t: String = text.into(); + if t == "-" { + dim(t) + } else { + sp(fg(TEXT), t) + } +} + +fn spans_width(spans: &[Span]) -> usize { + spans.iter().map(|s| fmt::width(&s.content)).sum() +} + +/// Clip spans to `width` columns, ending with "…" when cut. +fn fit(spans: Vec>, width: usize) -> Vec> { + if spans_width(&spans) <= width { + return spans; + } + let mut out = Vec::new(); + let mut used = 0; + for s in spans { + let w = fmt::width(&s.content); + if used + w < width { + used += w; + out.push(s); + continue; + } + let room = width - used; + if room > 0 { + out.push(Span::styled(fmt::truncate(&s.content, room), s.style)); + } + break; + } + out +} + +/// Exactly `width` columns: clipped, then padded on the right or left. +fn cell(spans: Vec>, width: usize, right: bool) -> Vec> { + let mut spans = fit(spans, width); + let pad = width.saturating_sub(spans_width(&spans)); + if pad > 0 { + if right { + spans.insert(0, Span::raw(" ".repeat(pad))); + } else { + spans.push(Span::raw(" ".repeat(pad))); + } + } + spans +} + +/// `left`, then `right` flush to the right edge; `right` is dropped when the +/// two would touch. +fn split_line( + mut left: Vec>, + right: Vec>, + width: usize, +) -> Vec> { + let (lw, rw) = (spans_width(&left), spans_width(&right)); + if rw == 0 || lw + rw + 2 > width { + return fit(left, width); + } + left.push(Span::raw(" ".repeat(width - lw - rw))); + left.extend(right); + left +} + +/// `key label` pairs: key bright, label dim. +fn hints(pairs: &[(&str, &str)]) -> Vec> { + let mut out = vec![Span::raw(" ")]; + for (i, (k, l)) in pairs.iter().enumerate() { + if i > 0 { + out.push(Span::raw(" ")); + } + out.push(sp(fg(MUTED).add_modifier(Modifier::BOLD), *k)); + out.push(dim(format!(" {l}"))); + } + out +} + +/// Only the pairs that fit, so a narrow footer never ends mid-word. +fn hints_fit(pairs: &[(&str, &str)], width: usize) -> Vec> { + let mut n = pairs.len(); + while n > 0 && spans_width(&hints(&pairs[..n])) > width { + n -= 1; + } + hints(&pairs[..n]) +} + +/// `▰▰▰▱▱▱` meter, colored by how full it is. +fn meter(used: u64, total: u64, cells: usize) -> Vec> { + if total == 0 { + return vec![sp(fg(RULE), "▱".repeat(cells))]; + } + let ratio = used as f64 / total as f64; + let fill = ((ratio * cells as f64).ceil() as usize).min(cells); + let fill = if used > 0 { fill.max(1) } else { 0 }; + let color = if ratio >= 0.9 { + ERR + } else if ratio >= 0.7 { + WARN + } else { + ACCENT + }; + vec![ + sp(fg(color), "▰".repeat(fill)), + sp(fg(RULE), "▱".repeat(cells - fill)), + ] +} + +fn pct(used: u64, total: u64) -> String { + if total == 0 { + "-".into() + } else { + format!("{}%", used * 100 / total) + } +} + +fn spinner(state: &State) -> &'static str { + SPINNER[(state.frame as usize) % SPINNER.len()] +} + +fn phase_color(p: Phase) -> Color { + match p { + Phase::Running => OK, + Phase::Failed | Phase::Exited => ERR, + Phase::Booting => ACCENT, + Phase::ShuttingDown | Phase::Stopping | Phase::Removing => WARN, + Phase::Stopped | Phase::Unknown => DIM, + } +} + +fn phase_glyph(state: &State, p: Phase) -> Span<'static> { + let g = match p { + p if p.is_transient() => spinner(state), + Phase::Running => "●", + Phase::Failed => "✕", + Phase::Exited => "◆", + Phase::Stopped => "○", + _ => "?", + }; + sp(fg(phase_color(p)), g) +} + +fn node_glyph(state: &State, n: &NodeEntry) -> Span<'static> { + if n.snap.error.is_some() { + sp(fg(ERR), "✕") + } else if n.snap.ok_at_ms.is_none() { + sp(fg(ACCENT), spinner(state)) + } else { + sp(fg(OK), "●") + } +} + +fn interval_text(ms: u64) -> String { + if ms.is_multiple_of(1000) { + format!("{}s", ms / 1000) + } else { + format!("{:.1}s", ms as f64 / 1000.0) + } +} + +/// Refresh state plus the time of the last applied poll, so every refresh +/// is visible even when no number changed. +fn live_pill(state: &State) -> Vec> { + let at = if state.updated_ms > 0 { + format!(" · {}", fmt::clock(state.updated_ms)) + } else { + String::new() + }; + if state.paused { + vec![sp(fg(WARN), "‖ paused"), dim(format!("{at} "))] + } else { + vec![ + sp(fg(OK), "● "), + sp( + fg(MUTED), + format!("live {}", interval_text(state.interval_ms)), + ), + dim(format!("{at} ")), + ] + } +} + +// ── screen ───────────────────────────────────────────────────────────────── + +pub fn draw(frame: &mut Frame, state: &mut State) { + let area = frame.area(); + if state.detail.is_some() { + draw_detail(frame, area, state); + if state.help { + draw_help(frame, area); + } + return; + } + state.hits.detail_tabs.clear(); + let short = area.height < 20; + let head_h = if short { 2 } else { 7 }; + let preview_on = state.preview && !short; + let preview_h = if preview_on { + let rest = area.height.saturating_sub(head_h + 3); + (rest * if area.height >= 32 { 40 } else { 35 } / 100).max(4) + } else { + 0 + }; + let chunks = Layout::default() + .direction(Direction::Vertical) + .constraints([ + Constraint::Length(head_h), + Constraint::Length(1), + Constraint::Length(1), + Constraint::Min(1), + Constraint::Length(preview_h), + Constraint::Length(1), + ]) + .split(area); + let t = totals(&state.nodes); + draw_header(frame, chunks[0], state, &t, short); + draw_tabs(frame, chunks[1], state); + let rows = state.rows(); + draw_list(frame, chunks[2], chunks[3], state, &rows); + if preview_h > 0 { + draw_preview(frame, chunks[4], state); + } + draw_footer(frame, chunks[5], state); + if state.help { + draw_help(frame, area); + } +} + +fn brand_line(state: &State, t: &Totals, width: usize) -> Line<'static> { + let mut left = vec![ + sp(fg(ACCENT).add_modifier(Modifier::BOLD), " ◆ "), + sp(fg(TEXT).add_modifier(Modifier::BOLD), "dstack"), + sp(fg(MUTED), " vmm"), + dim(format!( + " {} node{}", + t.nodes, + if t.nodes == 1 { "" } else { "s" } + )), + ]; + if state.sources.len() != t.nodes { + left.push(dim(format!(" · {} sources", state.sources.len()))); + } + if state.read_only { + left.push(dim(" · read-only")); + } + Line::from(split_line(left, live_pill(state), width)) +} + +fn draw_header(frame: &mut Frame, area: Rect, state: &State, t: &Totals, short: bool) { + let width = area.width as usize; + if short { + let mut l = vec![dim(" nodes ")]; + l.push(nodes_value(t)); + l.push(dim(" vms ")); + l.extend(phase_summary(state, t, true)); + l.push(dim(" vcpu ")); + l.push(val(pct(t.vcpu_used, t.vcpu_max))); + l.push(dim(" mem ")); + l.push(val(pct(t.mem_used_mb, t.mem_max_mb))); + frame.render_widget( + Paragraph::new(vec![brand_line(state, t, width), Line::from(fit(l, width))]), + area, + ); + return; + } + let rows = Layout::default() + .direction(Direction::Vertical) + .constraints([ + Constraint::Length(1), + Constraint::Length(5), + Constraint::Min(0), + ]) + .split(area); + frame.render_widget(Paragraph::new(brand_line(state, t, width)), rows[0]); + let split = if width >= 110 { 50 } else { 100 }; + let cols = Layout::default() + .direction(Direction::Horizontal) + .constraints([ + Constraint::Percentage(split), + Constraint::Percentage(100 - split), + ]) + .split(rows[1]); + + let stat = |label: &str, value: Vec>| { + let mut out = vec![dim(format!(" {label:<8}"))]; + out.extend(value); + out + }; + let lw = cols[0].width as usize; + let mut left = Vec::new(); + let mut nodes = vec![nodes_value(t)]; + if t.nodes > t.nodes_up + t.nodes_connecting { + nodes.push(dim(" · ")); + nodes.push(sp( + fg(ERR), + format!("{} down", t.nodes - t.nodes_up - t.nodes_connecting), + )); + } + if t.nodes_connecting > 0 { + nodes.push(dim(format!(" · {} connecting", t.nodes_connecting))); + } + left.push(stat("nodes", nodes)); + left.push(stat("vms", phase_summary(state, t, false))); + let mut vcpu = meter(t.vcpu_used, t.vcpu_max, 12); + vcpu.extend([ + val(format!(" {}", t.vcpu_used)), + dim(format!("/{} allocated", t.vcpu_max)), + ]); + left.push(stat("vcpu", vcpu)); + let mut mem = meter(t.mem_used_mb, t.mem_max_mb, 12); + mem.extend([ + val(format!(" {}", fmt::mem_mb(t.mem_used_mb))), + dim(format!("/{}", fmt::mem_mb(t.mem_max_mb))), + dim(format!(" · disk {}", fmt::disk_gb(t.disk_gb))), + ]); + left.push(stat("memory", mem)); + if t.gpus > 0 { + let mut gpu = meter(t.gpus_used as u64, t.gpus as u64, 12); + gpu.extend([ + val(format!(" {}", t.gpus_used)), + dim(format!("/{} attached", t.gpus)), + ]); + left.push(stat("gpu", gpu)); + } else { + left.push(stat("gpu", vec![dim("none on these nodes")])); + } + let left: Vec = left.into_iter().map(|l| Line::from(fit(l, lw))).collect(); + frame.render_widget(Paragraph::new(left), cols[0]); + if split < 100 { + draw_node_board(frame, cols[1], state); + } +} + +fn nodes_value(t: &Totals) -> Span<'static> { + let color = if t.nodes_up == t.nodes { OK } else { WARN }; + sp(fg(color), format!("{}/{} up", t.nodes_up, t.nodes)) +} + +/// `12 running · 1 booting · 2 failed` with the count of each phase present. +fn phase_summary(state: &State, t: &Totals, compact: bool) -> Vec> { + if t.vms == 0 { + return vec![dim("none")]; + } + let mut out = Vec::new(); + for (i, (p, n)) in t.by_phase.iter().enumerate() { + if i > 0 { + out.push(dim(if compact { " " } else { " · " })); + } + if compact { + out.push(phase_glyph(state, *p)); + out.push(val(n.to_string())); + } else { + out.push(sp( + fg(phase_color(*p)).add_modifier(Modifier::BOLD), + n.to_string(), + )); + out.push(dim(format!(" {}", p.label()))); + } + } + out +} + +/// Right half of the header: one line per node, like a status board. +fn draw_node_board(frame: &mut Frame, area: Rect, state: &State) { + let w = area.width as usize; + let mut lines = vec![Line::from(fit( + vec![dim(format!( + " {:<16} {:>5} {:>6} {:>6} {:>5} {}", + "node", "vms", "vcpu", "mem", "gpu", "rtt" + ))], + w, + ))]; + let room = (area.height as usize).saturating_sub(1); + let shown = if state.nodes.len() > room { + room.saturating_sub(1) + } else { + state.nodes.len() + }; + for n in state.nodes.iter().take(shown) { + let s = &n.snap; + let mut row = vec![ + Span::raw(" "), + node_glyph(state, n), + sp(fg(TEXT), format!(" {}", fmt::pad(&display_node(s), 14))), + ]; + if s.ok_at_ms.is_none() { + row.push(dim(format!( + " {:>5} {:>6} {:>6} {:>5} ", + "-", "-", "-", "-" + ))); + } else { + let (vu, vm) = ( + s.vcpu_used() as u64, + s.resources.as_ref().map_or(0, |r| r.max_vcpu as u64), + ); + let (mu, mm) = ( + s.memory_used_mb(), + s.resources.as_ref().map_or(0, |r| r.max_memory_mb as u64), + ); + let running = s.vms.iter().filter(|v| v.phase().is_live()).count(); + let level = |used: u64, max: u64| { + let p = pct(used, max); + let c = if max > 0 && used * 10 >= max * 9 { + ERR + } else if max > 0 && used * 10 >= max * 7 { + WARN + } else { + TEXT + }; + (p, c) + }; + let (vp, vc) = level(vu, vm); + let (mp, mc) = level(mu, mm); + row.extend([ + val(format!(" {:>5}", format!("{running}/{}", s.vms.len()))), + sp(fg(vc), format!(" {vp:>6}")), + sp(fg(mc), format!(" {mp:>6}")), + val(format!( + " {:>5}", + if s.gpus.is_empty() { + "-".to_string() + } else { + format!("{}/{}", s.gpus_used(), s.gpus.len()) + } + )), + Span::raw(" "), + ]); + } + row.push(match (&s.error, s.latency_ms) { + (Some(_), _) => sp(fg(ERR), "down"), + (None, Some(ms)) => dim(format!("{ms}ms")), + (None, None) => dim("…"), + }); + lines.push(Line::from(fit(row, w))); + } + if shown < state.nodes.len() { + lines.push(Line::from(dim(format!( + " +{} more · Fleet tab lists all", + state.nodes.len() - shown + )))); + } + frame.render_widget(Paragraph::new(lines), area); +} + +fn draw_tabs(frame: &mut Frame, area: Rect, state: &mut State) { + let width = area.width as usize; + let mut spans = vec![Span::raw(" ")]; + let mut x = 1usize; + state.hits.tabs.clear(); + for (i, tab) in Tab::ALL.into_iter().enumerate() { + let active = state.tab == tab; + let start = x; + let key = format!("{}", i + 1); + spans.push(dim(format!("{key} "))); + x += key.len() + 1; + let name_style = if active { + fg(ACCENT).add_modifier(Modifier::BOLD | Modifier::UNDERLINED) + } else { + fg(MUTED) + }; + spans.push(sp(name_style, tab.label())); + x += tab.label().len(); + let c = format!(" {}", state.count(tab)); + x += c.len(); + spans.push(sp(if active { fg(ACCENT) } else { fg(DIM) }, c)); + state.hits.tabs.push((start as u16, x as u16, area.y, tab)); + spans.push(Span::raw(" ")); + x += 4; + } + let mut right = Vec::new(); + if !state.filter.is_empty() && !matches!(state.input, Some(Input::Filter(_))) { + right.extend([ + dim("filter "), + sp(fg(WARN), state.filter.clone()), + dim(" "), + ]); + } + right.extend([ + dim("scope "), + sp(fg(MUTED), state.scope.label()), + dim(" sort "), + sp( + fg(MUTED), + format!( + "{} {}", + state.sort.label(), + if state.sort_rev { "↑" } else { "↓" } + ), + ), + Span::raw(" "), + ]); + frame.render_widget( + Paragraph::new(Line::from(split_line(spans, right, width))), + area, + ); +} + +// ── main list ────────────────────────────────────────────────────────────── + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +enum Col { + Name, + Node, + Status, + Uptime, + Vcpu, + Mem, + Disk, + Gpu, + Id, + Image, + App, + Ports, +} + +impl Col { + fn title(self) -> &'static str { + match self { + Col::Name => "NAME", + Col::Node => "NODE", + Col::Status => "STATUS", + Col::Uptime => "UPTIME", + Col::Vcpu => "VCPU", + Col::Mem => "MEM", + Col::Disk => "DISK", + Col::Gpu => "GPU", + Col::Id => "ID", + Col::Image => "IMAGE", + Col::App => "APP ID", + Col::Ports => "PORTS", + } + } + + fn width(self) -> usize { + match self { + Col::Name => 0, + Col::Node => 14, + Col::Status => 14, + Col::Uptime => 7, + Col::Vcpu => 7, + Col::Mem => 10, + Col::Disk => 6, + Col::Gpu => 5, + Col::Id => 8, + Col::Image => 16, + Col::App => 10, + Col::Ports => 16, + } + } + + fn right(self) -> bool { + matches!( + self, + Col::Uptime | Col::Vcpu | Col::Mem | Col::Disk | Col::Gpu + ) + } +} + +const GUTTER: usize = 2; +const SEP: usize = 2; +const NAME_MIN: usize = 20; + +/// The columns that fit `width`, and the width left for the name. +/// Compact under 100 columns, standard to 139, wide from 140. +fn columns(tab: Tab, width: usize) -> (Vec, usize) { + let mut cols = vec![Col::Name]; + if tab == Tab::Vms { + cols.push(Col::Node); + } + cols.extend([Col::Status, Col::Uptime, Col::Vcpu, Col::Mem]); + if width >= 100 { + cols.extend([Col::Disk, Col::Gpu, Col::Id, Col::Image]); + } + if width >= 140 { + cols.extend([Col::App, Col::Ports]); + } + loop { + let fixed: usize = cols.iter().map(|c| c.width() + SEP).sum::() - SEP; + let name = width.saturating_sub(GUTTER + fixed); + if name >= NAME_MIN || cols.len() <= 2 { + return (cols, name.max(8)); + } + cols.pop(); + } +} + +fn header_line(cols: &[Col], name_w: usize, width: usize) -> Line<'static> { + let mut spans = vec![Span::raw(" ".repeat(GUTTER))]; + for (i, c) in cols.iter().enumerate() { + if i > 0 { + spans.push(Span::raw(" ".repeat(SEP))); + } + let w = if *c == Col::Name { name_w } else { c.width() }; + spans.extend(cell(vec![dim(c.title())], w, c.right())); + } + Line::from(fit(spans, width)) +} + +fn draw_list(frame: &mut Frame, head: Rect, area: Rect, state: &mut State, rows: &[Row]) { + let width = area.width as usize; + state.list_height = area.height as usize; + state.hits.list_y = area.y; + state.hits.list_h = area.height; + state.clamp_selection(); + if state.tab == Tab::Events { + draw_events(frame, head, area, state, rows); + return; + } + let (cols, name_w) = columns(state.tab, width); + frame.render_widget(Paragraph::new(header_line(&cols, name_w, width)), head); + if rows.is_empty() { + frame.render_widget(Paragraph::new(empty_hint(state)), area); + return; + } + let sel = state.selected_index(); + let lines: Vec = rows + .iter() + .enumerate() + .skip(state.offset) + .take(area.height as usize) + .map(|(i, r)| { + let n = &state.nodes[r.node]; + let mut spans = vec![Span::raw(" ".repeat(GUTTER))]; + for (ci, c) in cols.iter().enumerate() { + if ci > 0 { + spans.push(Span::raw(" ".repeat(SEP))); + } + let w = if *c == Col::Name { name_w } else { c.width() }; + let content = match r.vm { + None => node_cell(state, n, *c), + Some(vi) => vm_cell(state, n, &n.snap.vms[vi], r, *c), + }; + spans.extend(cell(content, w, c.right())); + } + row_line(fit(spans, width), i == sel, width) + }) + .collect(); + frame.render_widget(Paragraph::new(lines), area); +} + +/// A row with the selection marker and background when selected. +fn row_line(mut spans: Vec>, selected: bool, width: usize) -> Line<'static> { + if !selected { + return Line::from(spans); + } + let w = spans_width(&spans); + if w < width { + spans.push(Span::raw(" ".repeat(width - w))); + } + if let Some(first) = spans.first_mut() { + if first.content.starts_with(' ') { + let rest: String = first.content.chars().skip(1).collect(); + *first = Span::raw(rest); + spans.insert(0, sp(fg(ACCENT), "▌")); + } + } + Line::from(spans).style(Style::default().bg(SEL_BG)) +} + +fn empty_hint(state: &State) -> Vec> { + let msg = if state.nodes.is_empty() { + "connecting…".to_string() + } else if !state.filter.is_empty() { + format!( + "nothing matches \"{}\" · esc clears the filter", + state.filter + ) + } else { + format!( + "nothing in scope \"{}\" · a cycles scope", + state.scope.label() + ) + }; + vec![Line::from(""), Line::from(dim(format!(" {msg}")))] +} + +fn node_cell(state: &State, n: &NodeEntry, col: Col) -> Vec> { + let s = &n.snap; + let up = s.ok_at_ms.is_some(); + let res = s.resources.as_ref(); + match col { + Col::Name => { + let fold = if state.folded.contains(&n.key) { + "▸ " + } else { + "▾ " + }; + let mut v = vec![ + dim(fold), + node_glyph(state, n), + sp( + fg(TEXT).add_modifier(Modifier::BOLD), + format!(" {}", s.name), + ), + ]; + if let Some(hub) = &s.via { + v.push(dim(format!(" via {hub}"))); + } + if up { + v.push(dim(format!( + " {} vm{}", + s.vms.len(), + if s.vms.len() == 1 { "" } else { "s" } + ))); + } + v + } + Col::Status => match (&s.error, s.ok_at_ms) { + (Some(_), Some(ok)) => vec![ + sp(fg(ERR), "down"), + dim(format!(" {}", fmt::ago(state.now_ms, ok))), + ], + (Some(_), None) => vec![sp(fg(ERR), "unreachable")], + (None, None) => vec![dim("connecting")], + (None, Some(_)) => vec![ + sp(fg(MUTED), "online"), + dim(s.latency_ms.map(|l| format!(" {l}ms")).unwrap_or_default()), + ], + }, + Col::Vcpu if up => frac( + s.vcpu_used().to_string(), + res.map(|r| r.max_vcpu.to_string()), + ), + Col::Mem if up => frac( + fmt::mem_mb(s.memory_used_mb()), + res.map(|r| fmt::mem_mb(r.max_memory_mb as u64)), + ), + Col::Disk if up => vec![val(fmt::disk_gb(s.disk_gb()))], + Col::Gpu if up && !s.gpus.is_empty() => { + frac(s.gpus_used().to_string(), Some(s.gpus.len().to_string())) + } + Col::Image => match &s.version { + Some(v) => vec![dim(format!("vmm {v}"))], + None => vec![], + }, + Col::App | Col::Ports if col == Col::Ports => vec![dim(s.endpoint.clone())], + _ => vec![], + } +} + +fn frac(used: String, max: Option) -> Vec> { + match max { + Some(m) => vec![val(used), dim(format!("/{m}"))], + None => vec![val(used)], + } +} + +fn vm_cell(state: &State, n: &NodeEntry, vm: &Vm, r: &Row, col: Col) -> Vec> { + let phase = vm.phase(); + let quiet = matches!(phase, Phase::Stopped | Phase::Unknown); + let stale = n.snap.error.is_some(); + let text = if quiet || stale { fg(MUTED) } else { fg(TEXT) }; + match col { + Col::Name => { + let mut v = Vec::new(); + if state.tab == Tab::Fleet { + v.push(sp(fg(RULE), if r.last { " └ " } else { " ├ " })); + } + v.push(sp(text, vm.name.clone())); + if let Some(a) = vm.activity().filter(|a| !a.is_empty()) { + let c = if phase == Phase::Failed { ERR } else { DIM }; + v.push(sp(fg(c), format!(" {a}"))); + } + v + } + Col::Node => vec![sp(fg(MUTED), display_node(&n.snap))], + Col::Status => { + let label_style = match phase { + Phase::Failed | Phase::Exited => fg(ERR), + _ if quiet => fg(DIM), + _ => fg(MUTED), + }; + let mut v = vec![ + phase_glyph(state, phase), + sp(label_style, format!(" {}", phase.label())), + ]; + if stale { + v.push(dim(" ?")); + } + v + } + Col::Uptime => vec![val(vm + .uptime_secs + .map(fmt::duration) + .unwrap_or_else(|| "-".into()))], + Col::Vcpu => vec![sp(text, vm.vcpu.to_string())], + Col::Mem => vec![sp(text, fmt::mem_mb(vm.memory_mb as u64))], + Col::Disk => vec![sp(text, fmt::disk_gb(vm.disk_gb as u64))], + Col::Gpu => vec![val(if vm.gpus.is_empty() { + "-".to_string() + } else { + vm.gpus.len().to_string() + })], + Col::Id => vec![dim(vm.short_id().to_string())], + Col::Image => vec![sp(fg(MUTED), vm.image.clone())], + Col::App => vec![dim(crate::snapshot::short(&vm.app_id, 10).to_string())], + Col::Ports => vec![dim(ports_text(vm))], + } +} + +fn ports_text(vm: &Vm) -> String { + if vm.ports.is_empty() { + return "-".into(); + } + vm.ports + .iter() + .map(|p| format!("{}→{}", p.host_port, p.vm_port)) + .collect::>() + .join(" ") +} + +// ── events ───────────────────────────────────────────────────────────────── + +fn draw_events(frame: &mut Frame, head: Rect, area: Rect, state: &State, rows: &[Row]) { + let width = area.width as usize; + let node_w = 14; + let vm_w = 18; + let ev_w = 20; + frame.render_widget( + Paragraph::new(Line::from(fit( + vec![dim(format!( + "{}{:<14} {} {} {} DETAIL", + " ".repeat(GUTTER + 2), + "TIME", + fmt::pad("NODE", node_w), + fmt::pad("VM", vm_w), + fmt::pad("EVENT", ev_w), + ))], + width, + ))), + head, + ); + if rows.is_empty() { + let msg = if state.journal.is_empty() { + "no events yet · state changes and guest events land here" + } else { + "nothing matches" + }; + frame.render_widget( + Paragraph::new(vec![Line::from(""), Line::from(dim(format!(" {msg}")))]), + area, + ); + return; + } + let sel = state.selected_index(); + let lines: Vec = rows + .iter() + .enumerate() + .skip(state.offset) + .take(area.height as usize) + .filter_map(|(i, r)| { + let e = &state.journal[r.event?]; + let spans = vec![ + Span::raw(" ".repeat(GUTTER)), + event_glyph(e), + Span::raw(" "), + dim(format!("{:<14}", fmt::stamp(e.at_ms))), + Span::raw(" "), + sp(fg(MUTED), fmt::pad(&e.node, node_w)), + Span::raw(" "), + sp( + fg(TEXT), + fmt::pad(e.vm_name.as_deref().unwrap_or("-"), vm_w), + ), + Span::raw(" "), + sp( + if e.bad { fg(ERR) } else { fg(MUTED) }, + fmt::pad(&e.title, ev_w), + ), + Span::raw(" "), + dim(e.body.replace('\n', " ")), + ]; + Some(row_line(fit(spans, width), i == sel, width)) + }) + .collect(); + frame.render_widget(Paragraph::new(lines), area); +} + +fn event_glyph(e: &JournalEntry) -> Span<'static> { + match e.kind { + EntryKind::Guest if e.bad => sp(fg(ERR), "·"), + EntryKind::Guest => sp(fg(DIM), "·"), + EntryKind::Change if e.bad => sp(fg(ERR), "◆"), + EntryKind::Change => sp(fg(ACCENT), "◆"), + EntryKind::NodeDown => sp(fg(ERR), "✕"), + EntryKind::NodeUp => sp(fg(OK), "●"), + EntryKind::Action => sp(fg(WARN), "▸"), + } +} + +// ── info (preview + detail) ──────────────────────────────────────────────── + +fn kv(key: &str, value: Vec>) -> Line<'static> { + let mut spans = vec![dim(format!(" {key:<12}"))]; + spans.extend(value); + Line::from(spans) +} + +fn section(title: &str) -> Line<'static> { + Line::from(vec![ + Span::raw(" "), + sp(fg(MUTED).add_modifier(Modifier::BOLD), title.to_string()), + ]) +} + +fn info_lines(state: &State, key: &RowKey, events: usize) -> Vec> { + match key { + RowKey::Node(k) => state + .node(k) + .map(|n| node_info(state, n)) + .unwrap_or_default(), + RowKey::Vm(nk, id) => state + .vm(nk, id) + .map(|(n, vm)| vm_info(state, n, vm, events)) + .unwrap_or_else(|| vec![Line::from(dim(" this VM is gone"))]), + RowKey::Event(seq) => match state.event(*seq) { + Some(e) => { + let mut l = vec![ + kv("when", vec![val(fmt::stamp(e.at_ms))]), + kv("node", vec![val(e.node.clone())]), + kv( + "vm", + vec![val(e.vm_name.clone().unwrap_or_else(|| "-".into()))], + ), + kv( + "event", + vec![event_glyph(e), Span::raw(" "), val(e.title.clone())], + ), + ]; + for (i, line) in e.body.lines().enumerate() { + l.push(kv( + if i == 0 { "detail" } else { "" }, + vec![val(line.to_string())], + )); + } + l.push(Line::from(dim(" enter opens the VM"))); + l + } + None => vec![], + }, + } +} + +fn vm_info(state: &State, n: &NodeEntry, vm: &Vm, events: usize) -> Vec> { + let phase = vm.phase(); + let mut status = vec![ + phase_glyph(state, phase), + val(format!(" {}", phase.label())), + ]; + if let Some(up) = vm.uptime_secs { + status.push(dim(format!(" · up {}", fmt::duration(up)))); + } + if let Some(a) = vm.activity().filter(|a| !a.is_empty()) { + let c = if phase == Phase::Failed { ERR } else { MUTED }; + status.push(sp(fg(c), format!(" · {a}"))); + } + if let Some(err) = &n.snap.error { + status.push(sp( + fg(WARN), + format!( + " · as of {} (node: {err})", + n.snap.ok_at_ms.map(fmt::clock).unwrap_or_default() + ), + )); + } + let mut l = vec![ + kv("status", status), + kv("id", vec![val(vm.id.clone())]), + kv("app id", vec![val(or_dash(&vm.app_id))]), + kv( + "instance", + vec![val(vm.instance_id.clone().unwrap_or_else(|| "-".into()))], + ), + kv( + "node", + vec![ + val(display_node(&n.snap)), + dim(format!(" {}", n.snap.endpoint)), + ], + ), + kv( + "image", + vec![ + val(or_dash(&vm.image)), + dim(if vm.image_version.is_empty() { + String::new() + } else { + format!(" v{}", vm.image_version) + }), + ], + ), + kv( + "resources", + vec![ + val(vm.vcpu.to_string()), + dim(" vcpu · "), + val(fmt::mem_mb(vm.memory_mb as u64)), + dim(" memory · "), + val(fmt::disk_gb(vm.disk_gb as u64)), + dim(" disk"), + ], + ), + kv( + "tee", + vec![if vm.no_tee { + sp(fg(WARN), "off · plain VM") + } else { + val("on") + }], + ), + ]; + if !vm.gpus.is_empty() { + l.push(kv("gpus", vec![val(vm.gpus.join(", "))])); + } + if !vm.networks.is_empty() { + l.push(kv("network", vec![val(vm.networks.join(", "))])); + } + if !vm.ports.is_empty() { + let ports = vm + .ports + .iter() + .map(|p| { + let addr = if p.host_address.is_empty() { + "127.0.0.1" + } else { + &p.host_address + }; + format!("{} {addr}:{}→{}", p.protocol, p.host_port, p.vm_port) + }) + .collect::>() + .join(", "); + l.push(kv("ports", vec![val(ports)])); + } + if let Some(url) = &vm.app_url { + l.push(kv("app url", vec![sp(fg(ACCENT), url.clone())])); + } + if events > 0 && !vm.events.is_empty() { + l.push(Line::from("")); + l.push(section("recent guest events")); + let mut evs: Vec<_> = vm.events.iter().collect(); + evs.sort_by(|a, b| b.timestamp_ms.cmp(&a.timestamp_ms)); + for e in evs.into_iter().take(events) { + l.push(Line::from(vec![ + dim(format!(" {} ", fmt::clock(e.timestamp_ms))), + sp(fg(MUTED), fmt::pad(&e.event, 18)), + val(format!(" {}", e.body.replace('\n', " "))), + ])); + } + } + l +} + +fn node_info(state: &State, n: &NodeEntry) -> Vec> { + let s = &n.snap; + let mut status = vec![node_glyph(state, n)]; + match (&s.error, s.ok_at_ms) { + (Some(e), ok) => { + status.push(sp(fg(ERR), format!(" {e}"))); + if let Some(ok) = ok { + status.push(dim(format!(" · last ok {}", fmt::ago(state.now_ms, ok)))); + } + } + (None, Some(_)) => { + status.push(val(" online")); + if let Some(l) = s.latency_ms { + status.push(dim(format!(" · {l}ms"))); + } + } + (None, None) => status.push(dim(" connecting")), + } + let mut l = vec![ + kv("status", status), + kv("endpoint", vec![val(s.endpoint.clone())]), + ]; + if let Some(hub) = &s.via { + l.push(kv("via hub", vec![val(hub.clone()), dim(" read-only")])); + } else if !n.writable { + l.push(kv("actions", vec![dim("read-only")])); + } + l.push(kv( + "vmm", + vec![ + val(s.version.clone().unwrap_or_else(|| "-".into())), + dim(s + .rev + .as_ref() + .map(|r| format!(" {}", crate::snapshot::short(r, 12))) + .unwrap_or_default()), + ], + )); + if let Some(r) = &s.resources { + let live = s.live_vms().count() as u64; + let mut cvms = meter(live, r.max_cvms as u64, 10); + cvms.extend([ + val(format!(" {live}")), + dim(format!("/{} slots", r.max_cvms)), + ]); + l.push(kv("cvms", cvms)); + let mut vcpu = meter(s.vcpu_used() as u64, r.max_vcpu as u64, 10); + vcpu.extend([ + val(format!(" {}", s.vcpu_used())), + dim(format!("/{}", r.max_vcpu)), + ]); + l.push(kv("vcpu", vcpu)); + let mut mem = meter(s.memory_used_mb(), r.max_memory_mb as u64, 10); + mem.extend([ + val(format!(" {}", fmt::mem_mb(s.memory_used_mb()))), + dim(format!("/{}", fmt::mem_mb(r.max_memory_mb as u64))), + ]); + l.push(kv("memory", mem)); + } + l.push(kv( + "disk", + vec![val(fmt::disk_gb(s.disk_gb())), dim(" provisioned")], + )); + l.push(kv("kms", vec![val(join_or_dash(&s.kms_urls))])); + l.push(kv("gateway", vec![val(join_or_dash(&s.gateway_urls))])); + if let Some(d) = &s.base_domain { + l.push(kv("base domain", vec![val(d.clone())])); + } + if !s.gpus.is_empty() { + l.push(Line::from("")); + l.push(section("gpus")); + for g in &s.gpus { + let owner = s + .vms + .iter() + .find(|v| v.gpus.iter().any(|slot| slot == &g.slot)) + .map(|v| v.name.clone()); + l.push(Line::from(vec![ + Span::raw(" "), + if g.is_free { + dim("○") + } else { + sp(fg(ACCENT), "●") + }, + sp(fg(TEXT), format!(" {:<14}", g.slot)), + sp(fg(MUTED), fmt::pad(&g.description, 32)), + dim(match owner { + Some(o) => format!(" {o}"), + None if g.is_free => " free".into(), + None => " in use".into(), + }), + ])); + } + } + l +} + +fn or_dash(s: &str) -> String { + if s.is_empty() { + "-".into() + } else { + s.to_string() + } +} + +fn join_or_dash(v: &[String]) -> String { + if v.is_empty() { + "-".into() + } else { + v.join(", ") + } +} + +fn draw_preview(frame: &mut Frame, area: Rect, state: &State) { + let width = area.width as usize; + let Some(key) = state.sel.clone() else { + return; + }; + let title = match &key { + RowKey::Node(k) => state.node(k).map(|n| display_node(&n.snap)), + RowKey::Vm(nk, id) => state + .vm(nk, id) + .map(|(n, v)| format!("{} · {} on {}", v.name, v.short_id(), n.snap.name)), + RowKey::Event(seq) => state + .event(*seq) + .map(|e| format!("{} · {}", e.node, e.title)), + } + .unwrap_or_default(); + let head = format!("── {title} "); + let rule = "─".repeat(width.saturating_sub(fmt::width(&head))); + let mut lines = vec![Line::from(vec![ + sp(fg(RULE), "── "), + sp(fg(MUTED), format!("{title} ")), + sp(fg(RULE), rule), + ])]; + lines.extend( + info_lines(state, &key, 3) + .into_iter() + .map(|l| Line::from(fit(l.spans, width))), + ); + lines.truncate(area.height as usize); + frame.render_widget(Paragraph::new(lines), area); +} + +// ── footer ───────────────────────────────────────────────────────────────── + +fn draw_footer(frame: &mut Frame, area: Rect, state: &State) { + let width = area.width as usize; + let line = match &state.input { + Some(Input::Filter(buf)) => { + let left = vec![ + sp(fg(ACCENT).add_modifier(Modifier::BOLD), " / "), + sp(fg(TEXT), buf.clone()), + sp(fg(ACCENT), "▏"), + ]; + split_line(left, hints(&[("enter", "keep"), ("esc", "clear")]), width) + } + Some(Input::Choose { + vm_name, node_key, .. + }) => { + let node = state + .node(node_key) + .map(|n| n.snap.name.clone()) + .unwrap_or_default(); + let mut left = vec![ + sp(fg(WARN).add_modifier(Modifier::BOLD), " ▸ "), + sp(fg(TEXT), vm_name.clone()), + dim(format!(" on {node} ")), + ]; + left.extend(hints(&[ + ("u", "start"), + ("d", "shutdown"), + ("r", "restart"), + ("k", "force stop"), + ("esc", "cancel"), + ])); + fit(left, width) + } + Some(Input::Confirm { + vm_name, + node_key, + action, + .. + }) => { + let node = state + .node(node_key) + .map(|n| n.snap.name.clone()) + .unwrap_or_default(); + let c = if action.is_forceful() { ERR } else { WARN }; + let mut left = vec![ + sp( + fg(c).add_modifier(Modifier::BOLD), + format!(" {} ", action.label()), + ), + sp(fg(TEXT), vm_name.clone()), + dim(format!(" on {node}? ")), + ]; + left.extend(hints(&[("y", "confirm"), ("n", "cancel")])); + fit(left, width) + } + None => match &state.toast { + Some(t) => fit( + vec![sp(fg(if t.bad { ERR } else { OK }), format!(" {}", t.text))], + width, + ), + None => hints_fit( + &[ + ("↑↓", "move"), + ("enter", "open"), + ("x", "act"), + ("/", "filter"), + ("a", "scope"), + ("s", "sort"), + ("v", "preview"), + ("←→", "fold"), + ("p", "pause"), + ("?", "help"), + ("q", "quit"), + ], + width, + ), + }, + }; + frame.render_widget(Paragraph::new(Line::from(line)), area); +} + +// ── detail ───────────────────────────────────────────────────────────────── + +fn draw_detail(frame: &mut Frame, area: Rect, state: &mut State) { + frame.render_widget(Clear, area); + let width = area.width as usize; + let chunks = Layout::default() + .direction(Direction::Vertical) + .constraints([ + Constraint::Length(1), + Constraint::Length(1), + Constraint::Length(1), + Constraint::Min(1), + Constraint::Length(1), + ]) + .split(area); + let Some(d) = state.detail.clone() else { + return; + }; + + // Title. + let mut title = vec![sp(fg(ACCENT).add_modifier(Modifier::BOLD), " ◆ ")]; + match &d.target { + RowKey::Vm(nk, id) => match state.vm(nk, id) { + Some((n, vm)) => { + let p = vm.phase(); + title.extend([ + sp(fg(TEXT).add_modifier(Modifier::BOLD), vm.name.clone()), + Span::raw(" "), + phase_glyph(state, p), + sp(fg(MUTED), format!(" {}", p.label())), + dim(format!(" on {}", display_node(&n.snap))), + ]); + } + None => title.push(dim("gone")), + }, + RowKey::Node(k) => { + if let Some(n) = state.node(k) { + title.extend([ + sp(fg(TEXT).add_modifier(Modifier::BOLD), display_node(&n.snap)), + Span::raw(" "), + node_glyph(state, n), + dim(format!(" {}", n.snap.endpoint)), + ]); + } + } + RowKey::Event(_) => {} + } + frame.render_widget( + Paragraph::new(Line::from(split_line(title, live_pill(state), width))), + chunks[0], + ); + + // Tabs. + let mut tabs = vec![Span::raw(" ")]; + let mut x = 1usize; + state.hits.detail_tabs.clear(); + for (i, t) in d.tabs().iter().enumerate() { + let active = *t == d.tab; + let start = x; + let key = format!("{} ", i + 1); + x += key.len() + t.label().len(); + tabs.push(dim(key)); + tabs.push(sp( + if active { + fg(ACCENT).add_modifier(Modifier::BOLD | Modifier::UNDERLINED) + } else { + fg(MUTED) + }, + t.label(), + )); + state + .hits + .detail_tabs + .push((start as u16, x as u16, chunks[1].y, *t)); + tabs.push(Span::raw(" ")); + x += 4; + } + let body_h = chunks[3].height as usize; + let right = detail_status(state, &d, body_h); + frame.render_widget( + Paragraph::new(Line::from(split_line(tabs, right, width))), + chunks[1], + ); + frame.render_widget( + Paragraph::new(Line::from(sp(fg(RULE), "─".repeat(width)))), + chunks[2], + ); + + // Body. + if let Some(dm) = state.detail.as_mut() { + dm.body_height = body_h; + } + let body: Vec = match d.tab { + DetailTab::Info => { + let lines = info_lines(state, &d.target, 8); + let max = lines.len().saturating_sub(body_h); + let scroll = d.scroll.min(max); + if let Some(dm) = state.detail.as_mut() { + dm.scroll = scroll; + } + lines + .into_iter() + .skip(scroll) + .take(body_h) + .map(|l| Line::from(fit(l.spans, width))) + .collect() + } + DetailTab::Events => { + let lines = detail_events(state, &d.target); + let max = lines.len().saturating_sub(body_h); + let scroll = d.scroll.min(max); + if let Some(dm) = state.detail.as_mut() { + dm.scroll = scroll; + } + lines + .into_iter() + .skip(scroll) + .take(body_h) + .map(|l| Line::from(fit(l.spans, width))) + .collect() + } + DetailTab::Log(ch) => log_body(&d, ch, body_h, width), + }; + frame.render_widget(Paragraph::new(body), chunks[3]); + + let foot = match &state.input { + Some(Input::Choose { .. }) | Some(Input::Confirm { .. }) => None, + _ => state + .toast + .as_ref() + .map(|t| vec![sp(fg(if t.bad { ERR } else { OK }), format!(" {}", t.text))]), + }; + let foot = match foot { + Some(f) => fit(f, width), + None if state.input.is_some() => { + // Reuse the main footer's prompt rendering. + draw_footer(frame, chunks[4], state); + return; + } + None => hints_fit( + &[ + ("tab", "next"), + ("1-5", "jump"), + ("↑↓", "scroll"), + ("←→", "pan"), + ("f", "follow"), + ("x", "act"), + ("esc", "back"), + ], + width, + ), + }; + frame.render_widget(Paragraph::new(Line::from(foot)), chunks[4]); +} + +fn detail_status(state: &State, d: &Detail, body_h: usize) -> Vec> { + let DetailTab::Log(ch) = d.tab else { + return vec![]; + }; + let Some(buf) = d.logs.get(ch.as_str()) else { + return vec![dim("loading… ")]; + }; + let mut out = Vec::new(); + if buf.error.is_some() { + out.push(sp(fg(ERR), "fetch failed ")); + } + if d.follow { + out.push(sp(fg(OK), "● ")); + out.push(sp(fg(MUTED), "following")); + } else { + let end = (d.scroll + body_h).min(buf.lines.len()); + out.push(dim(format!( + "lines {}–{end} of {}", + d.scroll + 1, + buf.lines.len() + ))); + } + if let Some(t) = buf.fetched_at_ms { + out.push(dim(format!(" · {}", fmt::clock(t)))); + } + if buf.loading { + out.push(sp(fg(ACCENT), format!(" {}", spinner(state)))); + } + out.push(Span::raw(" ")); + out +} + +fn log_body(d: &Detail, ch: LogChannel, body_h: usize, width: usize) -> Vec> { + let Some(buf) = d.logs.get(ch.as_str()) else { + return vec![Line::from(dim(" loading…"))]; + }; + let mut out = Vec::new(); + if let Some(e) = &buf.error { + out.push(Line::from(sp(fg(ERR), format!(" {e}")))); + } + if buf.lines.is_empty() && buf.error.is_none() { + out.push(Line::from(dim(if buf.fetched_at_ms.is_some() { + " (empty)" + } else { + " loading…" + }))); + } + let gutter = 6; + let room = width.saturating_sub(gutter + 1); + for (i, line) in buf + .lines + .iter() + .enumerate() + .skip(d.scroll) + .take(body_h - out.len().min(body_h)) + { + let text: String = skip_cols(line, d.pan); + out.push(Line::from(vec![ + sp(fg(RULE), format!("{:>5} ", i + 1)), + sp(fg(TEXT), fmt::truncate(&text, room)), + ])); + } + out +} + +/// `s` without its first `cols` display columns. +fn skip_cols(s: &str, cols: usize) -> String { + if cols == 0 { + return s.to_string(); + } + let mut used = 0; + let mut out = String::new(); + for c in s.chars() { + if used >= cols { + out.push(c); + } else { + used += fmt::width(c.encode_utf8(&mut [0; 4])); + } + } + out +} + +fn detail_events(state: &State, target: &RowKey) -> Vec> { + let entries: Vec<&JournalEntry> = state + .journal + .iter() + .filter(|e| match target { + RowKey::Vm(nk, id) => &e.node_key == nk && e.vm_id.as_ref() == Some(id), + RowKey::Node(nk) => &e.node_key == nk, + RowKey::Event(_) => false, + }) + .collect(); + if entries.is_empty() { + return vec![Line::from(dim(" no events seen yet"))]; + } + entries + .into_iter() + .map(|e| { + Line::from(vec![ + Span::raw(" "), + event_glyph(e), + dim(format!(" {} ", fmt::stamp(e.at_ms))), + sp( + if e.bad { fg(ERR) } else { fg(MUTED) }, + fmt::pad(&e.title, 20), + ), + if matches!(target, RowKey::Node(_)) { + sp( + fg(TEXT), + format!(" {}", fmt::pad(e.vm_name.as_deref().unwrap_or("-"), 18)), + ) + } else { + Span::raw("") + }, + dim(format!(" {}", e.body.replace('\n', " "))), + ]) + }) + .collect() +} + +// ── help ─────────────────────────────────────────────────────────────────── + +fn draw_help(frame: &mut Frame, area: Rect) { + let keys: &[(&str, &str)] = &[ + ("1 2 3 tab", "fleet · vms · events"), + ("↑↓ j k", "move"), + ("pgup pgdn g G", "page · top · bottom"), + ("←→ h l", "fold · unfold a node"), + ("z", "fold / unfold all"), + ("enter", "open detail"), + ("v", "preview pane"), + ("x", "start · shutdown · restart · stop"), + ("/", "filter by name, id, app, image, node"), + ("a", "scope: all · live · attention"), + ("s S I", "sort key · invert"), + ("r", "refresh now"), + ("p space", "pause"), + ("+ -", "refresh interval"), + ("in detail", "tab/1-5 switch · f follow · ←→ pan"), + ("q esc", "back · quit"), + ]; + let w = 56u16.min(area.width.saturating_sub(4)); + let h = (keys.len() as u16 + 4).min(area.height.saturating_sub(2)); + let rect = Rect { + x: area.x + (area.width.saturating_sub(w)) / 2, + y: area.y + (area.height.saturating_sub(h)) / 2, + width: w, + height: h, + }; + frame.render_widget(Clear, rect); + let block = Block::default() + .borders(Borders::ALL) + .border_type(BorderType::Rounded) + .border_style(fg(RULE)) + .title(Span::styled(" keys ", fg(MUTED))); + let inner = block.inner(rect); + frame.render_widget(block, rect); + let mut lines = vec![Line::from("")]; + lines.extend(keys.iter().map(|(k, v)| { + Line::from(vec![ + sp(fg(TEXT).add_modifier(Modifier::BOLD), format!(" {k:<16}")), + dim(v.to_string()), + ]) + })); + frame.render_widget(Paragraph::new(lines), inner); +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::snapshot::{NodeSnapshot, Resources}; + use ratatui::backend::TestBackend; + use ratatui::Terminal; + + fn sample() -> State { + let mut s = State::new(vec!["a".into(), "b".into()], 2000, false); + s.now_ms = 1_700_000_000_000; + let vm = |id: &str, name: &str, status: &str, boot: &str| Vm { + id: id.into(), + name: name.into(), + status: status.into(), + running: status == "running", + uptime_secs: Some(93_000), + boot_progress: boot.into(), + vcpu: 8, + memory_mb: 32768, + disk_gb: 100, + image: "dstack-0.5.3".into(), + app_id: "1a2b3c4d5e6f".into(), + ..Default::default() + }; + s.apply_nodes( + 0, + vec![NodeSnapshot { + name: "tdx-01".into(), + endpoint: "http://10.0.0.1:9080".into(), + ok_at_ms: Some(1), + latency_ms: Some(12), + version: Some("0.6.1".into()), + resources: Some(Resources { + max_cvms: 16, + max_vcpu: 64, + max_memory_mb: 262144, + }), + vms: vec![ + vm( + "11111111-aaaa", + "web-frontend-with-a-long-name", + "running", + "done", + ), + vm("22222222-bbbb", "worker", "running", "requesting app keys"), + vm("33333333-cccc", "名字很长的应用", "exited", ""), + ], + ..Default::default() + }], + true, + ); + s.apply_nodes( + 1, + vec![NodeSnapshot { + name: "tdx-02".into(), + endpoint: "http://10.0.0.2:9080".into(), + error: Some("connection refused".into()), + ..Default::default() + }], + true, + ); + s + } + + fn render(state: &mut State, w: u16, h: u16) -> Vec { + let mut term = Terminal::new(TestBackend::new(w, h)).unwrap(); + term.draw(|f| draw(f, state)).unwrap(); + let buf = term.backend().buffer().clone(); + (0..h) + .map(|y| { + (0..w) + .map(|x| buf[(x, y)].symbol().to_string()) + .collect::() + }) + .collect() + } + + #[test] + fn renders_at_every_breakpoint_without_panicking() { + for (w, h) in [(40, 10), (80, 24), (120, 30), (160, 40), (20, 5), (200, 60)] { + let mut s = sample(); + for tab in Tab::ALL { + s.set_tab(tab); + s.preview = true; + render(&mut s, w, h); + s.help = true; + render(&mut s, w, h); + s.help = false; + } + s.set_tab(Tab::Fleet); + s.move_to(1); + s.open_detail(); + for t in DetailTab::VM { + s.set_detail_tab(t); + render(&mut s, w, h); + } + } + } + + #[test] + fn fleet_view_shows_tree_and_selection() { + let mut s = sample(); + let screen = render(&mut s, 120, 30).join("\n"); + assert!(screen.contains("dstack")); + assert!(screen.contains("tdx-01")); + assert!(screen.contains("└ ")); + assert!(screen.contains("requesting app")); + assert!(screen.contains("unreachable")); + assert!(screen.contains("▌")); + // Standard width carries IMAGE but not PORTS. + assert!(screen.contains("IMAGE")); + assert!(!screen.contains("PORTS")); + } + + #[test] + fn columns_by_width() { + let (c, name) = columns(Tab::Fleet, 80); + assert_eq!( + c, + [Col::Name, Col::Status, Col::Uptime, Col::Vcpu, Col::Mem] + ); + assert!(name >= NAME_MIN); + let (c, _) = columns(Tab::Vms, 160); + assert!(c.contains(&Col::Ports) && c.contains(&Col::Node)); + let (c, _) = columns(Tab::Vms, 30); + assert!(c.len() >= 2); + } + + #[test] + fn columns_fill_the_width_exactly() { + for tab in [Tab::Fleet, Tab::Vms] { + for w in 30..=220usize { + let (cols, name) = columns(tab, w); + let fixed: usize = cols.iter().map(|c| c.width() + SEP).sum::() - SEP; + if name >= NAME_MIN { + assert_eq!(GUTTER + fixed + name, w, "{tab:?} {w}"); + } + } + } + assert_eq!( + fmt::width(&spans_text(&cell(vec![dim("名字很长的应用")], 7, false))), + 7 + ); + assert_eq!(fmt::width(&spans_text(&cell(vec![dim("7")], 5, true))), 5); + } + + fn spans_text(spans: &[Span]) -> String { + spans.iter().map(|s| s.content.as_ref()).collect() + } +}