Repository navigation
feat(os): collect the operating system status - #1214
karlitschek wants to merge 23 commits into
Conversation
AndyScherzinger
left a comment
There was a problem hiding this comment.
I reviewed this for correctness, security and performance, and deployed the branch to a nextcloud-docker-dev instance (NC 36 dev, PHP 8.5, Debian 13 container) to exercise it. Inline comments carry the details; this is the summary.
Blocking
- CI "NPM build" fails: the committed
js/serverinfo-main.mjs.licenseandjs/serverinfo-main.mjs.map.licenselist@vitejs/plugin-vue 6.0.10, whilepackage-lock.jsonpins6.0.7. They were generated from anode_modulesout of sync with the lock file;npm ci && npm run buildregenerates them correctly. The version bump itself is needed for the background job and command to register. - Missing or stale package lists turn into "no security updates" (reproduced): with the image's purged
/var/lib/apt/lists,apt-get -s dist-upgradereported 0 pending updates; afterapt-get updatethe same container reported 71 pending, 35 of them security. The updates concern only becomes unknown whenupdates === null, so the headline answer is a false all-clear wherever the lists are missing or old (seeOsVerdict.php). occ serverinfo:os-statusexit codes do not follow the monitoring-plugin convention it cites: "not checked yet" exits 1 (= WARNING), and an all-unknown verdict reports"level": "ok"and exits 0. In the Docker container 3 of 4 concerns were unknown and the command said OK. Monitoring plugins use 3 for UNKNOWN.
Correctness
- The first-seen key includes the candidate version, so a superseding security update resets the pending-days counter.
CoreLibraryProbetreats an upgraded systemd binary as a reboot reason, though systemd re-executes itself on upgrade; on Debian, which has no/run/reboot-requiredby default, that is a recurring false positive.- In LXC without lxcfs,
/proc/uptimeis the host's, so a container created after the host booted reports a core-library reboot straight away. - Smaller:
Thresholds::load()does not enforce warning ≤ error;nixos-unstableis not an os-releaseID; Raspbian is mapped to Debian's support dates though it has its own archive;/dev/shm/is shadowed by/dev/. - Worth stating in the UI later: Debian ships many fixes through point releases (
libc6 2.41-12+deb13u4fromDebian:13.7/stablein my run), so "0 security updates" does not mean fully patched.
Security
CommandRunneris well done: an argument vector toproc_open()with no shell, validated binary names, a fixedPATH,LC_ALL=C, stdin from/dev/null, an output cap and a kill on timeout. All arguments are constants, so I found no injection surface, andrestricted_modeis honoured.- The risk is exposure rather than execution. In this PR the data is only visible to admins through occ, but a list of unpatched packages with exact versions is an attacker's roadmap. Once #1217 exposes this through the token-authenticated monitoring API, consider returning counts there and keeping the package list admin-only.
- Package names and origins are stored as parsed from tool output (
\S+). Nothing here renders them, but #1218 should not pass them throughv-html. - The collector reads
/proc/1/cgroup,/sys/class/dmi/id/*and queries systemd over D-Bus. Everything degrades to unknown when denied, which is right; the README could list what is read so hardened setups know what to expect.
Performance
- From system cron this is cheap. A full refresh took 0.75 s in the container; with 71 pending updates,
apt-get -s dist-upgradetook 2.3 s and the remaining apt probes 0.4 s. - Under AJAX or webcron background-job modes the job runs inside a web request. The timeouts add up to roughly four minutes in the worst case (120 s for apt-get, 3 × 30 s, plus the 10 s and 5 s probes), all holding one PHP-FPM worker. Consider skipping the job unless
backgroundjobs_modeiscron, or shortening the timeouts there. - The status is stored as a lazy app-config array capped at 500 updates, which fits the long-text column on every supported database.
b2085c1 to
2e8c497
Compare
|
@AndyScherzinger Thanks for the review. All should be fixed now |
2e8c497 to
eb88b3f
Compare
eb88b3f to
60f5960
Compare
AndyScherzinger
left a comment
There was a problem hiding this comment.
Re-review of 60f59603. I deployed the whole stack (up to #1218) to the same nextcloud-docker-dev instance as before (NC 36 dev, PHP 8.5, Debian 13 container, background jobs from system cron) and re-ran the checks.
Verified fixed
- Missing package lists now make the pending updates unknown instead of an all-clear.
occ serverinfo:os-statusexits 3 when nothing has been collected yet (live:exit=3), and an unknown verdict also maps to 3.- First-seen times are tracked per package, so a superseding update keeps counting.
- systemd is no longer a core-library reboot reason. A changed file now also needs a package change since boot, and system containers skip the probe.
- Error thresholds are clamped to their warning thresholds.
- Collection runs only from system cron, and a new freshness concern warns when the status gets old or cron is not system cron.
- The status moved to app data. Live:
status.jsonis about 1.2 KB, and the legacycached_os_statusapp config key is gone. - The license files match the lock file again, and CI is green.
Remaining
One finding is left, inline: in an application container the command still reports ok and exits 0, although the container's own packages have pending security updates.
| $known = true; | ||
| } | ||
|
|
||
| if (Concern::rank($level) <= Concern::rank(Concern::LEVEL_INFO) && $this->get(Concern::UPDATES)?->level === Concern::LEVEL_UNKNOWN) { |
There was a problem hiding this comment.
This only turns the level unknown when an UPDATES concern exists and is unknown. In an application container OsVerdict never creates that concern (managesPackages() is false), so the result is ok and exit 0.
Live in the Docker container: occ serverinfo:os-status --refresh gives "level": "ok", "updates": "unknown" and exit 0. Earlier, apt-get -s dist-upgrade in that same container listed 71 pending updates, 35 of them from Debian-Security.
It is a fair design choice that the image's packages are not the admin's to patch in place. But ok/0 is still a claim about them, and monitoring will read it as healthy. I suggest one of:
- Emit the UPDATES concern as unknown (or a dedicated not-assessed state) in application containers, so that
level()and the exit code say unknown. - State explicitly in the output and the README that
okcovers host-side concerns only.
The same root cause makes the #1217 dashboard widget say "The system is up to date" in this container.
Turning off the update part
Multiple application serversSetups with two or more application servers are common. The status, the cache key and the first-seen times are currently shared, so the status shows whichever node last ran the job, and the thresholds reset when nodes disagree.
It would be good to settle this before #1217 and #1218 land, since it changes the storage format. Nitpicks
|
Collect what keeps the Linux distribution under Nextcloud secure, as far as the web server user can see it: pending updates with security fixes told apart (apt), reboots that installed updates wait for (the running kernel against /lib/modules, core libraries replaced since boot, the distribution's own flag), libraries replaced under the PHP serving Nextcloud, how long the release still receives security updates (bundled from endoflife.date by build/update-os-support.php, nothing fetched at runtime), automatic updates, held packages, failed system services, time synchronisation and mandatory access control. Containers, LXC guests and snaps are told apart, so nothing is claimed that cannot be known from inside. Commands run through proc_open() with an argument vector, a fixed search path, LC_ALL=C, a time limit and an output cap. A background job collects the status hourly. One verdict with configurable thresholds judges it, and occ serverinfo:os-status shows it, exiting 0/1/2 for monitoring checks. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
Without package lists apt-get -s finds no newer versions, which read as "no security updates". When the lists cannot be found and nothing is pending, the updates are now unknown, with a diagnostic. Updates found in lists past the warning age carry a listsStale fact, and occ reports the security update count as unknown when the updates are. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
occ serverinfo:os-status follows the monitoring-plugin exit codes, where 3 is UNKNOWN. A status that was never collected, could not be collected or has no verdict now exits 3 instead of 1 or 0. The overall level is unknown when nothing is wrong but the pending updates are unknown, so an unknown update state no longer reads as an all-clear. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
A security update superseded by a newer one before it was installed started a new first-seen entry, so the pending days, and with them the thresholds, started over. Hosts with frequently updated packages could stay below the error threshold indefinitely. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
systemd's package re-executes PID 1 on upgrade, so a newer systemd binary needs no reboot; on Debian, which has no /run/reboot-required by default, every systemd update became a reboot warning. In a system container without lxcfs the uptime is the host's, so a container created after the host booted reported a reboot straight away; the probe now skips system containers. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
With an error threshold below its warning threshold, a finding went straight from info to error and the warning stage disappeared. Loading the thresholds now raises each error value to its warning value. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
The commands can take minutes in the worst case. Under AJAX or webcron the background job runs inside a web request and would hold a web server worker that long, so it now does nothing unless background jobs run from system cron; occ serverinfo:os-status --refresh still works everywhere. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
Raspbian is a separate archive with its own security handling, so Debian's end dates do not necessarily apply; its support is unknown now. NixOS has ID=nixos in every channel, so "nixos-unstable" never matched and is gone from the rolling releases. /dev/shm/ was already covered by /dev/ among the ignored PHP mappings. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
…uirement Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
The license files listed @vitejs/plugin-vue 6.0.10 and the lock file had follow-redirects downgraded to 1.16.0, both from a node_modules out of sync with the lock file. The lock file is master's again with only the version bump, and npm ci && npm run build regenerated the license files. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
When stream_select() failed, for example because a signal interrupted it, the runner left its read loop as if the command had closed its output. The process was not killed, the result was neither timed out nor truncated, and a cut-off apt-get simulation counted as a complete list of pending updates. A failed wait is now retried, and after three failures in a row the process is killed and the result is marked as unreadable, which succeeded() treats as a failure like a timeout or truncation. CommandResult::failure() names what went wrong: the time limit, the output cap, the failed read, or else the first line the command printed. apt's diagnostic uses it, so a timeout no longer shows whatever line apt had printed before it was killed. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
…aries
Under open_basedir the system directories cannot be inspected, so
hasBinary('apt-get') was false and apt was never found, although
processes can still be started from those directories. Each later
lookup of a missing binary also tried every search directory again, up
to six forks per call.
IHost::binaryState() answers true, false, or null when only running the
binary can tell. hasBinary() keeps meaning "known to be there". apt is
available unless it is known to be missing, so the factory picks it and
running it gives the answer.
CommandRunner remembers per binary the path it ran from, or that it is
missing once every guess failed to start, so a missing binary costs its
forks once per process.
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Frank Karlitschek <[email protected]>
…tial apt-get update writes into lists/partial on every run, also when it fails. With an expired key on the security mirror every refresh failed, yet the lists counted as refreshed today and no security updates were pending. The age now comes from update-success-stamp, which a hook touches only after a successful update. Where that hook is not installed, as on Debian 12 and a plain Ubuntu 26.04 server, the newest InRelease (or Release) file is used: apt moves it into place only after verifying it, with the mirror's modification time, so the lists were refreshed then at the earliest. The new fixture is the www-data view of /var/lib/apt/lists on an up-to-date Debian 12 container, from the same capture run as the other Debian 12 fixtures. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
KernelProbe reported "kernel removed" whenever the running kernel had no directory in /lib/modules. A VPS that boots the provider's kernel, or a netbooted machine, never has one, so it got a reboot warning that a reboot could not clear and that became an error after 30 days. A missing running kernel is now a reason only when an installed kernel has the same flavour, which is the Arch case of modules deleted on upgrade, or when the newest installed kernel was installed after the boot. A distribution kernel upgraded on such a provider host then asks for a reboot until the next boot, and is quiet after it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
…boot CoreLibraryProbe took a libc or dbus inode change time after boot as an upgrade. A chmod or chown of the file, or a restore from a backup, also sets the change time, and gave a reboot reason that only a reboot cleared. The modification time is no alternative: dpkg keeps the build time there. A changed file now also needs its package to have changed after boot. dpkg rewrites /var/lib/dpkg/info/<package>.list on every install of a package, so its mtime says when libc6 or the dbus package was last installed. Without such a list, the package manager's database file must have changed after boot. Only where neither exists does the change time decide alone. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
savePhpSample() in web requests and firstSeen() in the background job both read, changed and wrote cached_os_first_seen without a lock. When they overlapped, one write dropped the other's entries, the "days pending" count of updates and reboots started over, and warnings came later than the thresholds say. When web requests first saw outdated PHP libraries is now kept in cached_os_php_outdated_since, so cached_os_first_seen has a single writer, the collector. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
With up to 500 listed updates the stored status comes to 100-200 KB of JSON. As a lazy app config value it was loaded and cached together with every app's lazy values, so any request that read a lazy key of any app carried it. The status is now os-status/status.json in the app's data. OsStatusStore keeps its method names. Reads come from admin pages and setup checks, and app data can be on object storage, so a distributed cache entry sits in front of the file: a save replaces it, and a read that finds no file remembers that too. Within a request the status is read once. A save removes the old cached_os_status app config value, which has left the lexicon. The small values stay in app config. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
…ding the run Nothing compared the time of collection with the job interval. When cron died, or the admin switched to AJAX or webcron, the job stopped collecting without a word, and the last status stayed green for weeks: on the page, in the setup checks and in occ serverinfo:os-status, whose exit code 0 kept monitoring quiet. In the collector only the pending updates were guarded, so an exception from any other probe dropped the whole run. The verdict has a freshness concern. It is a warning when the status is older than three job intervals, and never sooner than after six hours, or when background jobs do not run from system cron. It is unknown when nothing was collected. Its facts are collectedAt, ageSeconds and systemCron. A warning is a warning in the overall level as well, so an old status with unknown updates is no longer just "unknown". OsStatusService reads the cron mode and the interval the way the job does, so the verdict stays a function of its inputs. The occ command shows the age of the status, and its exit code follows the verdict. Every probe of the collector now runs on its own: one that throws is logged, costs its own finding and leaves a diagnostic. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
SupportTable took an optional ?string $supportDataFile in its constructor for tests. The container autowires constructor parameters by name, so a registered service or parameter called supportDataFile would have replaced the bundled table without anyone noticing. The constructor has no parameters now. Tests use SupportTable::fromFile() for a different table. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
The README promised that the PHP serving Nextcloud is checked for libraries an update replaced, and named RHEL ELS among the paid support extensions. Here nothing samples the PHP process yet, so that check always reads as unknown, and only Ubuntu Pro and Debian ELTS are detected. Both claims and the mention of reading the process maps are gone; the later parts of the stack add them back with the code. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
The name, versions and archives of a pending update were taken from the "Inst" lines of `apt-get -s` as they came and kept in the stored status. Anything a repository puts there, such as markup or very long strings, ended up in the status the page, the API and the notifications show. A line is now only kept when its package name matches the pattern the held packages and the reboot list already use, now one shared constant, when both versions are at most 128 characters of what Debian versions are made of (letters, digits and ".+~:_-"), and when every archive is at most 128 characters of letters, digits, spaces and ".+~:_/-". Spaces are allowed there because real archive labels carry them, e.g. "Debian Backports". Lines that fail are left out. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
…rchives An apt update counted as a security update whenever one of its archives was a suite ending in "-security". Any third-party repository could get its updates counted that way just by naming its suite like that, and so change what the status reports and how urgent it looks. The archive's label, the part before the colon in "Debian-Security:12/ stable-security", now also has to name the distribution: the os-release ID or an ID_LIKE entry, optionally followed by "-Security" or by Ubuntu Pro's "ESM" and "ESMApps". Derivatives such as Linux Mint keep their Ubuntu security updates, and Devuan its own. Apt gets the release from the factory through withRelease(); without one it does not claim to tell security updates apart. Labels are set by each repository, so this stops repositories that merely call a suite "*-security", not one that copies the distribution's label. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
New occ commands use the OCP console attributes instead of extending the private OC\Core\Command\Base. The command is now a plain class with #[AsCommand] and an #[Option] for --refresh; --output comes from supportsOutputFormat, so the options and the output stay the same. It still returns the monitoring plugin exit codes 0 to 3 as an int: ExitCode has no case for 3, "unknown", which Nagios-style checks need. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Frank Karlitschek <[email protected]>
60f5960 to
934a33d
Compare
Part 1 of 5, stacked on master:
AI disclosure: this change was written with Claude Code (claude-opus-5-5) and reviewed by the author; every commit carries an
Assisted-bytrailer.🤖 Generated with Claude Code