Skip to content

feat(jobs): show the background job queue health, failures by job, and two admin actions - #1237

Open
karlitschek wants to merge 3 commits into
masterfrom
feat/job-queue-health
Open

karlitschek wants to merge 3 commits into
masterfrom
feat/job-queue-health

Conversation

@karlitschek

Copy link
Copy Markdown
Member

Based on feat/os-status-collector (#1214).

AI disclosure: this change was written with Claude Code (claude-opus-5-5) and is to be reviewed by the author before it leaves draft; every commit carries an Assisted-by trailer.

馃 Generated with Claude Code

@karlitschek
karlitschek force-pushed the feat/os-status-collector branch from 6d327c2 to 34014ec Compare October 25, 2026 19:19
Base automatically changed from feat/os-status-collector to master October 25, 2026 19:24
@karlitschek
karlitschek force-pushed the feat/job-queue-health branch from 134f737 to 7b7b845 Compare October 25, 2026 19:28
@nextcloud-command nextcloud-command added the AI assisted This PR contains AI-assisted commits label Oct 25, 2026
karlitschek and others added 3 commits October 25, 2026 20:45
JobQueueInfo computed the queue size, reserved and stuck jobs and the
oldest last run, but nothing called it. GET /jobs now serves it for the
Background jobs page, together with the failed runs grouped by job
class, so that one job failing every few minutes no longer pushes every
other failure out of view.

The jobs table has indexes on class and on (last_checked, reserved_at),
none on last_run. The four totals were four queries, one of them a full
table scan for MIN(last_run); they are now one pass with conditional
sums. The result, the five largest classes and the 20 longest reserved
jobs are cached for 60 seconds, and the page asks only when the tab is
shown. The failures read job_runs through its status index.

Two actions change a single job. Both are admin-only, need the password
confirmed, are rate-limited and logged at info level with the admin,
the job id and the class, never the argument. Neither runs job code:

- Release reservation clears reserved_at of a job reserved for more
  than 12 hours, as IJobList::unlockJob() does. Cron already takes such
  a job again; releasing removes the stale reservation from statistics
  and from the OpenMetrics running jobs count.
- Run at next cron clears last_run and reserved_at, as
  IJobList::resetBackgroundJob() does, and also last_checked: cron
  moves that up to two days ahead for a timed job that is not due, so
  resetting the last run alone would not make the job run any sooner.
  A job reserved within the last 12 hours may be running and is
  refused.

The updates are written here instead of calling IJobList because both
OCP methods take an IJob, getById() builds the job and deletes it when
its class no longer exists, and their update is unconditional. These
apply only while reserved_at still holds the value the admin saw, so a
job that cron took in the meantime is left alone.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Frank Karlitschek <[email protected]>
The Background jobs tab showed the cron mode, its last run, the last 10
failed runs, the latest runs and the slowest jobs. It now also shows the
queue as tiles (queued, reserved now, stuck, oldest last run), the jobs
cron has reserved, failures grouped by job with their count and latest
failure, and the job kinds with the most queued jobs.

The queue comes from GET /jobs, which scans the jobs table, so the tab
reads it when it becomes visible and again with the minute poll while it
stays visible, never while another tab is shown. The last 10 failed runs
are no longer read with every poll; the grouped failures replace them.

A stuck job offers "Release reservation" and "Run at next cron", and a
failing job whose class has exactly one queued job offers "Run at next
cron": with several jobs of a class it would not be clear which one is
meant. Each action explains itself in a dialog, asks for the password,
reports the result in a toast and reads the queue again, also after a
refusal, because the job usually changed in the meantime.

The toasts come from @nextcloud/dialogs. The package does not
tree-shake and would add about 290 KB to the page, so it is loaded in
its own chunk when the first toast is shown.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Frank Karlitschek <[email protected]>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Frank Karlitschek <[email protected]>
@karlitschek
karlitschek force-pushed the feat/job-queue-health branch from 7b7b845 to c15cf94 Compare October 25, 2026 19:45
@karlitschek
karlitschek marked this pull request as ready for review October 25, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI assisted This PR contains AI-assisted commits

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants