Skip to content

feat(support): make the support file answer for the last week - #2020

Merged
frahlg merged 2 commits into
masterfrom
support-dump-week
Nov 15, 2026
Merged

frahlg merged 2 commits into
masterfrom
support-dump-week

Conversation

@frahlg

@frahlg frahlg commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Problem

A tester sent a support file at 12:54 on 15 November and asked why the battery
discharged overnight. The file could not say. logs/global.log covered
11:40–12:54, telemetry/ one hour and planner.json only the current plan. The
box still held the night in its own storage (/api/series, /api/history,
/api/control/journal, /api/mpc/diagnose/history), but the file did not take
it. The log ring filled in 75 minutes because three lines repeat every control
tick at INFO: two from pixii.lua (Pixii: setpoint ems_w=…,
Pixii: modbus_write_multi …) and Core's loadpoint fuse-clamp: below minimum step.

The redacted config also showed the owner's e-mail address twice
(drivers[].config.email and ev_charger.username), the site's exact
coordinates (weather.latitude, weather.longitude) and the ntfy topic,
which on the public server is the key that reads the owner's alerts.

What changes

Help report. A new section, "What happened over the last week", follows
Findings. It splits the last seven days at 06:00 and 22:00 household time, so
"last night" is one row, and gives per day and night: the battery's lowest and
highest charge, energy into and out of the battery, energy to the car (from
five-minute averages, so close rather than metered), grid import and export,
replans with their main reasons, the scored plan slots that missed and the
energy gap, time under the fuse limit, devices offline or in fault, warnings and
errors, and Core starts.

Bullets above the table say what stands out, so the section reads without the
table:

  • a period where the battery moved 1 kWh or more beyond what its plan slots
    asked for, in or out;
  • night discharge that did follow the plan, with the plan's figure, because it
    is what people ask about;
  • a device offline or in fault for 5 minutes or more, with the longest stretch;
  • time under the fuse limit;
  • plans that fell back to Core's own planner, with the last reason;
  • a period with an unusual number of replans;
  • a start with no stop before it (crash, power cut or killed process);
  • the week's most repeated warning;
  • missing history, and a week that starts late on a new box.

GET /api/support/report carries the section too, so agents that read it get
the week.

Archive. week/ holds seven days read from storage, not from memory:

File Source
week/series/<driver>__<metric>.csv every series, LoadSeriesBucketsOrRawContext as GET /api/series, five-minute buckets with min, max and count
week/control-journal.json ControlSpans, the spans GET /api/control/journal serves
week/plans.json the /api/mpc/diagnose/history summaries plus engine, status and fallback reason, read by json_extract so Go never holds the snapshots
week/notifications.json NotificationsBetween
week/events.json start, stop and restart events
week/summary.json the period table as data, a series index with units, and what could not be read
logs/warnings-week.log seven days of warnings and errors, one line per message pattern and hour

The last hour at stored resolution (telemetry/) and the log ring stay as they
were. manifest.json lists the new files and says what period each part
covers. The reads have a 60 s budget and name anything they leave out.

Logs for a week. Warnings and errors are now also counted per message and
hour, with numbers folded to # so the same warning with other figures counts
once. Core writes the counts every minute to a new state.db table,
log_digest, and prunes it after eight days. One hour keeps at most 200
patterns; the rest are counted in one row.

Why this rather than quieter logging: two of the three chatty lines come from
pixii.lua in device-drivers, and the next chatty driver would fill the ring
again. A ring of any size also empties on restart, and a beta box restarts at
every update; the hours before a crash are the ones a support file most needs.
Lowering Core's own fuse-clamp line would help the ring a little and is left as
a follow-up. The table uses CREATE TABLE IF NOT EXISTS, so an older Core
opens the same state.db and ignores it; no schema bump.

Redaction. email, e-mail, username and user_name are now sensitive
keys, and any e-mail address is removed by its shape from the config and the
logs, wherever it sits.

Coordinates are rounded to whole degrees, about 100 km: enough to show a
missing location or a wrong sign, not enough to find the house. That holds for
any key whose words include lat, lon, lng, latitude or longitude
(home_lat, latitude_deg; not latest_ms), in the config, in
planner.json, and in log text as lat=… in a URL or a JSON field in an HTTP
body a driver logs, such as a car's position.

notifications.ntfy.topic is hidden in the config. A failed publish quotes the
topic in its URL, so every file in the archive and GET /api/support/report
also replace the configured topic wherever it appears. The ntfy server stays
visible, so a reader can tell the public server from a private one. MQTT
topic keys in driver config stay too; they are wiring, not secrets.

CLI. ftw support waited 30 s; it now waits up to 2 minutes, Core's HTTP
write timeout, because the file reads a week of storage.

Size

Seeded week, measured with a throwaway test that called the handler:
5 drivers and 113 series (one-minute samples for seven days, five-second samples
for the last hour, values with noise so they compress worse than real ones),
6,000 journal spans with real ControlFeedback blobs, 1,050 plan snapshots of
about 45 kB, 60 notifications, 2,520 warning rows and a full log ring.

2.8 MB zipped, built in 1.8–2.0 s on an M-series Mac (measured again after
the coordinate rule, which scans the journal once more).

Part Zipped Unzipped
week/series/ 2.10 MB 7.4 MB
telemetry/ (last hour) 0.35 MB 1.6 MB
week/control-journal.json 0.22 MB 11.8 MB
logs/ 0.06 MB 0.9 MB
everything else 0.01 MB 0.3 MB

Not measured on a Raspberry Pi. Expect it to take several times longer; the 60 s
read budget and the 2 minute write timeout leave room.

Evidence

  • Tests first:
    • TestSupportDumpAnswersForTheLastWeek seeds a night where the battery
      discharged 8 kWh while every slot asked it to hold, a charger offline for
      two hours, a fuse limit, a fallback plan, a repeated warning and a restart.
      The report must name each, and the archive must hold each week/ file.
    • TestSupportWeekOnANewBox keeps a new box's first start from reading as a
      crash and folds the empty days into one row.
    • TestWeekFindingsNightDischargeAndCrash, TestSupportWeekPeriods
      (boundaries and the autumn clock change).
    • TestSupportDumpRemovesTheOwnersEmail,
      TestSupportDumpHidesWhereTheHouseIsAndTheNtfyTopic (config, a forecast
      URL, a failed ntfy publish, a car position in a driver log, and the
      report), TestIsCoordinateKey.
    • TestLogDigestAddsUpRepeatsWithinAnHour, TestPlannerReplansReadSolverFields.
    • TestLogRingDigestKeepsWarningsPastTickChatter,
      TestLogRingDigestCapsDistinctMessagesPerHour, TestLogPatternFoldsNumbers.
  • make verify passes locally.
  • Local rig: sim-ferroamp and sim-sungrow with this branch's Core, restarted
    twice. logs/warnings-week.log counted the same two warnings across both
    restarts (×2 first 13:39:12 last 13:42:16), the journal and events reached
    week/, and the clean restart raised no crash warning. Two faults found this
    way are fixed here: a new box's first start read as a crash, and the time of
    the first reading was a bucket's last sample.
  • Not done: the diagnostics modal's new progress text was not rendered; it shows
    only while a download runs. Not measured on a Pi.

For review

Follow-ups, not in this PR

  • manifest.json carries the box's hostname, which an owner may have named
    after themselves.
  • Core's loadpoint fuse-clamp line logs every tick at INFO; logging only when
    the clamp changes would let the ring cover more than an hour.
  • pixii.lua logs every command at INFO (device-drivers).

🤖 Generated with Claude Code

A support file sent at 12:54 could not say why the battery discharged
overnight: its log covered 75 minutes and its telemetry one hour. The
box held the night in storage; the file did not take it.

The help report gains "What happened over the last week", per day
(06-22) and night (22-06): battery charge range, battery, car and grid
energy, replans and why, plan slots missed and the energy gap, fuse
time, devices out of contact, fallback plans, warnings and Core starts,
with what stands out stated first.

The archive carries week/: every series in five-minute buckets, the
control journal, plan snapshots with solver fields, notifications and
start/stop events. Warnings and errors are counted per message and hour
in a new state.db table, log_digest, so a week of them outlasts the log
ring and a restart. A seeded week of 113 series gave a 2.8 MB archive.

The redacted config and logs no longer show the owner's e-mail address
or account username. `ftw support` waits up to two minutes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T13:51:26.084847Z c8846da PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8846da8f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +549 to +551
if p.SlotsScored > 0 {
var parts []string
if d := p.ActualOutWh - p.PlannedOutWh; math.Abs(d) >= supportPlanGapWh {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for unscored hold slots before judging plan adherence

When the battery discharges during slots that asked it to hold, this comparison cannot detect the discrepancy: scoreSlotDeliveryMetrics in go/internal/control/dispatch.go returns without recording a slot whenever the planned energy is within slotDeliveryIdleWhCutoff. Consequently, the zero-planned-energy spans seeded by TestSupportDumpAnswersForTheLastWeek cannot occur in production. An unexpected overnight discharge is reported only as a night-discharge note; if other slots were scored successfully, the note can even claim that the battery stayed within 1 kWh of its plan while ignoring the unscored discharge. Include hold-slot evidence in the comparison, or explicitly report incomplete plan coverage instead of asserting adherence.

Useful? React with 👍 / 👎.

Comment on lines +356 to +357
if sp.Reason == "fuse_limit" || sp.Reason == "fuse_cooldown" {
out.FuseLimitedS += overlap

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Merge overlapping fuse-limit intervals before summing duration

On a site with multiple chargers limited by the fuse simultaneously, this adds the same elapsed time once per charger. JournalVerdicts records independent device-function subjects in go/internal/api/api_control_journal.go, so two chargers limited for one hour produce two hours of FuseLimitedS. The report presents this as elapsed time during which the main fuse held power back, and the total can exceed the period's duration. Union the overlapping fuse-limit intervals before calculating the site-wide duration, or report separate per-device durations.

Useful? React with 👍 / 👎.

Round coordinates to whole degrees in the redacted config, planner.json
and log text (URL query fields and JSON numbers in logged bodies), and
hide notifications.ntfy.topic. A failed publish quotes the topic in its
URL, so every file in the support archive and GET /api/support/report
also replace the configured topic wherever it appears.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@frahlg
frahlg merged commit 59173a0 into master Nov 15, 2026
29 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant