Repository navigation
feat(support): make the support file answer for the last week - #2020
Conversation
A support file sent at 12:54 could not say why the battery discharged overnight: its log covered 75 minutes and its telemetry one hour. The box held the night in storage; the file did not take it. The help report gains "What happened over the last week", per day (06-22) and night (22-06): battery charge range, battery, car and grid energy, replans and why, plan slots missed and the energy gap, fuse time, devices out of contact, fallback plans, warnings and Core starts, with what stands out stated first. The archive carries week/: every series in five-minute buckets, the control journal, plan snapshots with solver fields, notifications and start/stop events. Warnings and errors are counted per message and hour in a new state.db table, log_digest, so a week of them outlasts the log ring and a restart. A seeded week of 113 series gave a 2.8 MB archive. The redacted config and logs no longer show the owner's e-mail address or account username. `ftw support` waits up to two minutes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c8846da8f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if p.SlotsScored > 0 { | ||
| var parts []string | ||
| if d := p.ActualOutWh - p.PlannedOutWh; math.Abs(d) >= supportPlanGapWh { |
There was a problem hiding this comment.
Account for unscored hold slots before judging plan adherence
When the battery discharges during slots that asked it to hold, this comparison cannot detect the discrepancy: scoreSlotDeliveryMetrics in go/internal/control/dispatch.go returns without recording a slot whenever the planned energy is within slotDeliveryIdleWhCutoff. Consequently, the zero-planned-energy spans seeded by TestSupportDumpAnswersForTheLastWeek cannot occur in production. An unexpected overnight discharge is reported only as a night-discharge note; if other slots were scored successfully, the note can even claim that the battery stayed within 1 kWh of its plan while ignoring the unscored discharge. Include hold-slot evidence in the comparison, or explicitly report incomplete plan coverage instead of asserting adherence.
Useful? React with 👍 / 👎.
| if sp.Reason == "fuse_limit" || sp.Reason == "fuse_cooldown" { | ||
| out.FuseLimitedS += overlap |
There was a problem hiding this comment.
Merge overlapping fuse-limit intervals before summing duration
On a site with multiple chargers limited by the fuse simultaneously, this adds the same elapsed time once per charger. JournalVerdicts records independent device-function subjects in go/internal/api/api_control_journal.go, so two chargers limited for one hour produce two hours of FuseLimitedS. The report presents this as elapsed time during which the main fuse held power back, and the total can exceed the period's duration. Union the overlapping fuse-limit intervals before calculating the site-wide duration, or report separate per-device durations.
Useful? React with 👍 / 👎.
Round coordinates to whole degrees in the redacted config, planner.json and log text (URL query fields and JSON numbers in logged bodies), and hide notifications.ntfy.topic. A failed publish quotes the topic in its URL, so every file in the support archive and GET /api/support/report also replace the configured topic wherever it appears. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Problem
A tester sent a support file at 12:54 on 15 November and asked why the battery
discharged overnight. The file could not say.
logs/global.logcovered11:40–12:54,
telemetry/one hour andplanner.jsononly the current plan. Thebox still held the night in its own storage (
/api/series,/api/history,/api/control/journal,/api/mpc/diagnose/history), but the file did not takeit. The log ring filled in 75 minutes because three lines repeat every control
tick at INFO: two from
pixii.lua(Pixii: setpoint ems_w=…,Pixii: modbus_write_multi …) and Core'sloadpoint fuse-clamp: below minimum step.The redacted config also showed the owner's e-mail address twice
(
drivers[].config.emailandev_charger.username), the site's exactcoordinates (
weather.latitude,weather.longitude) and the ntfy topic,which on the public server is the key that reads the owner's alerts.
What changes
Help report. A new section, "What happened over the last week", follows
Findings. It splits the last seven days at 06:00 and 22:00 household time, so"last night" is one row, and gives per day and night: the battery's lowest and
highest charge, energy into and out of the battery, energy to the car (from
five-minute averages, so close rather than metered), grid import and export,
replans with their main reasons, the scored plan slots that missed and the
energy gap, time under the fuse limit, devices offline or in fault, warnings and
errors, and Core starts.
Bullets above the table say what stands out, so the section reads without the
table:
asked for, in or out;
is what people ask about;
GET /api/support/reportcarries the section too, so agents that read it getthe week.
Archive.
week/holds seven days read from storage, not from memory:week/series/<driver>__<metric>.csvLoadSeriesBucketsOrRawContextasGET /api/series, five-minute buckets with min, max and countweek/control-journal.jsonControlSpans, the spansGET /api/control/journalservesweek/plans.json/api/mpc/diagnose/historysummaries plus engine, status and fallback reason, read byjson_extractso Go never holds the snapshotsweek/notifications.jsonNotificationsBetweenweek/events.jsonweek/summary.jsonlogs/warnings-week.logThe last hour at stored resolution (
telemetry/) and the log ring stay as theywere.
manifest.jsonlists the new files and says what period each partcovers. The reads have a 60 s budget and name anything they leave out.
Logs for a week. Warnings and errors are now also counted per message and
hour, with numbers folded to
#so the same warning with other figures countsonce. Core writes the counts every minute to a new state.db table,
log_digest, and prunes it after eight days. One hour keeps at most 200patterns; the rest are counted in one row.
Why this rather than quieter logging: two of the three chatty lines come from
pixii.luain device-drivers, and the next chatty driver would fill the ringagain. A ring of any size also empties on restart, and a beta box restarts at
every update; the hours before a crash are the ones a support file most needs.
Lowering Core's own fuse-clamp line would help the ring a little and is left as
a follow-up. The table uses
CREATE TABLE IF NOT EXISTS, so an older Coreopens the same state.db and ignores it; no schema bump.
Redaction.
email,e-mail,usernameanduser_nameare now sensitivekeys, and any e-mail address is removed by its shape from the config and the
logs, wherever it sits.
Coordinates are rounded to whole degrees, about 100 km: enough to show a
missing location or a wrong sign, not enough to find the house. That holds for
any key whose words include
lat,lon,lng,latitudeorlongitude(
home_lat,latitude_deg; notlatest_ms), in the config, inplanner.json, and in log text aslat=…in a URL or a JSON field in an HTTPbody a driver logs, such as a car's position.
notifications.ntfy.topicis hidden in the config. A failed publish quotes thetopic in its URL, so every file in the archive and
GET /api/support/reportalso replace the configured topic wherever it appears. The ntfy
serverstaysvisible, so a reader can tell the public server from a private one. MQTT
topickeys in driver config stay too; they are wiring, not secrets.CLI.
ftw supportwaited 30 s; it now waits up to 2 minutes, Core's HTTPwrite timeout, because the file reads a week of storage.
Size
Seeded week, measured with a throwaway test that called the handler:
5 drivers and 113 series (one-minute samples for seven days, five-second samples
for the last hour, values with noise so they compress worse than real ones),
6,000 journal spans with real
ControlFeedbackblobs, 1,050 plan snapshots ofabout 45 kB, 60 notifications, 2,520 warning rows and a full log ring.
2.8 MB zipped, built in 1.8–2.0 s on an M-series Mac (measured again after
the coordinate rule, which scans the journal once more).
week/series/telemetry/(last hour)week/control-journal.jsonlogs/Not measured on a Raspberry Pi. Expect it to take several times longer; the 60 s
read budget and the 2 minute write timeout leave room.
Evidence
TestSupportDumpAnswersForTheLastWeekseeds a night where the batterydischarged 8 kWh while every slot asked it to hold, a charger offline for
two hours, a fuse limit, a fallback plan, a repeated warning and a restart.
The report must name each, and the archive must hold each
week/file.TestSupportWeekOnANewBoxkeeps a new box's first start from reading as acrash and folds the empty days into one row.
TestWeekFindingsNightDischargeAndCrash,TestSupportWeekPeriods(boundaries and the autumn clock change).
TestSupportDumpRemovesTheOwnersEmail,TestSupportDumpHidesWhereTheHouseIsAndTheNtfyTopic(config, a forecastURL, a failed ntfy publish, a car position in a driver log, and the
report),
TestIsCoordinateKey.TestLogDigestAddsUpRepeatsWithinAnHour,TestPlannerReplansReadSolverFields.TestLogRingDigestKeepsWarningsPastTickChatter,TestLogRingDigestCapsDistinctMessagesPerHour,TestLogPatternFoldsNumbers.make verifypasses locally.sim-ferroampandsim-sungrowwith this branch's Core, restartedtwice.
logs/warnings-week.logcounted the same two warnings across bothrestarts (
×2 first 13:39:12 last 13:42:16), the journal and events reachedweek/, and the clean restart raised no crash warning. Two faults found thisway are fixed here: a new box's first start read as a crash, and the time of
the first reading was a bucket's last sample.
only while a download runs. Not measured on a Pi.
For review
TestIsSensitiveKeylistedusernameamong keys to keep visible. It camefrom fix(api): redact secrets in support-dump logs #1168 as a guard against matches that are too broad, with no triage use.
This PR moves it to the sensitive list on purpose;
userstays visible.minor, a visible new capability, so it needs the owner'sapproval.
api_drivers_debug.goin the driver-detailcode only. This PR changes
handleSupportDumpand redaction in the same fileand puts the new code in
api_support_week.go.Follow-ups, not in this PR
manifest.jsoncarries the box'shostname, which an owner may have namedafter themselves.
loadpoint fuse-clampline logs every tick at INFO; logging only whenthe clamp changes would let the ring cover more than an hour.
pixii.lualogs every command at INFO (device-drivers).🤖 Generated with Claude Code