diff --git a/.changeset/anonymous-daily-energy.md b/.changeset/anonymous-daily-energy.md new file mode 100644 index 000000000..b97cbb6a7 --- /dev/null +++ b/.changeset/anonymous-daily-energy.md @@ -0,0 +1,7 @@ +--- +"ftw": minor +--- + +Statistics are one answer on the box, for two anonymous daily reports. With FTW Cloud they are part of Cloud. Without Cloud they are on until you turn them off under Settings → FTW app → Anonymous statistics, which shows both messages; nobody is asked any more. A No, a fleet report switched off, or Cloud turned off on an earlier release stays off. + +Beside the fleet report, the box can now send one energy report a day: the date and that UTC day's solar generation, grid import and export, battery charge and discharge and vehicle charge, in whole kWh, never in the same message as the fleet report and at its own time. It skips any day with a gap in the energy record and any site a partner controls. Once a release names an Oblivious HTTP relay, both reports go through it, so FTW Cloud never sees the box's address. This release names none yet: the energy report is not sent, a box that said yes sends the fleet report straight to FTW Cloud as before, and a box that shares only by default sends nothing until a relay is named. diff --git a/.changeset/bundle-easee-cloud-138.md b/.changeset/bundle-easee-cloud-138.md new file mode 100644 index 000000000..d124e5f67 --- /dev/null +++ b/.changeset/bundle-easee-cloud-138.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +An Easee Cloud charger now starts a 6 A charge at once. Easee's firmware waits about five minutes before it starts a car at 6 A. The bundled `easee_cloud` driver (1.3.8) offers 7 A per phase for that start, and only when the per-phase cap that FTW sends with each order (`cap_amps_per_phase`) allows 7 A. It steps down to 6 A as soon as the charger reports charging, or after 90 s. A charger on a phase without room for 7 A starts at 6 A as before. `start_boost: false` in the driver config turns the 7 A start off. diff --git a/.changeset/cloud-history-tiles.md b/.changeset/cloud-history-tiles.md new file mode 100644 index 000000000..7c566cc3d --- /dev/null +++ b/.changeset/cloud-history-tiles.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +With FTW Cloud on and history in the owner's grant, the box sends FTW Cloud its last 48 hours of 5-minute history, so the iPhone app can draw a node's history while the box is offline. Each closed 12-hour tile is the same bytes the app gets from the box directly. The box adds each 5-minute bucket 30 seconds after it ends, and sends the tiles FTW Cloud lacks when it reconnects. It sends no history without NTP time, and none past the daily byte budget. The extra traffic is about 100 KB a day. diff --git a/.changeset/cloudwire-x.md b/.changeset/cloudwire-x.md new file mode 100644 index 000000000..60ef456a2 --- /dev/null +++ b/.changeset/cloudwire-x.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +FTW Cloud's protocol gains `x`, which carries a watch's exchange with its box through a one-shot request, so a watch can charge the car without its iPhone near. diff --git a/.changeset/down-group-pv-share.md b/.changeset/down-group-pv-share.md new file mode 100644 index 000000000..0a83a386c --- /dev/null +++ b/.changeset/down-group-pv-share.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +Batteries now take the whole charge FTW asks for when another battery is down. On a site with inverter groups, the PV of an inverter whose battery is offline, faulted or cannot charge was still counted as charge for that battery, and that part of the target went to no battery. With 3.3 kW of PV on the down battery's inverter and a 9.6 kW charge target, the other two batteries got 6.3 kW in total; when the target was smaller than that PV, they got nothing and the surplus went to the grid. That PV now counts as power on the AC bus, so the other batteries share the whole target by capacity, within their limits. diff --git a/.changeset/ev-command-cap-amps.md b/.changeset/ev-command-cap-amps.md new file mode 100644 index 000000000..d0680b61d --- /dev/null +++ b/.changeset/ev-command-cap-amps.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +Each charging order now tells the charger's driver how much current per phase FTW allows that charger right now, in a new `cap_amps_per_phase` field. It is the charger's share of the room below the fuse and the import limit, after the split between chargers, within the order's `max_amps_per_phase` and the charger's maximum. Until now a driver saw only the fuse rating. The Easee Cloud driver needs this to start a 6 A charge at 7 A only when 7 A fits, so a car on a tight phase is not stopped and restarted. The field adds no new limit, and FTW sends the same orders as before. diff --git a/.changeset/ev-inferred-site-headroom.md b/.changeset/ev-inferred-site-headroom.md new file mode 100644 index 000000000..c1480ca70 --- /dev/null +++ b/.changeset/ev-inferred-site-headroom.md @@ -0,0 +1,9 @@ +--- +"ftw": patch +--- + +Explain EV under-delivery with a conditional site-headroom projection from fresh three-phase measurements with confirmed import on every phase, keeping device-reported reasons distinct. + +Show the projected phase current against the working limit after the safety margin, and label it as a projection without denying other charger-reported information. + +Preserve the current one-phase and hourly import-limit explanations, and use the combined Easee source with bounded 6 A start assistance and power-observation ordering. diff --git a/.changeset/fleet-report-health.md b/.changeset/fleet-report-health.md new file mode 100644 index 000000000..e6cc19aab --- /dev/null +++ b/.changeset/fleet-report-health.md @@ -0,0 +1,5 @@ +--- +"ftw": minor +--- + +The anonymous daily fleet report also says, coarsely, how the box ran: how slow its slowest plan of the last day was, how many plans fell back, how often its drivers failed to read, and whether its newest update of the last week failed. Each is a bucket of a few words, under the same switch as the rest of the report. Settings shows them in the message preview. diff --git a/.changeset/interchangeable-batteries-share.md b/.changeset/interchangeable-batteries-share.md new file mode 100644 index 000000000..0f6dff22d --- /dev/null +++ b/.changeset/interchangeable-batteries-share.md @@ -0,0 +1,5 @@ +--- +"ftw": patch +--- + +When all batteries have the same power per kWh, they share the plan's total again. In a planner mode FTW followed each battery's own share of the plan. The planner pays the same for a kWh on either battery, so it often put a whole slot on one and the next on the other: one battery worked while the other stood at 0 W, their charge levels drifted apart, and on a large load only one battery's planned rate was used and the rest came from the grid. FTW now splits the live battery target across such batteries by the energy each can still give or take. The fuller battery discharges faster and the emptier one charges faster, so their charge levels draw closer while they work. Both cover a large load within their power limits: a battery at its limit stays there and the others take the rest. A battery that reports it cannot charge or discharge hands its share to the others. In an idle slot, or when FTW does not know a battery's charge level, the split is by capacity as before. Batteries whose charge or discharge limit per kWh differs by more than 5 %, such as one on a smaller hybrid inverter, still follow their own planned energy, and so does every battery in a fleet where any one differs. There is no new setting. diff --git a/.changeset/watch-exchange-through-cloud.md b/.changeset/watch-exchange-through-cloud.md new file mode 100644 index 000000000..2cd168a84 --- /dev/null +++ b/.changeset/watch-exchange-through-cloud.md @@ -0,0 +1,5 @@ +--- +"ftw": minor +--- + +An Apple Watch can charge the car without its iPhone near. Its commands reach the box through FTW Cloud as short encrypted exchanges the box answers itself, and an owner's phone adds the watch by its key, with a passkey step-up. The box lists `cmd.exchange` while FTW Cloud carries them. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c652ae13..275d56f30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,56 @@ # Changelog +## 0.155.1 + +### Patch Changes + +- 8ac2750: Charge now works on a one-phase charger again when the app asks for three phases. On a charger whose levels stop below 6 A on three phases, such as a 32 A one-phase charger with the levels 1.4, 2.3 and 3.7 kW, the app sent Charge now as three-phase, because FTW reported three phases for an auto charger. A three-phase order there has no level, so the car got 0 W. FTW now runs it on one phase unless the charger is set to 3p, and reports one phase for such a charger, so the app sends what it can run. +- 5ae590a: A charger that keeps charging after FTW tells it to stop or slow down now shows as Not following, not Limited. In 0.155.0 an Easee Cloud charger that went on drawing 3.1 kW after FTW cut it to 0 W for the main fuse showed "Limited by the main fuse" for as long as it drew. A limit explains why FTW ordered less; it does not explain a charger that takes more. FTW now warns once the charger has drawn more than its order for four minutes, and raises an alarm ten minutes later. A car that takes less than a limited order is judged as it is under a plan order. Sending the same order on every tick no longer pushes the earlier order out of the two-minute response window, so a slow cloud charger is not judged on an order it could not yet have applied. +- 62ed7ea: Charging is no longer cut twice when a cloud charger is slow to apply a cut. Easee Cloud takes 30–120 s to apply a new current, but FTW cut again after 20 s, before the first cut could show on the site meter, so a kettle on top could take the car from 10,400 W to 0 W. FTW now lets a cut it has sent cover the overage until the charger can have applied it: the time the driver declares, or 3 minutes. Only what a new load adds is cut. A raise still on its way now counts in a cut, so the phase is not left over the limit when the raise lands. Once a phase's averaged current passes 1.25 times the fuse rating, which a gG fuse carries for an hour, FTW cuts at once whatever is on its way. +- bbeac31: Charging starts again promptly after a pause. In 0.155.0, an Easee Cloud charger that was paused and then asked to start waited up to three minutes, shown as limited by the main fuse, while the phases had plenty of room: FTW still counted the higher current it had ordered before the pause as on its way. A pause now ends that reservation, because the charger follows its latest order and the site meter shows its draw until the pause lands. In the reported case the restart now comes after 12 s instead of 2 min 47 s. +- 5b697a5: A one-phase charger now starts while FTW raises the battery. Since 0.147.2 a car on a one-phase charger, or on a charger set to 1p, could sit at 0 W shown as limited by the main fuse with every phase far below its limit: FTW took the battery's whole new order, which the site meter could not show yet, off the room on the car's one phase, so a three-phase battery counted three times there. In the reported case Charge now waited 42 s; on a busy 16 A site with the battery stepping up every half minute the car did not start for 9 minutes. The battery's order now counts as a three-phase battery puts it, the same current on each phase. A three-phase charger's limit does not change. +- b0cb7d4: A car that charges on one phase of a three-phase charger is planned again once FTW sees it. FTW plans such a car at a third of each three-phase order, but it gave the planner the charger's three-phase minimum, 4.1 kW, with those one-phase levels. The planner dropped every level below that minimum and planned no charging, so the car stopped about five minutes into the session and missed its goal. The planner now gets the charger's three-phase levels divided by three, for example 1.4–3.7 kW. Each level is a third of an order the charger accepts, so the car draws what the plan counts. +- 2e84710: A one-phase car, as many plug-in hybrids are, now gets what it can take from a three-phase charger. Such a car draws the order's current per phase on one phase only, so a 4.8 kW three-phase order gave it 1.6 kW while the plan counted 4.8 kW, and the charger panel said it was not following. FTW now sees it on the site meter: when its order rises, one phase rises by about the order's current and the other two stay put, while the charger's own reading shows the car drawing that much. For the rest of that charging session the plan counts the car's one-phase levels, up to the charger's current per phase (3.7 kW at 16 A), the charger stays on three phases, and the order names the current the car should take. The charger panel says "Charging on one phase, at most 3.7 kW" instead of "Not following". A house load, a three-phase car or a two-phase car is never taken for a one-phase car, and the next car starts unknown. The fuse limit is unchanged: it counts a three-phase order's current on every phase. +- 04dbdd2: Planned charging and Charge now run on three phases. In 0.155.0 a loadpoint set to auto could charge a three-phase car on one phase all night: with an Easee step table that also lists one-phase levels, the planner spread the charge at 3.1–3.5 kW, which only one phase can deliver, and FTW switched the charger to one phase to follow it. On a 3×20 A site the car got about 3.3 kW instead of about 10 kW. One phase never helps the fuse: it puts all the current on one phase, caps the car near 3.7 kW, and an Easee has to pause to switch back. Auto, and an unset phase mode, now mean three phases on a charger that can run 6 A on each, and a planned order is raised to that minimum (4.14 kW at 230 V). Only charging from spare solar below it uses one phase, with the same hold against flapping as before, also when that solar-only loadpoint has a goal. The planner gets only steps the charger can run on three phases. A charger whose step table, or maximum, stays below 6 A on three phases is a one-phase charger and keeps its one-phase steps and orders; so does a loadpoint set to 1p. A one-phase charger above 16 A with a longer table still needs phase mode 1p. +- 04dbdd2: FTW no longer cuts a charging car when every phase has room. In 0.155.0 a car charging on one phase under auto was cut from 3.5 kW to 1.6 kW and then to 0 W, again and again, while the busiest phase read about 16 A on a 20 A fuse: dividing the fuse room between cars read the order as three-phase, gave the car's own current no credit, and cut it to what was left on that phase. Keeping a car's current order on the same phases is now never a cut there; only a measured overage on the site meter cuts. An order that moves the same watts onto other phases is still held to the room on those phases. + +## 0.155.0 + +### Minor Changes + +- 767b144: Add a Home Assistant ingress mode. The HA app can use the owner's HA login for setup and settings, including installs with a saved FTW password. The ingress listener accepts Supervisor requests and local health checks; other clients cannot open the UI. Ordinary native installs keep their existing access rules. + +### Patch Changes + +- 8d5c6bf: Charge now keeps charging after an update or restart. Before, a restart turned it into a pause until someone pressed Start again. It still ends on Stop, when the car is unplugged or full, and a different car on the charger does not inherit it. +- c14d31d: A charger is never asked for more current per phase than it supports. When FTW moves a charger that can run three phases onto one phase, the order now stops at that charger's own current per phase instead of the site fuse. Before, a 16 A charger on a 25 A fuse could be offered 18–25 A on one phase while charging from surplus or on Charge now. +- 6aa082f: EV charging no longer pushes a phase past the fuse when a charger is slow to apply a new current, when a charge moves between three phases and one, or when one car is raised while another is cut. Each raise now counts as current on the phases its command names until the charger can have applied it: the time its driver declares, or 3 minutes when it declares none. A car takes only room that no other charger's raise holds. After a rise, a charger that declares no response time may wait up to 3 minutes before it climbs further. +- fc621be: Charging no longer stalls or switches off when the phase mode is empty or auto, when a battery-level PV unlock is saved but not active, or on a one-phase site. The site-meter limit is now the only fuse guard for chargers: it measures room on the phases each charger is told to use, and a three-phase offer is never below 6 A per phase. +- a2330c0: Home use is marked "≈" as soon as a device stops answering. Before, Core kept the device's last power for up to 90 s and still called Home use measured, so the page could show a wrong value and "% self-powered now" while every watt came from the grid. The last value is still shown until the age limit, but as an estimate that names the device. A device the watchdog has taken offline now stays offline until it delivers data again. Each failed read used to bring it back for a moment, so Home use flipped every few seconds and the watchdog reported the device lost again each time. +- b5702e7: When the hourly import limit holds a car back, FTW now says so instead of blaming the main fuse. The charger's reason is `import_limit` ("Limited by the hourly import limit") whenever this hour's import budget leaves less room than the fuse, and a battery boost stopped by that budget says "Hourly import limit stopped boost". `fuse_limit` now means the main fuse alone, and the web page says "Limited by the main fuse". +- 98e5c79: Home use is no longer marked estimated because an OCPP charger has no car. When the charger reports Available, no car is connected and no current flows, so its power counts as a measured 0 W in live Home use and in history. Each heartbeat or other message from the charger keeps that reading current, and it ends as soon as a car arrives. Unavailable and Faulted still leave the charger's power unknown. The Available a charger sends for itself (connector 0) at boot or reconnect no longer reads as an unplugged car. Core now asks OCPP chargers for a heartbeat every 30 s instead of every 60 s, so one late heartbeat no longer marks an idle charger offline: one more small request and reply a minute per charger. A heartbeat_interval_s set in the config is kept. +- db5aec8: An idle OCPP charger no longer loses its connection about once a minute. The OCPP library closed a charger socket that had sent nothing for 60 s, the same as a 60 s heartbeat, so a charger that only sends heartbeats could be dropped as each one arrived, reconnect, and briefly show Home use as "≈". A charger socket may now stay silent for five heartbeat intervals (150 s with the default 30 s heartbeat). FTW's own watchdog still decides when a quiet charger is offline. + +## 0.154.4 + +### Patch Changes + +- 5e8e417: Stop full backup creation and verification when the HTTP client disconnects. Ctrl-C releases the backup and update locks instead of leaving work running on the box. +- df21447: A failed full backup now says why. Core writes when a backup starts, ends and fails to the journal, with the phase, table and row count it reached. A backup stopped by the two-hour limit for a live copy names that limit instead of only "context deadline exceeded". GET /api/backups keeps the reason, table and row count until the next backup starts, and `ftw backup` shows the last failure before it begins. When the raw copy does not fit in /dev/shm, Core warns that it is copying on the data disk and reports how much space earlier backups' scratch files still take there. +- 7ad6948: Core now removes what a stopped full backup left behind before it accepts a new one. A backup cut short when Core stops, for example by an update, left its raw copy in /dev/shm until the next reboot. On a Raspberry Pi that held more than a gigabyte of RAM and sent the next backup's copy to the SD card, where it ran past its two-hour limit. At start Core removes such raw copies, the stage and verification folders and any unfinished archive in the backups folder, and logs each path and its size. It leaves them when another backup is running, and it removes nothing else. +- aa8bc17: Report the longest step and step count beside each total phase time when a history write or forecast read fails. Archive failures also report connection, preparation, locks, write and commit times before cleanup. This separates one slow operation from many short operations without logging sample values or changing storage behavior. +- 18f75ac: Home use on the energy flow no longer shows "≈" when every device is read, and "% self-powered now" is back. The page marked Home use as an estimate whenever /api/status carried a second, energy-averaged value, which it always did. It now follows Core's `load_quality` alone, and /api/status no longer sends `load_estimate`. + + Home use has one value. 0.154.3 was never published as a beta; its change also ships here: Home use is the site meter minus every device that is read, shown when one device is offline (marked "≈"), and blank only when the site meter is missing or stale. The Home Assistant sensor "Load Power (energy average)" announced for 0.154.3 is removed before release; "Load Power" stays. + +## 0.154.3 + +### Patch Changes + +- 01ab443: Show Home use again when one device is offline. Live Home use is the energy balance as is: the site meter minus every device that is read, as before 0.141. A device that is offline, or a setup problem, no longer blanks it; the dashboard marks it "estimated" with "≈". A negative value is shown, since it points at production FTW cannot see. After sunset at the site's map pin an offline inverter counts as 0 W. Home use is blank only when the site meter is missing or stale. `/api/status` adds `load_quality` and `load_reason` as hints. + + Home use now has two series. The live display value is logged as the `load_display_w` metric. The energy-based Home use from closed history intervals, with its quality flags, stays what load models train on. Home Assistant gets a second sensor, "Load Power (energy average)", next to "Load Power". + ## 0.154.2 ### Patch Changes diff --git a/Makefile b/Makefile index afe4082ae..c24d24bb0 100644 --- a/Makefile +++ b/Makefile @@ -12,7 +12,7 @@ # make clean — remove all build artifacts .PHONY: help test compose-migration-test container-boundary-test release-workflow-test build build-arm64 build-amd64 release release-linux release-linux-arm64 release-linux-amd64 \ - run-sim sim-ocpp dev fmt vet clean e2e ci ci-ui ci-hw-pi docs \ + run-sim sim-ocpp dev fmt vet clean e2e siterig siterig-full siterig-random siterig-coverage siterig-judge ci ci-ui ci-hw-pi docs \ verify verify-all install-hooks driver-repository-validate driver-versions protocol-boundary cloudlink-boundary \ drivers drivers-present driver-versions-across-pin @@ -121,6 +121,32 @@ release-workflow-test: e2e: drivers-present cd go && FTW_E2E=1 go test ./test/e2e -v -timeout 180s +# FTW against device-simulator's house with a real fuse +# (go/test/site/README.md). SITERIG_SET is short, full or scenario names; +# SITERIG_LANES runs that many at once on ports 100 apart. +SITERIG_SET ?= short +SITERIG_LANES ?= 2 +siterig: drivers-present + cd go && FTW_SITE=$(SITERIG_SET) FTW_SITE_LANES=$(SITERIG_LANES) \ + go test -count=1 -timeout 300m -run 'TestSite$$' -v ./test/site/ + +siterig-full: + $(MAKE) siterig SITERIG_SET=full + +# Random cases: SEED=n replays one, COUNT=m runs m seeds from SEED. +SEED ?= 1 +COUNT ?= 1 +siterig-random: + $(MAKE) siterig SITERIG_SET=random:$(SEED):$(COUNT) + +# Which fault-catalog entries have a scenario: CATALOG=path/to/catalog.yaml +siterig-coverage: + cd go && FTW_SITE_CATALOG=$(CATALOG) go test -count=1 -run TestCatalogCoverage -v ./test/site/ + +# Judge a recorded run again with the current criteria: DIR=results directory +siterig-judge: + cd go && FTW_SITE_JUDGE=$(DIR) go test -count=1 -run 'TestJudge$$' -v ./test/site/ + driver-repository-validate: drivers-present cd go && go run ./cmd/ftw-driver-repository publish -unsigned -drivers ../drivers -output ../dist/driver-repository -base-url https://example.invalid/releases/download/drivers-local -repository https://github.com/srcfl/ftw diff --git a/README.md b/README.md index c36c7d66b..3a794b9a3 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,9 @@ See [CONTRIBUTING.md](CONTRIBUTING.md). Community help is best effort; Join the [FTW Discord](https://discord.gg/UK2ygPBu8N) for questions and community discussions. +See [project statistics](https://stats.ftw.energy/) for public development +activity and anonymous usage totals. + ## Architecture FTW has three explicit modules: diff --git a/config.example.yaml b/config.example.yaml index c2ea888bb..c8e9d86a7 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -192,7 +192,7 @@ ocpp: path: / username: ftw password: "" # required when enabled; use a long random string - heartbeat_interval_s: 60 + heartbeat_interval_s: 30 # Serve wss:// instead of ws://. client_ca_file additionally requires each # charger to present a certificate signed by that CA whose CN or DNS SAN # matches the identity in the URL — OCPP 2.0.1 security profile 3. @@ -220,16 +220,20 @@ state: app_link: enabled: true -# Anonymous fleet ping. Once a day, at a time that moves, the box tells the -# FTW relay six things: its FTW version, its release channel, which driver -# types it runs, roughly how much battery it has, its price zone and roughly -# how old the install is. No id, no key, no serial, no site name, no counter -# and no timestamp — nothing in the message says which box sent it. Two -# limits: those six fields still describe a household, so a rare mix may be -# the only one of its kind, and the relay sees the address during the request. -# It saves only daily totals, so the result counts reports, not unique boxes. -# Settings → FTW app → Fleet statistics shows the exact message this box would send. On by -# default; `enabled: false` turns it off without a restart. +# Anonymous daily reports. Statistics are one answer on the box: part of FTW +# Cloud, and on for a box without Cloud until its owner turns them off under +# Settings → FTW app → Anonymous statistics, which shows each message. Two +# reports, each alone and at its own time. The fleet report carries six +# things: FTW version, release channel, driver types, roughly how much +# battery, price zone and roughly how old the install is. The energy report +# carries a closed UTC day's six whole-kWh sums. No box id, no key, no serial, +# no site name, no counter. Once a release names a relay, both go through it, +# sealed with a random number drawn new for each report and the day it was +# sent, so FTW Cloud never sees the address; until then the fleet report goes +# straight to `endpoint`, which sees the address, and the energy report is not +# sent. +# `enabled: false` from an earlier release is read once and kept as "off"; it +# is no longer a switch of its own. fleet_ping: enabled: true # endpoint: https://cloud.ftw.energy/v1/fleet diff --git a/config.local.example.yaml b/config.local.example.yaml index 2e5f36f87..a5af8fa53 100644 --- a/config.local.example.yaml +++ b/config.local.example.yaml @@ -52,7 +52,7 @@ ocpp: port_v201: 8888 username: ftw password: sim-ocpp - heartbeat_interval_s: 60 + heartbeat_interval_s: 30 state: # `make dev` starts the Go process from go/, so keep disposable state in diff --git a/docs/adr/0008-ftw-cloud.md b/docs/adr/0008-ftw-cloud.md index 8ad971d96..9a5ba5274 100644 --- a/docs/adr/0008-ftw-cloud.md +++ b/docs/adr/0008-ftw-cloud.md @@ -162,6 +162,9 @@ Owners who decline use the box locally without Cloud. Local-only users get a separate question, with Yes as the suggested action. Nothing starts before an active answer. No leaves local control working. +(Replaced on 24 October 2026: a box without Cloud shares the two anonymous +reports until its owner turns them off; see "Anonymous daily reports and one +statistics answer" below.) This statistics-only link publishes no phone keys, pairing codes, Noise key, read grant, live state or remote access. It reuses the signed upload path. A policy or purpose change requires a new version and a fresh answer. @@ -212,6 +215,7 @@ The privacy notice says so. | Store | Kept | |---|---| | Live state | Memory only; readings dropped an hour after the box goes quiet | +| History tiles | Memory only; each 12-hour tile of 5-minute history until 48 hours after it ends or the roster runs out, box online or not | | Registry | Until purge, or 90 days without a roster | | APNs registrations | Until purge, or 30 days without the box | | Events | A one-hour ring in memory | @@ -328,6 +332,8 @@ who declines its terms pauses Cloud on that phone and keeps its local keys. The daily anonymous report also waits for the current project choice. Its saved config switch cannot bypass a missing answer or a local No. +(Replaced on 24 October 2026; see "Anonymous daily reports and one +statistics answer" below.) A new policy rotates the stats id and starts a new reporting window. An unset wall clock starts that window only once time is known; no earlier hours are filled in. @@ -349,3 +355,107 @@ even if an owner previously chose to contribute. The owner policy response reports this readiness so native clients can explain the wait. Pausing the service policy stops collection while withdrawal and retention still run. Deploying this code does not sign the DPIA or accept a household's consent. + +### Anonymous daily reports and one statistics answer, October 2026 + +Fredrik decided this on 24 October 2026, so that stats.ftw.energy can show +energy without C8's household rows, and so that statistics are one thing: +"antingen delar man med sig av stats eller inte". + +**One answer.** Statistics are one answer on the box that covers both +anonymous daily reports, with no second setting: + +- With FTW Cloud they are part of Cloud, with no separate off switch, as in + the 17 October amendment. A Cloud grant under older terms sends nothing + until the owner accepts the current ones. +- Without Cloud they are on until the owner turns them off, under Settings → + FTW app → Anonymous statistics, where the box shows each message. Nobody is + asked any more; the local question and its "nothing before an answer" rule + are gone. +- Stored state is read once (`cloudlink.MigrateStatistics`). A local No, a + fleet report switched off in the config (`fleet_ping.enabled: false`), or + Cloud turned off before this release (whose screen said that stops + statistics) stays off. Every other box is on. `fleet_ping.enabled` is no + longer a switch. +- A box without Cloud no longer starts a statistics-only Cloud link. Those + already running keep working until turned off. Hourly C8 rows for local + boxes therefore have no way in; if C8 is ever activated for them, it needs + its own consent. +- The hourly C8 upload follows the same answer: off, or Cloud under older + terms, sends no hour whatever the grant carries, and an answer that is off + takes the stats purpose out of the grant. Under older terms, asking for + statistics is refused until the owner accepts the current terms. +- A stored grant counts only through the running link, which honours it only + with the key it was made with. + +**Two reports, never one message.** Each leaves at its own random time: + +- the fleet report (`go/internal/fleetping`): version, channel, drivers, + battery size, price zone, install age, as before; +- the energy report (`go/internal/energyreport`): once per closed UTC day, + 1 to 12 hours after it closes, the date and six whole-kWh sums (solar + generation, grid import and export, battery charge and discharge, vehicle + charge). A day with a gap in the energy ledger is skipped, never sent in + part. Never for a site a partner controls. + +With a fleet of about eight boxes, a driver set such as Sungrow, Easee, +myuplink and tesla-ha in SE4 points at one home. Energy sent beside it would +be that home's energy again. So the two share no field and never travel +together. + +**Path.** Both go through one independent Oblivious HTTP relay (RFC 9458) to +FTW Cloud's gateway, `POST /v1/ohttp` in `ftwcloudd`, as one padded inner +request each (`go/protocol/reportwire`, 1,536 bytes; the relay always sees +1,591). Inside, each report carries a random id and the UTC date it was +sent. The gateway reads the schema, adds the report to its own day totals +(`/v1/fleet/public`, `/v1/energy/public`) and keeps nothing per report but +a hash of its id, for three days, so a replay or a retry counts once +(RFC 9458, section 6.5). A report sent before yesterday is refused. Both +public views list closed days only, so nobody can match a new count to the +report that just arrived. +`go/internal/statsrelay` holds the relay URL and the gateway key as +compile-time pins. The box encrypts only to the pinned key and never asks the +service for one, so every box on a release uses the same key and a key +swapped in at the service opens nothing. Until both pins are set, the energy +report is not sent, and the fleet report posts straight to `/v1/fleet` as +before, but only from a box that said yes: FTW Cloud's terms or a local yes. +A box that shares only by default never posts straight to FTW Cloud, which +would see its address; it waits for the relay (Fredrik, 24 October 2026). A +build with the relay sends the fleet report to FTW Cloud only through it. +`POST /v1/fleet` stays for boxes on older releases; it can go one stable +release after the first that pins the relay. A collector the household set +in `fleet_ping.endpoint` is still posted to directly. + +**Who sees what.** The relay sees the box's address, the time and a +ciphertext of fixed length, never a report. Cloudflare's relay, if chosen, +keeps those logs for about 124 days. The gateway sees each report, the +relay's address and the time, never the box's address. FTW Cloud keeps per +date only the counts and sums; a public energy date appears once no report +may still name it, four days after it ends, and needs at least 5 reports. A +sum that one to four of them had any of shows null: five reports with one car +would otherwise publish that car's charge. + +Limits, plainly: + +- The relay and Sourceful together could join reports to addresses by time. + The relay's independence is the control. +- FTW Cloud holds readings from homes whose owners turned on the readable + copy, including 48 hours of history tiles. The same daily sums could be + computed from them and matched with a report. The gateway does not do this + and keeps nothing to match with, but that is a rule in code, not one the + cryptography enforces. +- The floor of five counts reports, not boxes; Fredrik accepted this on + 24 October 2026. An anonymous report cannot prove that a real box sent + it. The relay alone can drop the other reports and add fakes to isolate + one home's day, and that shows on our side as missing reports. Anyone + else can do it only on a day when one real home reported. Rejecting + replays and listing closed days only remove the cheap attacks. Anonymous + tokens (Privacy Pass) are the later fix, once the fleet grows or someone + shows a need. + +**Open for Fredrik's lawyer, not decided here.** Reports that are anonymous +to Sourceful through OHTTP should be outside GDPR. Reading data from the +owner's equipment to make them may still need consent under the ePrivacy +rule (Swedish LEK, chapter 9, section 28). If it does, neither the opt-out for +boxes without Cloud nor the requirement with Cloud holds, and both need an +active yes. Fredrik chose this design; it is built as he asked. C8 stays off. diff --git a/docs/adr/0009-watch-exchange.md b/docs/adr/0009-watch-exchange.md index 13acbadb8..090798be7 100644 --- a/docs/adr/0009-watch-exchange.md +++ b/docs/adr/0009-watch-exchange.md @@ -2,10 +2,11 @@ - Status: accepted. This repository has milestones 1, 2, 3, 5 and 6: the command log on the box, the registry names, the watch role with its - enrolment, the exchange socket with its sessions, so the box says - `cmd.exchange`, and the wire for FTW Cloud's one-shot read. The relay's - half is milestone 4, in srcfl/ftw-webapp. Nothing else described here has - shipped. + enrolment, the exchange sessions, and the wire for FTW Cloud's one-shot + read. The relay stopped on 17 October 2026, and its exchange socket went + with it. [Through FTW Cloud](#through-ftw-cloud) (accepted by Fredrik on + 23 October 2026) moves the exchange onto FTW Cloud's one-shot request, where the + box says `cmd.exchange` again. - Date: 2026-10-14 - Decided by: Fredrik on 14 October 2026. Before the design: a watch controls the box directly as well as through the phone; an unlocked watch on the @@ -186,6 +187,49 @@ to remove it.** it when the box returns or when the roster runs out after 24 hours. An owner can suspend the watch at FTW Cloud from the phone until then. +## Through FTW Cloud + +Accepted by Fredrik on 23 October 2026, after he asked to finish FTW Cloud. The +relay was stopped on 17 October 2026, and since the one-door decision of 15 +October every phone reaches its box through FTW Cloud's tunnel. FTW Cloud +carries ciphertext and never chooses an action; that is the line this ADR +keeps, and the watch's commands now sit on the same side of it as the +phone's. + +1. **The exchange rides the one-shot request.** A watch's exchange message + goes to `POST /v1/app/once` as `x {d}`. The service sends the box + `x {dev, id, d}` on the uplink and returns the box's `d` in message 2. + Decisions 1 and 2 stand, apart from the socket: the sessions, their + limits, the shapes and the 400 ms floor are the box's, unchanged. The + wire is in [protocol-cloud.md](../protocol-cloud.md#exchange). +2. **The phone adds its watch by key.** Decision 4's code had to cross the + service inside a handshake from a device the roster does not list yet. + Instead the owner's phone sends the watch's public key over its own + session, with the same step-up: `POST /api/app-link/pairing + {"role": "watch", "key": …}`. The box lists the watch in its next + roster. The code path is gone: a request for a watch without a key is + refused, and a hello with a code is answered with silence. +3. **A watch has `tun`.** The roster gives a watch `tun` whenever it gives + a phone `tun`, so the service admits its exchange with the readable copy + off. The box answers a watch's key on the exchange alone. +4. **Reads stay as decision 7 says**, with one change: with the copy off a + watch on its own shows what it holds and its age. Reading `now` through + the exchange is not built. +5. **The exchange takes the charger's hold alone.** Decision B said a + watch reads and charges. The watch role's `ftw.dispatch.write` reaches + further, so the exchange answers any op but `loadpoint.hold` with + `E_OP_NOT_GRANTED`. +6. **Replays end nothing, and a restart forgets.** A session proves itself + with its first transport message before it writes anything, and a + watch holds two. The command log lives in memory, so a box that + restarted between a command and its retry would act again; the watch + sees the uptime in `hello_ok` go back and does not resend. + +The service learns that a watch of the home sent a request and when, and +when the box answered; the relay learned the same. One `cmdId` sent by the +phone and through FTW Cloud still acts once, because the command log keys +it by the phone that added the watch. + ## What does not change - Every exchange command takes the same `onCmd` path as the stream: expiry, diff --git a/docs/architecture.md b/docs/architecture.md index 70bcd14b1..8b7e1f327 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -614,10 +614,26 @@ The box's anonymous daily report is separate from the Cloud readable-copy grant. Once a day it posts its FTW version and channel, the driver types in use, a battery-capacity bucket, its price zone and an install-age bucket. It answers how many reports arrive and what they describe, which helps size engineering -bets. - -It goes over HTTPS to `https://cloud.ftw.energy/v1/fleet`. It never enters the -Noise session. The HTTP handler validates the fixed payload, adds it to the UTC +bets. It is one of two anonymous daily reports under one statistics answer, with +the energy report beside it (ADR 0008). + +Since `ftw.fleet/2` it also says, in four buckets, how the box ran: the +slowest plan of the last 24 hours (`none`, `0-1s`, `1-5s`, `5s+`), how many of +those plans fell back (`none`, `0`, `1-3`, `4+`), the share of driver polls +that failed (`none`, `0`, `0-1%`, `1%+`) and the newest update of the last 7 +days (`none`, `ok`, `failed`). Core reads them from the plans it already keeps +(`planner_diagnostics`), a 24-hour count of the driver registry's polls in +the telemetry store and the updater's status. The poll count lives in memory, +so until it covers six hours, as after a restart, the share says `none`. +An OCPP charger's messages are not polls and are not counted. A box that does not update itself says `none`, so how it +was installed does not become a field. The service accepts both schemas and +counts health only from reports that carry it. + +Once a release names an Oblivious HTTP relay (`go/internal/statsrelay`), it goes +through that relay to FTW Cloud's gateway, sealed, so FTW Cloud never sees the +box's address. Until then it goes over HTTPS to +`https://cloud.ftw.energy/v1/fleet`, and only from a box that said yes. It never +enters the Noise session. The HTTP handler validates the fixed payload, adds it to the UTC day's counters and drops it. It keeps 90 calendar days of totals, not raw reports. An operator reads totals only on the loopback metrics listener. An explicitly saved former default resolves to Cloud; a private collector keeps @@ -627,7 +643,11 @@ Fleet totals must not form a household record. The message carries no gateway ID name, no counter and no timestamp — nothing in it says which box sent it — values are bucketed rather than reported, the version travels only when it is a release tag so a developer's build reports as unknown, and the send time is -drawn fresh each day rather than sitting in one slot. +drawn fresh each day rather than sitting in one slot. Through the relay the +sealed request also carries a random number, drawn new for each report so FTW +Cloud counts a report once if it arrives twice, and the UTC day it was sent, so +an old request is refused. Neither names the box, and FTW Cloud keeps a hash of +the number for three days only. Driver names get their own rule, because a driver file's name is whatever the thing that installed it called it, and a household can install its own. A name @@ -673,15 +693,20 @@ Three things this does not fix, and each is said on the Settings screen or the Cloud service's loopback stats response. The fields still describe a household, so the payload remains a quasi-identifier: a beta box in a small price zone with a big battery may be the only one like it, and coarse buckets with a small field set -are what keep that population large rather than a proof that it is. The endpoint -also sees the source IP while the request is open, though the Cloud service writes no -request log or source address. And with no id there is no way to dedupe, so the totals count -reports, not unique boxes. - -A failed send is forgotten, never retried. Settings → FTW app → Fleet statistics renders the -exact payload from the same call the sender uses, so the claim is checkable -rather than promised. It is on by default; saving `enabled: false` opts out -without a restart. See [`go/internal/fleetping`](../go/internal/fleetping). +are what keep that population large rather than a proof that it is. The health +buckets are each at most four words wide, describe the last day or week rather +than the house, and most boxes share the common answer; the service keeps each +field's daily count apart, never the combination. The endpoint +also sees the source IP while the request is open when the report goes straight +to it, though the Cloud service writes no request log or source address; through +the relay it does not. And nothing ties two reports to one box, so the totals +count reports, not unique boxes. + +A failed send is forgotten, never retried. Settings → FTW app → Anonymous +statistics renders the exact payload from the same call the sender uses, and +what goes beside it, so the claim is checkable rather than promised. It follows +the one statistics answer, whose switch is on that screen. See +[`go/internal/fleetping`](../go/internal/fleetping). ## Releases diff --git a/docs/home-assistant-app.md b/docs/home-assistant-app.md index 0ff0f3b88..a4f744df3 100644 --- a/docs/home-assistant-app.md +++ b/docs/home-assistant-app.md @@ -45,20 +45,25 @@ Only one FTW may control a site. ## 4. Open FTW and set it up -Choose **Open web UI** (or go to `http://:8080`). +Choose **Open web UI** inside Home Assistant. App versions with ingress use +HA's login; no separate FTW password is needed. You can enable **Show in +sidebar** on the app page. HA requires an administrator to open FTW. +The direct web port on the home network is closed. + A new install opens the setup wizard. Follow it to add your devices. The -[setup guide](setup-guide/en.md) explains each step. -The button opens FTW directly on your home network. The app does not use -Home Assistant ingress. For remote access, pair an FTW client as described below. +[setup guide](setup-guide/en.md) explains each step. An existing app with a +saved FTW password uses HA's login after the update too. Its settings and +history stay in place. -Give Home Assistant a fixed address (DHCP reservation in the router), so the -FTW apps and your devices keep finding it. +Give Home Assistant a fixed address (DHCP reservation in the router), so your +devices keep finding it. ## 5. Connect the FTW app (iPhone) or webapp The new FTW webapp and iOS app need new FTW 0.x — this app provides it. -In FTW's web UI choose **Settings → FTW app** and follow -[Pair with one iPhone](pairing.md). If you used the webapp with an older FTW, +Ingress opens the web UI through HA. Direct LAN access from a separate FTW +client is not part of this mode. For a separate client, enable FTW Cloud and +follow [Pair with one iPhone](pairing.md). If you used the webapp with an older FTW, remove it from the home screen and pair again. ## 6. Home Assistant entities (optional) @@ -87,7 +92,7 @@ This restores its version and data together. Keep other FTW instances stopped. |---|---| | The app is not in the store | Check the repository URL, reload the page, or **⋮ → Check for updates**. | | Install fails | Open **Settings → System → Logs → Supervisor**. A download or checksum error means the package could not be fetched; try again later. | -| Web UI does not open | Check the app **Log**. Port 8080 must be free; another app on 8080 stops FTW. | +| Web UI does not open | Check the app **Log**. Open it through HA. Ingress port 18099 must be free. | | FTW cannot reach a device | Check the device's IP address and that it is on the same network. | | The FTW app/webapp says the box is too old | Make sure you installed **FTW (0.x beta)**, not the old **FTW (beta)**. | diff --git a/docs/ocpp.md b/docs/ocpp.md index 0b400173a..9955b75b5 100644 --- a/docs/ocpp.md +++ b/docs/ocpp.md @@ -135,7 +135,7 @@ ocpp: path: / # default / username: ftw password: - heartbeat_interval_s: 60 + heartbeat_interval_s: 30 ``` **Credentials are mandatory.** FTW refuses to start with `enabled: true` and an diff --git a/docs/protocol-cloud.md b/docs/protocol-cloud.md index d5330cd9d..bde98a520 100644 --- a/docs/protocol-cloud.md +++ b/docs/protocol-cloud.md @@ -6,9 +6,13 @@ or the phone. The bytes live in `go/protocol/cloudwire`, with golden vectors in its `testdata`: `cloud-vectors.json` for the uplink, `cloud-app-vectors.json` for one whole app session, its [tunnel](#tunnel) messages and a pairing proof, and -`cloud-once-vectors.json` for one [one-shot read](#one-shot-read), and +`cloud-once-vectors.json` for one [one-shot read](#one-shot-read), +`cloud-once-x-vectors.json` for one request of the [exchange](#exchange) +through the service, and `cloud-shard-vectors.json` for which hosts a client may follow and how it -reads a restart's spread ([Shards and restarts](#shards-and-restarts)). +reads a restart's spread ([Shards and restarts](#shards-and-restarts)), and +`cloud-tile-vectors.json` for the three shapes of a [tile](#tiles-ftwhistoryread) +and how they lay back together. `srcfl/ftw-cloud`, the phone and the watch replay them. `cloudwire.View` is the phone's [freshness](#freshness) rules in Go, and the cases listed there are its tests. @@ -43,7 +47,7 @@ for both live in `go/protocol`, under Apache-2.0 |---|---|---|---| | Uplink | Box | `wss://cloud.ftw.energy/v1/box` | TLS 1.3, a handshake bound to the TLS session on both sides, every box message signed | | App session | Phone | `wss://cloud.ftw.energy/v1/app?home=` | TLS 1.3, then Noise IK to the service's static key. Inside it, the phone's own Noise session with its box rides as [`tun`](#tunnel) | -| One-shot read | Watch | `POST https://cloud.ftw.energy/v1/app/once?home=` | TLS 1.3, then Noise IK to the service's static key, inside one request | +| One-shot read | Watch | `POST https://cloud.ftw.energy/v1/app/once?home=` | TLS 1.3, then Noise IK to the service's static key, inside one request. Inside it, a watch's [exchange](#exchange) with its box rides as `x` | | Extension read | Extension | `GET https://cloud.ftw.energy/v1/now` | TLS and an extension token; the data inside is box-signed | | Push | Service | APNs | Box-signed payloads where they carry data | @@ -360,7 +364,7 @@ with two different values, so neither side's check passes through it. |---|---|---| | `boxNonce` | bstr, 32 B | Echo | | `ekm` | bstr, 32 B | The service's exporter value for this TLS session | - | `have` | map text→text | Tile id to etag, for the tiles the service holds | + | `have` | map text→text | Tile id to etag, for the closed tiles the service holds | | `rosterSeq` | uint | The roster number the service holds | | `statsPolicy` | text, optional | The statistics policy this service is approved to ingest. Missing means no statistics; the box sends none | @@ -376,9 +380,10 @@ with two different values, so neither side's check passes through it. before this check passes.** A proxy that reads `hello` learns the box's public key and grant id, and cannot use them. 4. **Box → service:** a `roster`, at every connect (see [Roster](#roster)); - then one full message of each class the grant allows; then the tiles - missing from `have`. While the link waits for a purge, the box sends only - `bye` instead (see [Revocation](#revocation-and-deletion)). + then one full message of each class the grant allows; then the trailing + tile whole and the closed tiles missing from `have` (see + [`tiles`](#tiles-ftwhistoryread)). While the link waits for a purge, the + box sends only `bye` instead (see [Revocation](#revocation-and-deletion)). 5. After that the box sends on change, by the cadence rules. ## Classes @@ -617,16 +622,71 @@ when its own rules (enabled, cooldown) allow. Delivery is best effort. ### `tiles`: `ftw.history.read` -The body is the app protocol's `hist.chunk` body: `tileId`, `etag`, `res`, -`startMs`, `stepMs`, `series`, `data` and `partial`. A closed tile has the -same tile id, etag and bytes as the box serves over Noise. +The box's 5-minute history of the last 48 hours, so a phone can draw it while +the box cannot answer. The body is the app protocol's `hist.chunk` body: -- `res` is `5m`. The service keeps 48 hours. -- `series` is drawn from `grid_w`, `pv_w`, `battery_w` and `load_w`, which the - box serves today, and `ev_w` and `battery_soc`, which C2 adds. -- When a 5-minute bucket closes, the box appends to the trailing tile. Only - then does `startMs` differ from the Noise meaning: it is the first new - bucket, and `partial` is true. +| Field | Type | Meaning | +|---|---|---| +| `tileId` | text | `5m/1/`: the 12-hour tile, at stride 1, as the app protocol names it | +| `etag` | text | FNV-1a of `data`, as the app protocol computes it | +| `res` | text | `5m` | +| `startMs` | int | Wall time of the first point. The tile's start, except in an append | +| `stepMs` | uint | 300,000 | +| `series` | [text] | `grid_w`, `pv_w`, `battery_w` and `load_w`, in that order and no other: the series the app asks for over Noise | +| `data` | bstr | int32 LE, one block per series, the app protocol's missing value where the box has no record | +| `partial` | bool | The tile is still filling | + +A body is one of three shapes. `cloudwire.Tiles.Check` holds them, and +`cloud-tile-vectors.json` has one of each. A tiles envelope is at most 4 KiB +(`TileEnvelopeMax`); a whole tile is about 2.6 KB. + +- **Closed:** `partial` false, `startMs` the tile's start, all 144 points. + It has the same tile id, etag and bytes as the box serves over Noise for a + query of those series at 5 minutes. +- **Whole:** `partial` true, `startMs` the tile's start, all 144 points, with + the missing value from the first bucket that has not closed. It replaces + every piece of that tile the receiver holds. +- **Append:** `partial` true, `startMs` a later bucket, and only the buckets + that closed since the last piece of that tile: `data` holds as many points + per series as fit. The receiver lays it over what it holds. + +The box counts a bucket closed 30 s after it ends, when the history writer +has committed it. Then: + +- It sends an append for the trailing tile, or the tile whole when no piece + of it has gone on this connection or after 11 appends, so the service holds + at most 12 pieces of it and the box rereads it once an hour. +- When the tile's last bucket closes it sends the closed tile instead, and + nothing for the next tile until a bucket of it closes. +- Once an hour it packs the closed tiles of the last 48 hours again and + sends any whose etag changed, as when the ledger repaired a gap. +- At each connect, after the full state, it sends the trailing tile whole + and every closed tile of the last 48 hours that `hello_ok.have` lacks at + that etag, newest first. `resync {what: [tiles]}` sends the tiles it names, + or all of them when it names none, whatever `have` says. +- When `ftw.history.read` leaves the grant it forgets what it sent, since the + service drops what it held; when it comes back, the box sends every tile + again, as at connect. + +The box reads its ledger at most once per tick, one tile at a time, so a +reading never waits long behind a pass. It sends tiles only while NTP has +synced its clock, since tiles are dated by wall time, and only under a grant +with `ftw.history.read`, which it checks again before each one. They count +against the byte budget: about 100 KB a day. + +The service keeps each tile until 48 hours after it ends, so up to 60 hours +of history, and drops them all when the roster runs out. A phone reads them +only while the roster is in force, so with the box away that is 18 to 24 +hours after the box was last heard. The service sends the box +`resync {what: [tiles]}`, at most once per 5 minutes, when a roster gives +history back and when room frees after it dropped a tile for lack of it. + +`cloudwire.Lay` puts what a receiver holds back together: a closed tile as +it is, and the trailing tile from its newest whole piece with each later +append laid over it, in the order the box sent them, by `wallMs` and then +`seq`, since `seq` starts again when the box does. A later closed tile +replaces an earlier one, never an older one a later. The trailing tile stays +partial, and its etag is never offered as held. ### `hb`: every device in the roster @@ -850,7 +910,7 @@ a lane 0 frame closes the uplink. A fuzz test covers it. | `challenge` | `{v, nonce}` | First message. Unsigned | | `hello_ok` | See Handshake | Signed by the service key | | `watch` | `{hz: 1}` | Raises the period to 1 s for 60 s. While any phone of the home looks, the service sends it at once, every 50 s after that, and again when the box connects. Without it the box returns to 5 s. The box ignores any other `hz` | -| `resync` | `{what, tiles}` | `what` holds `full`, `tiles` or `roster`; `tiles` lists tile ids. The box honours at most one per 5 minutes and ignores the rest | +| `resync` | `{what, tiles}` | `what` holds `full`, `tiles` or `roster`; `tiles` lists tile ids, and none means every tile of the last 48 hours. The box honours at most one per 5 minutes and ignores the rest. The service sends one for tiles when a roster gives history back, and when room frees after it dropped a tile | | `purged` | `{atMs, what, grantId, statsId}` | Signed by the service key. `what` is `home`, `copy` or `stats`. Sent once every deletion it names is done | | `tun` | `{dev, sid, d}` | A phone's own session with the box. See [Tunnel](#tunnel). Only to a box that lists `tun` in `hello.caps` | @@ -898,6 +958,9 @@ as it decides for the same session over the relay. - **Effective scope** of a device is its `scopes` intersected with `grant.scopes`. It is empty while the readable copy is off, and a device with `tun` still opens a session for its [tunnel](#tunnel). +- A watch has `tun` whenever a phone may have it. For a watch it opens the + [exchange](#exchange) and nothing else: the box answers a watch's key on + the stream with silence. - A device that leaves the roster loses its sessions (4403), tokens and APNs registrations within a second of the service receiving the roster, which needs the uplink to be up. A device whose `epoch`, `role` or effective scope @@ -937,7 +1000,7 @@ as it decides for the same session over the relay. | Field | Type | Meaning | |---|---|---| -| `envelopes` | [Envelope] | Box envelopes in rising `seq`, each the box's own CBOR map, unchanged: for each class the device may read, the newest one the service holds; the newest `live` that carried `sources`; and the newest `hb`. Empty after an hour without the box, and while the readable copy is off | +| `envelopes` | [Envelope] | Box envelopes in rising `seq`, each the box's own CBOR map, unchanged: for each class the device may read but `tiles`, the newest one the service holds; the newest `live` that carried `sources`; and the newest `hb`. Empty after an hour without the box, and while the readable copy is off. A phone asks for tiles with `tiles.get` | | `lastSeenMs` | int | Service wall time when it last heard from the box | | `online` | bool | The box's uplink is up now | | `more` | uint | How many envelopes did not fit in message 2. They follow it as ordinary frames, in rising `seq` | @@ -959,7 +1022,7 @@ the table. |---|---|---| | `watch` | Any device | `{hz}`: 1 when the phone looks at the home, 0 when it looks away. It holds until the next `watch` or the end of the session | | `tun` | A device with `tun` | `{sid, d}`. The phone's session with its box. See [Tunnel](#tunnel). The service answers `tun` the same way | -| `tiles.get` | Devices with `ftw.history.read` | `{have}`, tile id to etag. The service answers with the `tiles` envelopes the phone lacks | +| `tiles.get` | Devices with `ftw.history.read` | `{have}`, tile id to etag of the closed tiles the phone holds. The service answers with the `tiles` envelopes the phone lacks: each closed tile it holds that `have` does not name at that etag, and every piece of the trailing tile, oldest tile first and each tile's pieces in the order the box sent them. Then `tiles.end {n}`, where `n` counts them. It answers this while the box is away too, and a device without the scope gets `tiles.end {n: 0}` | | `token.get` | Any device | Answered with `token {token, expiresInS}`. A new token replaces the device's old one | | `push.register` | Any device | `{apnsToken, environment, kind, activityId, app}`. One registration per APNs token; a Live Activity and its push-to-start each register their own. `app` is `iphone`, the default when absent, or `watch`, and picks the APNs topic | | `push.unregister` | Any device | `{apnsToken}` | @@ -1219,12 +1282,15 @@ exactly as `/v1/app` admits a phone and keeps nothing after the answer. one-shot handshake and an app session handshake never stand in for each other. 2. Message 1 carries one unsigned envelope: `read`, `token.get`, - `push.register` or `push.unregister`. The service refuses any other type, - an owner's `suspend` among them, with 400 before message 2. + `push.register`, `push.unregister` or `x` (see [Exchange](#exchange)). + The service refuses any other type, an owner's `suspend` among them, + with 400 before message 2. 3. The service admits the watch by the rules of the app session: its key in the home's roster, a roster that has not run out, no suspension, and a - non-empty effective scope, so the readable copy must be on. A refusal is 403 with an empty body, before message 2, and does - not say which check failed. + non-empty effective scope, so the readable copy must be on. For `x` the + roster entry's `tun` stands in for the effective scope, as it does for a + phone's tunnel, so commands work with the copy off. A refusal is 403 with + an empty body, before message 2, and does not say which check failed. 4. The answer is 200 with `content-type: application/octet-stream`. Its body is message 2 and then, for a `read`, the envelopes that did not fit, as transport messages. Each message, message 2 included, comes after its @@ -1249,8 +1315,95 @@ has stored the change. 4 KiB; the whole answer within 10 s. Past a limit the service answers 429, a longer message 1 gets 400, and a service without a Noise key answers 501. -When FTW Cloud is off, a watch on its own reads `now` on the box's exchange -through the relay instead, and prices and the plan wait for the phone. +With the readable copy off, a watch on its own shows what it holds and the +age of it, and prices and the plan wait for the phone. + +## Exchange + +A watch's commands go to its box through the box's exchange +([ADR 0009](adr/0009-watch-exchange.md), decisions 1 and 2): short Noise +sessions, `ftw.exchange.v1`, IK to the box's static key, which live on the +box and answer one request at a time. The relay that first carried them is +gone (17 October 2026), so each request rides one +[one-shot request](#one-shot-read) to the service, and the service hands it +to the box on the uplink. It reads neither the request nor the answer. + +1. The watch posts a one-shot request whose message 1 carries `x {d}`. `d` + is one exchange message: handshake message 1 with the watch's `hello` + (608 bytes), or a request of one lane 0 frame, `cmd` or `now` (536). +2. The service admits the watch as step 3 of the one-shot read says, with + `tun` in place of the effective scope, and only a roster entry with role + `watch`. +3. It sends `x {dev, id, d}` down the uplink: `dev` the roster id its own + handshake proved, `id` a number of its own for this request. It never + takes either from the watch. +4. The box answers `x {dev, id, d}` with the same `dev` and `id`. `d` is the + exchange's answer: message 2 with `hello_ok` (1,072 bytes), or two lane + 0 frames (1,072), `cmd.ack` and `cmd.result`, or `now` and a `tick`. An + empty `d` is the box's silence: a stranger, a key that is not the device + `dev` names, a spent session, or a message it could not read. +5. The service answers 200 with message 2 carrying `x {d}`, the box's `d` + unchanged, and keeps nothing. + +**`x`** + +| Field | Type | Meaning | +|---|---|---| +| `dev` | text | The roster device id of the watch. On the uplink only; the service sets it | +| `id` | uint | The service's number for one request, below 2³², which the box copies into its answer. On the uplink only | +| `d` | bstr | One exchange message, or empty for no answer | + +- **No answer.** The service answers `x` with an empty `d` at once when the + box is not connected, did not say `x` in `hello.caps`, or has 8 requests + open; and after 5 s without the box's answer. The watch cannot tell + these apart, and does not need to: an exchange session that got no + answer is spent, and a command that got none is sent again under its + `cmdId`, which the box's command log answers with the stored outcome, + unless the box restarted meanwhile (below). +- **A restart forgets.** The command log lives in the box's memory. A box + that restarted between a command and its retry no longer knows the + `cmdId`, and would act again. `hello_ok.clock.uptimeMs` says so: a new + session's uptime below the one the command was sent in means a restart. + The watch then sends the command no more, and says no answer came, which + may mean it ran. +- **The box checks it again.** `dev` must name a watch of this box; one + that does not costs the box nothing, not even a Diffie-Hellman. The box + runs the handshake and admits the watch by its key, exactly as ADR 0009 + says, and answers only if the key the handshake proved is the device + `dev` names. A request is tried only against that device's sessions. +- **Replays end nothing.** A new session stamps nothing and writes nothing + until its first transport message decrypts. A watch holds at most 2 + sessions and the box 8; past either, the oldest session that has not + proved itself goes first, never the newcomer. The box takes 12 handshakes + a minute from one watch and 60 in all. +- **What the service can do.** Drop, delay or replay a request. It cannot + read or forge `d`. A replayed message 1 opens a session nobody can use + and ends no working one; a replayed request fails in its session's + transport, and a replayed `cmd` in a new session meets the command log. + Every `cmd` carries its own expiry, in box uptime. +- **What a watch may do.** The exchange takes `cmd` with `loadpoint.hold` + alone, which Charge now, Pause and Back to the plan send, and `now`. Any + other op is answered `E_OP_NOT_GRANTED` before it runs, whatever the + watch role's scopes reach (ADR 0009, decision B). +- **The box's share.** `x` answers count against the tunnel's 24-hour + budget and against their own tenth of it, and stop once the tunnel has + spent nine tenths, so phones keep the last of it. The box takes 120 + requests a minute. A request whose answer could not leave is dropped + unanswered and runs nothing. +- **What the service learns.** That a watch of this home sent a request + and when, and when the answer came. Every request and every answer after + the handshake has one length, and the box answers a `cmd` no sooner than + 400 ms after it arrived, applied or refused, so the service cannot tell a + read from a command, or a refusal from a success. +- **Limits.** The one-shot read's: one request at a time per device and 30 + a minute. A command takes two (the handshake and the `cmd`) and a third + when its result had not settled. `d` is at most 2,048 bytes either way. +- **Enrolment.** An owner's phone adds a watch over its own box session, + with a passkey step-up, by the watch's public key: + `POST /api/app-link/pairing {"role": "watch", "key": ""}`. The + box lists it in the next roster, so its first request is admitted like + any other. No code crosses the service, and a request for a watch + without a key is refused. ## Extension token @@ -1495,7 +1648,9 @@ writes do not gain access. CGNAT is not a private network for this choice. deletes the home's unpublished rows before accepting another report. If the box is offline, this happens on reconnect. - **Silence:** after an hour without the box's socket, the service serves only - `lastSeenMs` and no readings. When the roster runs out, it closes sessions + `lastSeenMs`, no readings, and the tiles, each until 48 hours after its + end or until the roster runs out. A tile is dated history and never reads + as now. When the roster runs out, it closes sessions and refuses reads. After 30 days it drops tokens and APNs registrations. After 90 days without a valid roster it deletes the home as in a purge. - **A new home** that sends no roster within 24 hours is deleted. @@ -1576,7 +1731,12 @@ applied. Per home, the service keeps in memory the newest envelope of each class, the newest `live` that carried `sources`, 48 hours of tiles and a one-hour ring of -events. It writes no live readings to disk; the statistics files hold the +events. For tiles that is at most five closed tiles and 16 pieces of the +trailing tile, each as the box signed it, and at most 64 KiB per home; a +tile that fails `cloudwire.Tiles.Check` or is over 4 KiB is dropped and +counted. All homes together hold at most a fixed number of tile bytes, past +which new tiles are dropped and counted. `have` in `hello_ok` names the +closed tiles held. It writes no live readings to disk; the statistics files hold the only readings it stores. Its registry holds the home id, the box key, the roster, suspensions, the current and deleted `statsId`s, APNs registrations and token hashes. @@ -1665,11 +1825,17 @@ constants, so `srcfl/ftw-cloud` need not spell them by hand. | Phone messages | 20 in a burst, then 2 a second | | App session message 1 | Within 10 s | | One-shot reads | One at a time per device, 30 a minute; message 1 at most 4 KiB; answered within 10 s | +| Exchange through the service | Answered within 5 s or with an empty `d`; 8 open per box; `d` at most 2,048 bytes | +| Exchange at the box | 8 sessions, 2 per watch; 12 handshakes a minute per watch, 60 in all; 120 requests a minute; `x` a tenth of the tunnel budget, stopped at nine tenths | | Roster `ttlS`, renewal | 24 hours, every 6 hours | | Extension token | 7 days | | Alert age for buttons | 10 minutes | | `box.unreachable` after | 600 s | -| Readings served after silence | 1 hour | +| Readings served after silence | 1 hour; tiles until 48 hours after their end, or until the roster runs out | +| Tiles | `5m`, 12-hour tiles, the last 48 hours: at most 5 closed and 16 pieces of the trailing one, 64 KiB, per home; 4 KiB an envelope | +| Bucket counted closed | 30 s after it ends | +| Trailing tile sent whole | At each connect, after 11 appends, and when its first bucket closes | +| Closed tiles packed again | Every hour, sent when the etag changed | | Sessions closed | When the roster runs out | | Tokens and APNs dropped after | 30 days | | Home deleted after | 90 days without a roster; 24 hours for a new home without one | @@ -1692,3 +1858,9 @@ codes, Noise key or browser subscriptions. It grants no remote access. Core also rejects tunnels locally in that mode and emits no live or event data. Collection stays off until the signed C8 review; the default operator setting is empty. Withdrawal and retention continue while it is paused. + +Since 24 October 2026 a box without Cloud no longer starts a statistics-only +roster: its statistics are the two anonymous daily reports, through the +Oblivious HTTP relay (ADR 0008, "Anonymous daily reports and one statistics +answer"). Statistics-only rosters already running keep working until the +owner turns statistics off. diff --git a/docs/setup-guide/home-assistant-sv.md b/docs/setup-guide/home-assistant-sv.md index b0b41132a..bfdb4a6f4 100644 --- a/docs/setup-guide/home-assistant-sv.md +++ b/docs/setup-guide/home-assistant-sv.md @@ -44,16 +44,23 @@ Bara en FTW får styra anläggningen. ## 4. Öppna FTW och ställ in -Välj **Öppna webbgränssnittet** (eller `http://:8080`). +Välj **Öppna webbgränssnittet** inne i Home Assistant. Appversioner med +ingress använder din HA-inloggning; du behöver inget eget FTW-lösenord. +Du kan slå på **Visa i sidofältet** på appens sida. HA kräver en administratör +för att öppna FTW. Den direkta webbporten på hemnätverket är stängd. + Följ installationsguiden och lägg till dina enheter ([guide](sv.md)). +Det gäller också en befintlig app med ett sparat FTW-lösenord: efter uppdatering +öppnar du FTW via HA. Inställningar och historik finns kvar. + Ge Home Assistant en fast adress (DHCP-reservation i routern). -Knappen öppnar FTW direkt på hemnätverket. Appen använder inte Home Assistants -ingress. För fjärråtkomst parkopplar du en FTW-klient enligt nästa steg. ## 5. Koppla FTW-appen eller webappen Nya FTW-webappen och iOS-appen kräver nya FTW 0.x, som den här appen ger. -I FTW: **Inställningar → FTW app**, och följ [parkopplingen](../pairing.md). +Ingress ger åtkomst till webbgränssnittet genom HA. Direkt LAN-åtkomst från +en separat FTW-klient ingår inte i detta läge. För en separat klient behöver +du aktivera FTW Cloud och följa [parkopplingen](../pairing.md). Har du använt webappen mot äldre FTW: ta bort den från hemskärmen och parkoppla igen. @@ -67,5 +74,5 @@ parkoppla igen. Native-installationens automatiska återgång finns inte här; HA äger uppdateringen. - Syns inte appen: kontrollera repo-URL:en och ladda om. Misslyckas installationen: se **Inställningar → System → Loggar → Supervisor**. - Öppnas inte webbgränssnittet: kolla appens logg, port 8080 måste vara ledig. + Öppnas inte webbgränssnittet: öppna appen genom HA och kolla appens logg. Ingressport 18099 måste vara ledig. - Fråga i [Discord](https://discord.gg/UK2ygPBu8N) och bifoga appens logg. diff --git a/drivers/BUNDLED_SOURCE.json b/drivers/BUNDLED_SOURCE.json index cf4aefca8..20055645f 100644 --- a/drivers/BUNDLED_SOURCE.json +++ b/drivers/BUNDLED_SOURCE.json @@ -16,7 +16,7 @@ "from the signed channel. Run scripts/sync-bundled-drivers.sh to update." ], "repository": "srcfl/device-drivers", - "commit": "aa0ee46ac071ae8400657b463845735316263cdc", + "commit": "a7b6a3d873f7e5e1fdf63d8a80e576aa1f858754", "source_dir": "drivers/lua", "drivers": [ "ambibox_v2x", "ctek", "ctek_hybrid", "ctek_v2", "deye", "easee_cloud", diff --git a/go/cmd/ftw/app_link.go b/go/cmd/ftw/app_link.go index 7a346bf3e..4ac24905e 100644 --- a/go/cmd/ftw/app_link.go +++ b/go/cmd/ftw/app_link.go @@ -81,7 +81,7 @@ func (a *appSite) Snapshot() appproto.Snapshot { live := a.tel.LiveMeasurementNow(siteMeterDriver, opts) snap := appproto.Snapshot{Mode: mode, ControlRev: rev, FuseA: fuseA, GridW: live.GridW, PVW: live.PVW, BatteryW: live.BatteryW, EVW: live.EVW, - LoadW: live.HouseholdW, LoadWKnown: live.Valid} + LoadW: live.LoadW, LoadWKnown: live.LoadKnown} uptimeMs := time.Since(a.started).Milliseconds() seen := map[string]bool{} @@ -766,11 +766,11 @@ func startAppLink( // the same whichever session reads it. views := agentViews{site: &cloudSite{site: site, api: gateway, cars: carsRT}, cars: carsRT} - // handlerFor builds one handler for one session: a phone's stream, or a - // watch's exchange. They share the ports above, which are read-only - // apart from the mode and the command log, and the mode goes through - // control's own validation. - handlerFor := func() appuplink.HandlerBuilder { + // handlerFor builds one handler for one session: a phone's stream, or + // with exchange set a watch's exchange. They share the ports above, which + // are read-only apart from the mode and the command log, and the mode + // goes through control's own validation. + handlerFor := func(exchange bool) appuplink.HandlerBuilder { return func( sender appproto.Sender, caller apiauth.Caller, @@ -793,6 +793,7 @@ func startAppLink( Caller: caller, Grants: grants, Commands: cmds, + Exchange: exchange, Access: agents.access, Agents: agents.state, Audit: agents.audit, @@ -818,16 +819,17 @@ func startAppLink( } uplink, err := appuplink.New(appuplink.Options{ - Enroll: enroll, - Handler: handlerFor(), - Arrivals: arrivalsFor(ctx, tel), + Enroll: enroll, + Handler: handlerFor(false), + ExchangeHandler: handlerFor(true), + Arrivals: arrivalsFor(ctx, tel), }) if err != nil { return enroll, nil, enabled, err } agents.access.Drop = func(key []byte) { uplink.DropSessionsByAppKey(key) } agents.boxPage = func(by string) (*appproto.Handler, error) { - return handlerFor()(discardSender{}, boxPageCaller(by), boxPageGrant{}) + return handlerFor(false)(discardSender{}, boxPageCaller(by), boxPageGrant{}) } go sweepAgents(ctx, agents) @@ -940,10 +942,10 @@ func (a *appLinkAPI) MintPairingCode(role string) ([]byte, time.Time, error) { return code, expires, appLinkError(err) } -// MintWatchCode issues a watch code bound to the owner phone that asked. -func (a *appLinkAPI) MintWatchCode(parentID string) ([]byte, time.Time, error) { - code, expires, err := a.enroll.MintWatchCode(parentID, appenroll.WatchTTL) - return code, expires, appLinkError(err) +// AddWatch adds a watch by its key under the owner phone that asked. +func (a *appLinkAPI) AddWatch(parentID string, key []byte) (string, error) { + grant, err := a.enroll.AddWatch(parentID, key) + return grant.DeviceID, appLinkError(err) } func (a *appLinkAPI) MintSpokenCode(role string) (string, time.Time, error) { @@ -1068,6 +1070,8 @@ func appLinkError(err error) error { return api.ErrWatchAppRole case errors.Is(err, appenroll.ErrWatchNeedsOwner): return api.ErrWatchNeedsOwner + case errors.Is(err, appenroll.ErrWatchKey): + return api.ErrWatchAppKey default: return err } diff --git a/go/cmd/ftw/bootstrap.go b/go/cmd/ftw/bootstrap.go index d9383bdc7..3b7e90702 100644 --- a/go/cmd/ftw/bootstrap.go +++ b/go/cmd/ftw/bootstrap.go @@ -21,7 +21,11 @@ import ( // runBootstrap serves the setup wizard when config.yaml does not exist yet. func runBootstrap(configPath, webDir, driverDir string) { - slog.Info("no config found — starting setup wizard", "url", "http://localhost:8080/setup") + if homeAssistantIngressEnabled() { + slog.Info("no config found — starting setup wizard", "access", "Home Assistant ingress") + } else { + slog.Info("no config found — starting setup wizard", "url", "http://localhost:8080/setup") + } mux := http.NewServeMux() var initialWriteMu sync.Mutex @@ -171,13 +175,17 @@ func bootstrapAssistant(save func(*config.Config) error, restart func() error) h writeBootstrapJSON(w, http.StatusBadRequest, map[string]string{"error": "send one setup request"}) return } - hash, err := api.HashInitialLANPassword(q.Password) + var hash string + var err error + if !api.HomeAssistantIngress(r) { + hash, err = api.HashInitialLANPassword(q.Password) + } if err != nil { writeBootstrapJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()}) return } cfg := config.NewSetup() - cfg.API.LANAuth, cfg.LANPasswordHash = true, hash + cfg.API.LANAuth, cfg.LANPasswordHash = !api.HomeAssistantIngress(r), hash if err := save(cfg); err != nil { writeBootstrapJSON(w, http.StatusConflict, map[string]string{"error": err.Error()}) return diff --git a/go/cmd/ftw/bootstrap_assistant_test.go b/go/cmd/ftw/bootstrap_assistant_test.go index 6db95984e..2d0ebf776 100644 --- a/go/cmd/ftw/bootstrap_assistant_test.go +++ b/go/cmd/ftw/bootstrap_assistant_test.go @@ -7,6 +7,7 @@ import ( "strings" "testing" + "github.com/srcfl/ftw/go/internal/api" "github.com/srcfl/ftw/go/internal/config" "github.com/srcfl/ftw/go/internal/state" ) @@ -107,3 +108,28 @@ func TestSetupCannotEnableChangedMeterConfiguration(t *testing.T) { t.Fatal("old readings accepted for another endpoint") } } + +func TestHomeAssistantBootstrapNeedsNoSecondPassword(t *testing.T) { + t.Setenv("FTW_API_TOKEN", "") + path := filepath.Join(t.TempDir(), "config.yaml") + restarted := make(chan struct{}, 1) + handler := api.WithHomeAssistantIngress(secureBootstrapMutations(bootstrapAssistant(func(cfg *config.Config) error { return saveInitialConfig(path, cfg) }, func() error { restarted <- struct{}{}; return nil }))) + req := httptest.NewRequest("POST", "http://ha.local/api/setup/assistant", strings.NewReader(`{"confirmed":true}`)) + req.RemoteAddr = "172.30.32.2:1234" + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Ingress-Path", "/api/hassio_ingress/test") + req.Header.Set("X-Remote-User-Id", "owner") + w := httptest.NewRecorder() + handler.ServeHTTP(w, req) + if w.Code != 200 { + t.Fatalf("%d: %s", w.Code, w.Body.String()) + } + <-restarted + cfg, err := config.Load(path) + if err != nil { + t.Fatal(err) + } + if !cfg.MeasurementOnly() || cfg.API.LANAuth || cfg.LANPasswordHash != "" || len(cfg.Drivers) != 0 { + t.Fatal("HA setup invented a password or equipment") + } +} diff --git a/go/cmd/ftw/cloud_link.go b/go/cmd/ftw/cloud_link.go index d9a133573..128d74ffd 100644 --- a/go/cmd/ftw/cloud_link.go +++ b/go/cmd/ftw/cloud_link.go @@ -50,6 +50,9 @@ type cloudSite struct { // With it, the link sends the car class, and a charger's level and // goal fields are views of its session's car. cars *cars.Runtime + // history is the energy ledger, read as the app's history over Noise + // reads it, so a closed tile is the same bytes either way. + history *appHistory sessMu sync.Mutex sessions map[string]*plugSession @@ -63,6 +66,16 @@ type plugSession struct { n int } +var _ cloudlink.HistorySite = (*cloudSite)(nil) + +// HistoryWindow serves the tiles class. +func (c *cloudSite) HistoryWindow(ctx context.Context, names []string, stepMs, fromMs, toMs int64) (map[string][]protocol.HistorySample, error) { + if c.history == nil { + return nil, errors.New("no energy ledger") + } + return c.history.Window(ctx, names, stepMs, fromMs, toMs) +} + func (c *cloudSite) Statistics(ctx context.Context, now time.Time) (cloudwire.Stats, bool) { if srv := c.api.srv.Load(); srv != nil { return srv.CloudStatistics(ctx, now) @@ -384,17 +397,18 @@ func (n *cloudNotices) notify(kind string) { // enrolment, which holds the phones the roster lists; without either it // returns nil and the routes say FTW Cloud is unavailable. tunnel is the app // uplink's session table, nil when the app link is off, and then the box -// takes no tun and links only for a grant. +// takes no tun and links only for a grant. exchange is the watches' door, nil +// with the app link off; it rides beside the tunnel. func startCloudLink( ctx context.Context, statePath, build string, st *state.Store, enroll *appenroll.Identity, tel *telemetry.Store, ctrl *control.State, ctrlMu *sync.Mutex, revision *control.Revision, siteMeterStale time.Duration, measurementOptions func() telemetry.ForecastOptions, priceSvc *prices.Service, gateway *lateAPI, notices *cloudNotices, - tunnel *appuplink.Tunnel, carsRT *cars.Runtime, webPush *notifications.WebPush, + tunnel *appuplink.Tunnel, exchange *appuplink.Exchange, carsRT *cars.Runtime, webPush *notifications.WebPush, ) *cloudlink.Link { if st == nil || enroll == nil { return nil } - site := &cloudSite{api: gateway, cars: carsRT, + site := &cloudSite{api: gateway, cars: carsRT, history: &appHistory{st: st}, site: newAppSite(tel, ctrl, ctrlMu, revision, siteMeterStale, measurementOptions)} if priceSvc != nil && priceSvc.Zone != "" && priceSvc.Store != nil { site.prices = &appPrices{svc: priceSvc} @@ -417,6 +431,9 @@ func startCloudLink( if tunnel != nil { opts.Tunnel = tunnel } + if exchange != nil { + opts.Exchange = exchange + } link, err := cloudlink.New(opts) if err != nil { slog.Warn("FTW Cloud unavailable", "err", err) @@ -425,6 +442,9 @@ func startCloudLink( if tunnel != nil { tunnel.Attach(link) } + if exchange != nil { + exchange.Attach(link) + } enroll.OnChange(link.RosterChanged) // A code shown links the box at once, so a QR that names the home can. enroll.OnPairingCode(link.RosterChanged) @@ -468,6 +488,11 @@ func (c *cloudSession) caps(caps []string) []string { if link.TakesTun() { caps = append(caps, appproto.CapCloudTun) } + // A watch reaches the box through FTW Cloud's exchange, so the phone + // offers to add one only while the link carries it. + if link.TakesX() { + caps = append(caps, appproto.CapCmdExchange) + } return caps } diff --git a/go/cmd/ftw/cloud_roster_test.go b/go/cmd/ftw/cloud_roster_test.go index 6f6d9ab78..9b3673a2d 100644 --- a/go/cmd/ftw/cloud_roster_test.go +++ b/go/cmd/ftw/cloud_roster_test.go @@ -53,11 +53,7 @@ func TestTheRosterNarrowsAWatchToWhatItsPhoneHolds(t *testing.T) { } rosterPhone(t, id, apiauth.RoleOwner) phone := rosterPhone(t, id, apiauth.RoleOwner) - code, _, err := id.MintWatchCode(phone.DeviceID, appenroll.WatchTTL) - if err != nil { - t.Fatal(err) - } - watch, err := id.AuthoriseWatch(rosterKey(t), code) + watch, err := id.AddWatch(phone.DeviceID, rosterKey(t)) if err != nil { t.Fatal(err) } diff --git a/go/cmd/ftw/display_load.go b/go/cmd/ftw/display_load.go new file mode 100644 index 000000000..073ceade6 --- /dev/null +++ b/go/cmd/ftw/display_load.go @@ -0,0 +1,26 @@ +package main + +import ( + "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// Display load is stored as a Core metric series, apart from every driver. +const ( + displayLoadDriver = "ftw:site" + displayLoadMetric = "load_display_w" +) + +// displayLoadSample is the live Home use clients show, for the metric store. +// Nothing while the site meter is unknown: that is what clients show too. +func displayLoadSample(tel *telemetry.Store, siteMeter string, nowMs int64, options telemetry.ForecastOptions, sun historySun) (state.Sample, bool) { + if tel == nil { + return state.Sample{}, false + } + options.PVDark = sun.darkness + live := tel.LiveMeasurementNow(siteMeter, options) + if !live.LoadKnown { + return state.Sample{}, false + } + return state.Sample{Driver: displayLoadDriver, Metric: displayLoadMetric, TsMs: nowMs, Value: live.LoadW, Unit: "W"}, true +} diff --git a/go/cmd/ftw/display_load_test.go b/go/cmd/ftw/display_load_test.go new file mode 100644 index 000000000..4d45690dc --- /dev/null +++ b/go/cmd/ftw/display_load_test.go @@ -0,0 +1,74 @@ +package main + +import ( + "context" + "path/filepath" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// Two series: the live display load is logged as a Core metric exactly as +// clients show it (negative included). The history tick's Load stays the +// placeholder the source-energy writer replaces, so training, which reads +// HouseholdIntervals, never sees the display value. +func TestDisplayLoadIsLoggedApartFromTrainingLoad(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + if err := st.EnableHistoryAggregation(); err != nil { + t.Fatal(err) + } + tel := telemetry.NewStore() + for _, d := range []string{"meter", "battery"} { + tel.EnsureDriverHealth(d) + } + tel.Update("meter", telemetry.DerMeter, 1200, nil, nil) + tel.Update("battery", telemetry.DerBattery, 2000, nil, nil) // negative balance + tel.RecordDriverSuccess("meter") + tel.RecordDriverSuccess("battery") + ctrl := tickPersistControl{SiteMeterDriver: "meter"} + at := time.Now().UnixMilli() + if _, err := persistTelemetryTick(st, tel, ctrl, at, time.Minute, testEnergyIdentity(st)); err != nil { + t.Fatal(err) + } + if err := st.FlushHistory(context.Background()); err != nil { + t.Fatal(err) + } + var got []state.Sample + for i := 0; i < 50 && len(got) == 0; i++ { + got, err = st.LoadSeries(displayLoadDriver, displayLoadMetric, at-1, at+1, 0) + if err != nil { + t.Fatal(err) + } + if len(got) == 0 { + time.Sleep(20 * time.Millisecond) + } + } + if len(got) != 1 || got[0].Value != -800 { + t.Fatalf("display load series = %+v, want one -800 W sample", got) + } + history, err := st.LoadHistory(at-1, at+1, 0) + if err != nil { + t.Fatal(err) + } + for _, h := range history { + if h.Missing&state.MissingLoad == 0 || h.LoadW != 0 { + t.Fatalf("display load leaked into the history Load column: %+v", h) + } + } + if intervals, _, err := st.HouseholdIntervals(context.Background(), 0, at, 15*60000); err != nil || len(intervals) != 0 { + t.Fatalf("training intervals from one display tick: %v %v", intervals, err) + } +} + +func TestDisplayLoadNotLoggedWithoutSiteMeter(t *testing.T) { + tel := telemetry.NewStore() + if _, ok := displayLoadSample(tel, "meter", time.Now().UnixMilli(), telemetry.ForecastOptions{}, historySun{}); ok { + t.Fatal("display load logged without a site meter") + } +} diff --git a/go/cmd/ftw/energy_report.go b/go/cmd/ftw/energy_report.go new file mode 100644 index 000000000..e40d8b84b --- /dev/null +++ b/go/cmd/ftw/energy_report.go @@ -0,0 +1,49 @@ +package main + +import ( + "context" + "log/slog" + "sync" + + "github.com/srcfl/ftw/go/internal/cloudlink" + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/energyreport" + "github.com/srcfl/ftw/go/internal/state" +) + +// The anonymous daily energy report, wired to the box. What it sends and how +// is decided in go/internal/energyreport; this file only gives it the ledger +// and the owner's answer. + +// startEnergyReport builds the sender and starts its schedule. It is built +// even when nothing may be sent, so Settings can show the report this box +// would send. A build with no relay or gateway key sends nothing at all. +func startEnergyReport(ctx context.Context, cfg *config.Config, cfgMu *sync.RWMutex, st *state.Store, answer statsAnswer) *energyreport.Sender { + if st == nil { + return nil + } + sender, err := energyreport.New(energyreport.Options{ + Enabled: energyReportAllowed(cfg, cfgMu, answer), + Ledger: st, + KV: st, + ClockSynced: cloudlink.ClockSynced, + }) + if err != nil { + slog.Warn("energy report unavailable", "err", err) + return nil + } + go sender.Run(ctx) + return sender +} + +// energyReportAllowed is the fleet report's answer, and no site a partner +// controls: as for the hourly project statistics, a partner's contract has to +// let such a home in first. +func energyReportAllowed(cfg *config.Config, cfgMu *sync.RWMutex, answer statsAnswer) func() bool { + return func() bool { + cfgMu.RLock() + partner := cfg.Nova != nil && cfg.Nova.Enabled + cfgMu.RUnlock() + return !partner && answer.shares() + } +} diff --git a/go/cmd/ftw/energy_report_test.go b/go/cmd/ftw/energy_report_test.go new file mode 100644 index 000000000..f2423a58f --- /dev/null +++ b/go/cmd/ftw/energy_report_test.go @@ -0,0 +1,134 @@ +package main + +import ( + "encoding/json" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/fleetping" + "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/statsrelay" + "github.com/srcfl/ftw/go/internal/statsrelay/statsrelaytest" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// Both reports read one answer: part of FTW Cloud, on for a box without it +// until the owner turns it off. The energy report also leaves out a site a +// partner controls. +func TestBothReportsReadTheOneStatisticsAnswer(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + cfg := &config.Config{FleetPing: &config.FleetPing{Enabled: true}} + cfgMu := &sync.RWMutex{} + answer := statsAnswer{st: st} + energy, fleet := energyReportAllowed(cfg, cfgMu, answer), answer.shares + check := func(name string, wantEnergy, wantFleet bool) { + t.Helper() + if energy() != wantEnergy || fleet() != wantFleet { + t.Fatalf("%s: energy %v fleet %v, want %v %v", name, energy(), fleet(), wantEnergy, wantFleet) + } + } + + check("before the stored state is read", false, false) + migrateStatistics(cfg, cfgMu, st, nil) + check("a box without Cloud, never asked", true, true) + + // The old switch is read once and is no longer a second setting. + cfg.FleetPing.Enabled = false + check("the retired fleet switch", true, true) + + cfg.Nova = &config.Nova{Enabled: true} + check("a partner controls the site", false, true) + cfg.Nova = nil + + if err := st.SaveConfig("statistics/share_v1", "off"); err != nil { + t.Fatal(err) + } + check("the owner turned statistics off", false, false) + + // A stored Cloud grant counts only through a running link, which honours + // it only with its key (cloudlink's tests). Without one, the box's own + // answer stands. + grant, _ := json.Marshal(map[string]any{"policy": cloudwire.CloudTermsPolicy, "purposes": []string{"serve", "stats"}}) + if err := st.SaveConfig("cloud/grant_v1", string(grant)); err != nil { + t.Fatal(err) + } + check("a stored grant with no link", false, false) +} + +// A box that shares only by default posts nothing straight to FTW Cloud: its +// fleet report waits for the relay. A yes, the relay or the household's own +// collector lets it go. +func TestADefaultBoxSendsNothingStraightToFTWCloud(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + migrateStatistics(&config.Config{}, &sync.RWMutex{}, st, nil) + answer := statsAnswer{st: st} + client, _, _ := statsrelaytest.New(t, time.Now()) + direct, _ := fleetProvider(config.DefaultFleetPingEndpoint, nil) + relayed, _ := fleetProvider(config.DefaultFleetPingEndpoint, client) + own, _ := fleetProvider("https://collector.example.test/fleet", nil) + if fleetReportAllowed(answer, config.DefaultFleetPingEndpoint, direct)() { + t.Fatal("a box that never said yes posted straight to FTW Cloud") + } + if !fleetReportAllowed(answer, config.DefaultFleetPingEndpoint, relayed)() || !fleetReportAllowed(answer, "https://collector.example.test/fleet", own)() { + t.Fatal("the relay or the household's collector was held back") + } + if err := st.SaveConfig("statistics/share_v1", "on"); err != nil { + t.Fatal(err) + } + if !fleetReportAllowed(answer, config.DefaultFleetPingEndpoint, direct)() { + t.Fatal("a yes did not send") + } +} + +// A fleet report switched off before this release stays off. +func TestASwitchedOffFleetReportStaysOff(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + cfg := &config.Config{FleetPing: &config.FleetPing{Enabled: false}} + migrateStatistics(cfg, &sync.RWMutex{}, st, nil) + if (statsAnswer{st: st}).shares() { + t.Fatal("an earlier off was turned on") + } +} + +// A build with a relay sends the fleet report to FTW Cloud only through it. +// A collector the household chose is posted to directly, and a build without +// a relay keeps the direct path. +func TestTheFleetReportTakesTheRelayToFTWCloud(t *testing.T) { + client, _, _ := statsrelaytest.New(t, time.Now()) + for name, c := range map[string]struct { + endpoint string + relay bool + want bool + }{ + "FTW Cloud with a relay": {config.DefaultFleetPingEndpoint, true, true}, + "FTW Cloud without a relay": {config.DefaultFleetPingEndpoint, false, false}, + "a household's collector": {"https://collector.example.test/fleet", true, false}, + } { + var relay *statsrelay.Client + if c.relay { + relay = client + } + p, err := fleetProvider(c.endpoint, relay) + if err != nil { + t.Fatal(err) + } + if _, through := p.(*fleetping.RelayProvider); through != c.want { + t.Fatalf("%s: through the relay %v", name, through) + } + } +} diff --git a/go/cmd/ftw/ev_planning_limits.go b/go/cmd/ftw/ev_planning_limits.go index da187e397..ff7706735 100644 --- a/go/cmd/ftw/ev_planning_limits.go +++ b/go/cmd/ftw/ev_planning_limits.go @@ -5,7 +5,9 @@ import ( "github.com/srcfl/ftw/go/internal/loadpoint" ) -func planningStepsForLoadpoint(st loadpoint.State, cfg *config.Config) []float64 { +// planningLimitsForLoadpoint is the power range and the steps the planner +// gets for a loadpoint's session. +func planningLimitsForLoadpoint(st loadpoint.State, cfg *config.Config) (minW, maxW float64, steps []float64) { lp := loadpoint.Config{MinChargeW: st.MinChargeW, MaxChargeW: st.MaxChargeW, AllowedStepsW: st.AllowedStepsW} for _, configured := range cfg.Loadpoints { if configured.ID == st.ID { @@ -13,5 +15,6 @@ func planningStepsForLoadpoint(st loadpoint.State, cfg *config.Config) []float64 break } } - return loadpoint.PlanningSteps(lp, loadpoint.SiteFuse{MaxAmps: cfg.Fuse.MaxAmps, Voltage: cfg.Fuse.Voltage, PhaseCnt: cfg.Fuse.Phases}) + site := loadpoint.SiteFuse{MaxAmps: cfg.Fuse.MaxAmps, Voltage: cfg.Fuse.Voltage, PhaseCnt: cfg.Fuse.Phases} + return loadpoint.PlanningLimits(st, lp, site) } diff --git a/go/cmd/ftw/ev_planning_limits_test.go b/go/cmd/ftw/ev_planning_limits_test.go index 36795b906..6881ccc39 100644 --- a/go/cmd/ftw/ev_planning_limits_test.go +++ b/go/cmd/ftw/ev_planning_limits_test.go @@ -1,23 +1,54 @@ package main import ( + "math" + "slices" + "testing" + "time" + "github.com/srcfl/ftw/go/internal/config" "github.com/srcfl/ftw/go/internal/loadpoint" - "testing" + "github.com/srcfl/ftw/go/internal/mpc" ) func TestConfiguredEVLimitsReachPlanner(t *testing.T) { cfg := &config.Config{Fuse: config.Fuse{MaxAmps: 16, Phases: 3, Voltage: 230}, Loadpoints: []config.Loadpoint{{ID: "garage", PhaseMode: "1p"}}} st := loadpoint.State{ID: "garage", MinChargeW: 1380, MaxChargeW: 3680} - steps := planningStepsForLoadpoint(st, cfg) + _, _, steps := planningLimitsForLoadpoint(st, cfg) if len(steps) != 12 || steps[1] != 1380 || steps[len(steps)-1] != 3680 { t.Fatalf("1-phase limits lost on 3-phase site: %v", steps) } cfg.Loadpoints[0].PhaseMode = "3p" st.MinChargeW = 4140 st.MaxChargeW = 11000 - steps = planningStepsForLoadpoint(st, cfg) + _, _, steps = planningLimitsForLoadpoint(st, cfg) if steps[1] != 4140 || steps[len(steps)-1] != 10350 { t.Fatalf("3-phase limits lost: %v", steps) } } + +// Device-simulator rig s14: once the car was marked one-phase the plan had +// no charging for it. With the limits main.go gives the planner, the plan +// charges the car in at least one slot at a level Core can order: one of +// the car's levels, three times which is within the charger's range. +func TestOnePhaseCarPlanChargesAtALevelCoreCanOrder(t *testing.T) { + cfg := &config.Config{Fuse: config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230}, Loadpoints: []config.Loadpoint{{ID: "garage", PhaseMode: "auto"}}} + st := loadpoint.State{ID: "garage", MinChargeW: 4140, MaxChargeW: 11000, CarPhases: 1} + minW, maxW, steps := planningLimitsForLoadpoint(st, cfg) + start := time.Now().Truncate(15 * time.Minute) + slots := make([]mpc.Slot, 8) + for i := range slots { + slots[i] = mpc.Slot{StartMs: start.Add(time.Duration(i) * 15 * time.Minute).UnixMilli(), LenMin: 15, PriceOre: 100, + Limits: mpc.PowerLimits{MaxImportW: 17250}} + } + p := mpc.Params{Mode: mpc.ModeArbitrage, CapacityWh: 10000, SoCMin: .1, SoCMax: .9, InitialSoC: .5, ChargeEfficiency: 1, DischargeEfficiency: 1, + Loadpoint: &mpc.LoadpointSpec{ID: "garage", CapacityWh: 60000, Levels: 11, SoCMax: .55, InitialSoC: .5, TargetSoC: .55, TargetSlotIdx: 7, + PluggedIn: true, ChargeEfficiency: 1, MinChargeW: minW, MaxChargeW: maxW, AllowedStepsW: steps}} + orderable := func(w float64) bool { + return w > 0 && slices.ContainsFunc(steps, func(s float64) bool { return math.Abs(s-w) < 1 }) && + 3*w >= st.MinChargeW && 3*w <= st.MaxChargeW + } + if !slices.ContainsFunc(mpc.Optimize(slots, p).Actions, func(a mpc.Action) bool { return orderable(a.LoadpointW) }) { + t.Fatalf("no slot charges the car at a level Core can order (limits %.0f-%.0f W, levels %v)", minW, maxW, steps) + } +} diff --git a/go/cmd/ftw/fleet_ping.go b/go/cmd/ftw/fleet_ping.go index 592ebed7e..37f89fbfd 100644 --- a/go/cmd/ftw/fleet_ping.go +++ b/go/cmd/ftw/fleet_ping.go @@ -6,6 +6,7 @@ import ( "path" "path/filepath" "strconv" + "strings" "sync" "time" @@ -13,7 +14,10 @@ import ( "github.com/srcfl/ftw/go/internal/config" "github.com/srcfl/ftw/go/internal/drivers" "github.com/srcfl/ftw/go/internal/fleetping" + "github.com/srcfl/ftw/go/internal/selfupdate" "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/statsrelay" + "github.com/srcfl/ftw/go/internal/telemetry" ) // The anonymous fleet ping, wired to the box. @@ -62,6 +66,9 @@ func startFleetPing( cfg *config.Config, cfgMu *sync.RWMutex, st *state.Store, + answer statsAnswer, + tel *telemetry.Store, + updates *selfupdate.Checker, version string, managedDriverDir string, ) (*fleetping.Pinger, error) { @@ -69,7 +76,7 @@ func startFleetPing( endpoint := cfg.FleetPing.Resolved() cfgMu.RUnlock() - provider, err := fleetping.NewHTTPProvider(endpoint, nil) + provider, err := fleetProvider(endpoint, statsrelay.Pinned()) if err != nil { return nil, err } @@ -77,15 +84,18 @@ func startFleetPing( installedAt := firstSeen(st, time.Now()) pinger, err := fleetping.New(fleetping.Options{ - Enabled: func() bool { return fleetPingEnabled(cfg, cfgMu)() && cloudlink.StatisticsChoice(st) }, + Enabled: fleetReportAllowed(answer, endpoint, provider), Facts: func() fleetping.Facts { // Drivers can be installed without restarting FTW. Read the signed // catalogue at the same time as the config so tomorrow's report and // the Settings preview both see the install. catalogue := fleetCatalogue(st, managedDriverDir) + health := fleetHealth(st, tel, updates, time.Now()) cfgMu.RLock() defer cfgMu.RUnlock() - return fleetping.FactsFromConfig(cfg, version, catalogue, installedAt) + f := fleetping.FactsFromConfig(cfg, version, catalogue, installedAt) + f.Health = health + return f }, Provider: provider, }) @@ -97,7 +107,114 @@ func startFleetPing( return pinger, nil } -// fleetPingEnabled reads the switch, under the config lock, at each send. +// fleetHealth reads how the box ran lately, for the report's health buckets: +// the planner's last 24 hours from the plans it kept, the drivers' last 24 +// hours of polls once the count covers six of them, and the newest update of +// the last 7 days. A source that is missing or too short, as the updater is +// on a box that does not update itself, reads as nothing to report. +func fleetHealth(st *state.Store, tel *telemetry.Store, updates *selfupdate.Checker, now time.Time) fleetping.Health { + var h fleetping.Health + if st != nil { + plans, slowestMs, fallbacks, err := st.PlannerDay(now.Add(-24 * time.Hour).UnixMilli()) + if err != nil { + slog.Debug("fleet report: planner day unavailable", "err", err) + } else { + h.Plans, h.SlowestSolve, h.Fallbacks = plans, time.Duration(slowestMs*float64(time.Millisecond)), fallbacks + } + } + // Counts that cover under six hours, as after an update's restart, say + // too little: a start-up burst of failed polls would read as trouble. + if tel != nil { + if polls, errors, covered := tel.PollsLastDay(now); covered >= 6*time.Hour { + h.Polls, h.PollErrors = polls, errors + } + } + if updates != nil { + u := updates.Status() + if u.Action == "update" && !u.UpdatedAt.IsZero() && now.Sub(u.UpdatedAt) <= 7*24*time.Hour { + switch u.State { + case "done": + h.Update = "ok" + case "failed": + h.Update = "failed" + } + } + } + return h +} + +// fleetProvider sends the report to FTW Cloud through the relay when this +// build names one, and only then: FTW Cloud then never sees the box's +// address. A build without a relay posts straight to /v1/fleet as before. An +// endpoint the household chose is its own collector and is posted to +// directly. +func fleetProvider(endpoint string, relay *statsrelay.Client) (fleetping.Provider, error) { + if relay != nil && strings.TrimSpace(endpoint) == config.DefaultFleetPingEndpoint { + return fleetping.NewRelayProvider(relay), nil + } + return fleetping.NewHTTPProvider(endpoint, nil) +} + +// statsAnswer is the one answer for both anonymous daily reports, the fleet +// report and the energy report: part of FTW Cloud, and on for a box without +// it until the owner turns it off. One reader, so the two reports cannot come +// to read different answers. It asks the running Cloud link, which counts a +// grant only with the key it was made with; with no link, the box's own +// answer alone. fleet_ping.enabled is no longer a second switch; +// migrateStatistics read it once. +type statsAnswer struct { + st *state.Store + link *cloudlink.Link +} + +func (a statsAnswer) shares() bool { + if a.link != nil { + return a.link.SharesStatistics() + } + return a.st != nil && cloudlink.SharesStatistics(a.st) +} + +func (a statsAnswer) saidYes() bool { + if a.link != nil { + return a.link.SaidYesToStatistics() + } + return a.st != nil && cloudlink.SaidYesToStatistics(a.st) +} + +// fleetReportAllowed is the one answer, with one more rule for the direct +// path (Fredrik, 24 October 2026): a box that shares only by default, never +// having said yes, posts nothing straight to FTW Cloud, which would see its +// address; it waits for the relay. Through the relay, or to a collector the +// household chose, the answer alone decides. +func fleetReportAllowed(answer statsAnswer, endpoint string, provider fleetping.Provider) func() bool { + _, relayed := provider.(*fleetping.RelayProvider) + direct := !relayed && strings.TrimSpace(endpoint) == config.DefaultFleetPingEndpoint + return func() bool { + return answer.shares() && (!direct || answer.saidYes()) + } +} + +// migrateStatistics turns what an earlier release stored into the one +// answer, once: a fleet report switched off in the config stays off. Then +// the Cloud grant follows it, so an answer that came out off also stops an +// hourly upload the grant still carried. +func migrateStatistics(cfg *config.Config, cfgMu *sync.RWMutex, st *state.Store, link *cloudlink.Link) { + if st == nil { + return + } + if err := cloudlink.MigrateStatistics(st, !fleetPingEnabled(cfg, cfgMu)()); err != nil { + slog.Warn("could not record the statistics answer; nothing is sent until it is", "err", err) + return + } + if link != nil { + if err := link.ReconcileStatistics("statistics answer"); err != nil { + slog.Warn("could not make the FTW Cloud grant follow the statistics answer", "err", err) + } + } +} + +// fleetPingEnabled reads the old switch under the config lock. Since the one +// statistics answer it is read once, by migrateStatistics. // // On() and not a second reading of nil. A config posted to /api/config never // passes through applyDefaults and configreload.Apply swaps it in whole, so a diff --git a/go/cmd/ftw/fleet_ping_test.go b/go/cmd/ftw/fleet_ping_test.go index 142136992..cfb02f3a0 100644 --- a/go/cmd/ftw/fleet_ping_test.go +++ b/go/cmd/ftw/fleet_ping_test.go @@ -13,13 +13,17 @@ import ( "strings" "sync" "testing" + "time" "github.com/srcfl/ftw/go/internal/api" "github.com/srcfl/ftw/go/internal/components" "github.com/srcfl/ftw/go/internal/config" "github.com/srcfl/ftw/go/internal/control" "github.com/srcfl/ftw/go/internal/driverrepo" + "github.com/srcfl/ftw/go/internal/fleetping" + "github.com/srcfl/ftw/go/internal/selfupdate" "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/telemetry" ) // A config saved from the Settings screen does not pass through applyDefaults: @@ -117,7 +121,7 @@ func TestRunningFleetPingSeesADriverInstalledLater(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() - pinger, err := startFleetPing(ctx, cfg, cfgMu, store, "v1.4.0", active) + pinger, err := startFleetPing(ctx, cfg, cfgMu, store, statsAnswer{st: store}, nil, nil, "v1.4.0", active) if err != nil { t.Fatalf("startFleetPing: %v", err) } @@ -339,7 +343,7 @@ func TestNoNameTheHouseholdChoseReachesThePostedPing(t *testing.T) { // path unchanged. Payload() is what sendOnce marshals and posts. ctx, cancel := context.WithCancel(context.Background()) cancel() - pinger, err := startFleetPing(ctx, cfg, cfgMu, store, "v1.4.0", active) + pinger, err := startFleetPing(ctx, cfg, cfgMu, store, statsAnswer{st: store}, nil, nil, "v1.4.0", active) if err != nil { t.Fatalf("startFleetPing: %v", err) } @@ -469,7 +473,7 @@ func TestARepositoryRootInsideTheBundledTreeIsNotTheShippedCatalogue(t *testing. ctx, cancel := context.WithCancel(context.Background()) cancel() - pinger, err := startFleetPing(ctx, cfg, cfgMu, store, "v1.4.0", active) + pinger, err := startFleetPing(ctx, cfg, cfgMu, store, statsAnswer{st: store}, nil, nil, "v1.4.0", active) if err != nil { t.Fatalf("startFleetPing: %v", err) } @@ -499,3 +503,53 @@ func slicesContain(haystack []string, needle string) bool { } return false } + +// The report's health comes from the plans the box kept, the drivers' polls +// and the newest update, and leaves only as buckets. A box without an updater +// reads as no update, like a box that had none. +func TestFleetHealthReadsTheBoxsLastDay(t *testing.T) { + store := openStore(t) + now := time.Now() + for i, js := range []string{`{"solver":{"solve_ms":6100,"fallback":true}}`, `{"solver":{"solve_ms":300}}`} { + if err := store.SaveDiagnostic(now.Add(-time.Duration(i+1)*time.Hour).UnixMilli(), "scheduled", "SE3", 0, 96, js); err != nil { + t.Fatal(err) + } + } + tel := telemetry.NewStore() + for range 99 { + tel.CountPoll(false) + } + tel.CountPoll(true) + updates := selfupdate.New(selfupdate.Config{StatusPath: filepath.Join(t.TempDir(), "update-status.json")}, nil) + if err := updates.WriteStatus(selfupdate.UpdateStatus{State: "failed", Action: "update", Target: "v9.9.9", UpdatedAt: now.Add(-48 * time.Hour)}); err != nil { + t.Fatal(err) + } + + // Just after a start the poll count covers too little to say anything. + if h := fleetHealth(store, tel, updates, now); h.Polls != 0 || h.PollErrors != 0 || h.Plans != 2 { + t.Fatalf("a fresh count was reported: %+v", h) + } + // Seven hours on, it covers enough; the polls and plans are still in + // their 24 hours. + h := fleetHealth(store, tel, updates, now.Add(7*time.Hour)) + if h.Plans != 2 || h.SlowestSolve != 6100*time.Millisecond || h.Fallbacks != 1 || h.Polls != 100 || h.PollErrors != 1 || h.Update != "failed" { + t.Fatalf("%+v", h) + } + p := fleetping.Build(fleetping.Facts{Health: h}, now) + if p.Solve != "5s+" || p.Fallbacks != "1-3" || p.Errors != "1%+" || p.Update != "failed" { + t.Fatalf("%+v", p) + } + + // A week on, the failed update is no longer news; a restart is not an + // update at all. + if h := fleetHealth(store, tel, updates, now.Add(6*24*time.Hour)); h.Update != "" { + t.Fatalf("a week-old update still reads %q", h.Update) + } + _ = updates.WriteStatus(selfupdate.UpdateStatus{State: "done", Action: "restart", UpdatedAt: now}) + if h := fleetHealth(store, tel, updates, now); h.Update != "" { + t.Fatalf("a restart reads as an update: %q", h.Update) + } + if h := fleetHealth(store, tel, nil, now); h.Update != "" || h.Plans != 2 { + t.Fatalf("without an updater: %+v", h) + } +} diff --git a/go/cmd/ftw/history_dark_pv.go b/go/cmd/ftw/history_dark_pv.go index 40e0e82e0..4d4de6864 100644 --- a/go/cmd/ftw/history_dark_pv.go +++ b/go/cmd/ftw/history_dark_pv.go @@ -36,6 +36,19 @@ func (s historySun) zenith(atMS int64) (float64, bool) { return sunpos.At(time.UnixMilli(atMS), s.Lat, s.Lon).ZenithDeg, true } +// darkness tells live Home use how dark it is at the site, by the same +// sunset and civil-dusk limits as history. +func (s historySun) darkness(at time.Time) telemetry.Darkness { + z, placed := s.zenith(at.UnixMilli()) + switch { + case !placed || z <= sunsetZenithDeg: + return telemetry.DarknessUnknown + case z <= civilDuskZenithDeg: + return telemetry.DarknessTwilight + } + return telemetry.DarknessNight +} + // darkPV records an unknown PV source as 0 W while the sun is down at the // site: physics, not interpolation. Past civil dusk the 0 W is as good as a // reading and carries no marker, so a night with a sleeping inverter also diff --git a/go/cmd/ftw/history_source_load_test.go b/go/cmd/ftw/history_source_load_test.go index fb290398a..7ee07e270 100644 --- a/go/cmd/ftw/history_source_load_test.go +++ b/go/cmd/ftw/history_source_load_test.go @@ -2,11 +2,15 @@ package main import ( "context" + "encoding/json" "math" "path/filepath" "testing" "time" + "github.com/lorenzodonini/ocpp-go/ocpp1.6/core" + + "github.com/srcfl/ftw/go/internal/ocpp" "github.com/srcfl/ftw/go/internal/state" "github.com/srcfl/ftw/go/internal/telemetry" ) @@ -120,3 +124,186 @@ func TestHistorySourceQualificationDoesNotInventMissingFlows(t *testing.T) { } } } + +// An OCPP charger with no car sends no MeterValues. Its Available status is +// the evidence that nothing flows, so the household stays measured. +func TestHistoryCountsAnEmptyOCPPChargerAsMeasured(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + defer st.Close() + if err := st.EnableHistoryAggregation(); err != nil { + t.Fatal(err) + } + tel := telemetry.NewStore() + h := ocpp.NewHandler(tel, 60) + h.SetApprovedIDs([]string{"charger"}) + h.OnConnect("charger") + if _, err := h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusAvailable, ErrorCode: core.NoError}); err != nil { + t.Fatal(err) + } + tel.Update("grid", telemetry.DerMeter, 2000, nil, nil) + tel.RecordDriverSuccess("grid") + if live := tel.LiveMeasurementNow("grid", telemetry.ForecastOptions{}); live.LoadQuality != telemetry.LoadMeasured || live.LoadW != 2000 { + t.Fatalf("live Home use %s (%s), %v W", live.LoadQuality, live.LoadReason, live.LoadW) + } + base := time.Now().Add(-10 * time.Minute).Truncate(5 * time.Minute).UnixMilli() + for i := int64(0); i <= 30; i++ { + p, _ := buildHistoryPoint(tel, tickPersistControl{SiteMeterDriver: "grid"}, base+i*10000, time.Minute) + if p.Sources.Missing != 0 || p.Missing&^state.MissingSoC != 0 { + t.Fatalf("empty charger marked history %b, sources %b", p.Missing, p.Sources.Missing) + } + p.Sources.AtMS = base + i*10000 + if err := st.RecordTickWithEnergy(p, nil, nil); err != nil { + t.Fatal(err) + } + } + if err := st.FlushHistory(context.Background()); err != nil { + t.Fatal(err) + } + d, err := st.DailyEnergy(base, base+300000) + if err != nil { + t.Fatal(err) + } + if d.Missing&state.MissingLoad != 0 || math.Abs(d.LoadWh-2000.0/12) > 0.001 { + t.Fatalf("household %v Wh, quality %s", d.LoadWh, d.Missing.Quality()) + } +} + +// Between heartbeats an Available charger's 0 W is up to a minute old. The +// time-aligned balance calls that time_skew; the chart's quality comes from +// the source rows, so it must not say the measurements do not balance. +func TestHistoryChartKeepsSourceQualityForAnAgeingOCPPZero(t *testing.T) { + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + defer st.Close() + if err := st.EnableHistoryAggregation(); err != nil { + t.Fatal(err) + } + tel := telemetry.NewStore() + tel.Update("grid", telemetry.DerMeter, 2000, nil, nil) + tel.RecordDriverSuccess("grid") + // What the OCPP handler publishes 40 s after the last heartbeat. + at := time.Now().Add(-40 * time.Second).UnixMilli() + data, _ := json.Marshal(map[string]any{"type": "ev", "w": 0.0, "connected": false, "forecast_power": telemetry.ForecastPowerSample{Version: 1, Known: true, MeasuredAtMS: at, ReceivedAtMS: at}}) + tel.Update("charger", telemetry.DerEV, 0, nil, data) + tel.RecordDriverSuccess("charger") + base := time.Now().Add(-10 * time.Minute).Truncate(5 * time.Minute).UnixMilli() + for i := int64(0); i <= 30; i++ { + p, _ := buildHistoryPoint(tel, tickPersistControl{SiteMeterDriver: "grid"}, base+i*10000, time.Minute) + if p.Missing&state.BalanceMismatch == 0 || p.Sources.Missing != 0 { + t.Fatalf("want a skewed aligned balance and complete sources, got %b and %b", p.Missing, p.Sources.Missing) + } + p.Sources.AtMS = base + i*10000 + if err := st.RecordTickWithEnergy(p, nil, nil); err != nil { + t.Fatal(err) + } + } + if err := st.FlushHistory(context.Background()); err != nil { + t.Fatal(err) + } + points, err := st.LoadHistoryContext(context.Background(), base, base+300000, 0) + if err != nil { + t.Fatal(err) + } + if len(points) == 0 { + t.Fatal("no chart points") + } + for _, p := range points { + if q := p.Missing.Quality(); q == "balance_mismatch" || q == "incomplete" { + t.Fatalf("chart point at %d says %s", p.TsMs, q) + } + } +} + +// An idle charger often sends only heartbeats. At Core's default interval, +// with ±1 s jitter, its Available 0 W is at most 31 s old at any 2 s history +// tick: inside the 60 s limit, and its health never lapses. The old 60 s +// default loses ticks to the same jitter. +func TestHistoryKeepsAnIdleOCPPChargerMeasuredBetweenHeartbeats(t *testing.T) { + var c ocpp.Config + c.Defaults() + if lost := idleOCPPStaleTicks(t, time.Duration(c.HeartbeatIntervalS)*time.Second, true); lost != 0 { + t.Fatalf("default heartbeat %d s: %d ticks lost the charger", c.HeartbeatIntervalS, lost) + } + if lost := idleOCPPStaleTicks(t, time.Minute, false); lost == 0 { + t.Fatal("a 60 s heartbeat lost no tick; the test cannot see the edge") + } +} + +// idleOCPPStaleTicks plays ten heartbeats at interval ±1 s through the real +// handler and counts history ticks whose source row misses a flow. Between +// beats the charger's reading and health are aged by the time since the beat. +func idleOCPPStaleTicks(t *testing.T, interval time.Duration, record bool) int { + t.Helper() + const watchdog = time.Minute // site default; history uses it as its age limit + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + defer st.Close() + if err := st.EnableHistoryAggregation(); err != nil { + t.Fatal(err) + } + tel := telemetry.NewStore() + h := ocpp.NewHandler(tel, int(interval/time.Second)) + h.SetApprovedIDs([]string{"charger"}) + h.OnConnect("charger") + if _, err := h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusAvailable, ErrorCode: core.NoError}); err != nil { + t.Fatal(err) + } + age := func(beat time.Time, since time.Duration) { + r := tel.Get("charger", telemetry.DerEV) + var d map[string]any + if err := json.Unmarshal(r.Data, &d); err != nil { + t.Fatal(err) + } + at := beat.Add(-since) + d["forecast_power"] = telemetry.ForecastPowerSample{Version: 1, Known: true, MeasuredAtMS: at.UnixMilli(), ReceivedAtMS: at.UnixMilli()} + r.Data, _ = json.Marshal(d) + r.UpdatedAt = at + tel.DriverHealthMut("charger").LastSuccess = &at + } + base := time.Now().Add(-time.Hour).Truncate(5 * time.Minute).UnixMilli() + tick, lost := base, 0 + for beat := 0; beat < 10; beat++ { + gap := interval + []time.Duration{-time.Second, 0, time.Second}[beat%3] + if _, err := h.OnHeartbeat("charger", &core.HeartbeatRequest{}); err != nil { + t.Fatal(err) + } + beatAt := time.Now() + for since := time.Duration(beat%4) * 500 * time.Millisecond; since < gap; since += 2 * time.Second { + age(beatAt, since) + tel.Update("grid", telemetry.DerMeter, 2000, nil, nil) + tel.RecordDriverSuccess("grid") + tel.WatchdogScan(watchdog) + p, _ := buildHistoryPoint(tel, tickPersistControl{SiteMeterDriver: "grid"}, tick, watchdog) + if p.Sources.Missing != 0 { + lost++ + } + if record { + p.Sources.AtMS = tick + if err := st.RecordTickWithEnergy(p, nil, nil); err != nil { + t.Fatal(err) + } + } + tick += 2000 + } + } + if record { + if err := st.FlushHistory(context.Background()); err != nil { + t.Fatal(err) + } + d, err := st.DailyEnergy(base, tick) + if err != nil { + t.Fatal(err) + } + if d.Missing&(state.MissingLoad|state.MissingEV) != 0 || d.LoadWh <= 0 { + t.Fatalf("history over %d s: %v Wh, quality %s", (tick-base)/1000, d.LoadWh, d.Missing.Quality()) + } + } + return lost +} diff --git a/go/cmd/ftw/http_server.go b/go/cmd/ftw/http_server.go index 92f8673f8..88e86b8d7 100644 --- a/go/cmd/ftw/http_server.go +++ b/go/cmd/ftw/http_server.go @@ -2,7 +2,10 @@ package main import ( "net/http" + "os" "time" + + "github.com/srcfl/ftw/go/internal/api" ) const ( @@ -13,7 +16,15 @@ const ( httpIdleTimeout = 60 * time.Second ) +func homeAssistantIngressEnabled() bool { + return os.Getenv("FTW_BUNDLE") == "home_assistant_addon" && os.Getenv("FTW_HA_INGRESS") == "1" +} + func newHTTPServer(addr string, handler http.Handler) *http.Server { + if homeAssistantIngressEnabled() { + addr = ":18099" + handler = api.WithHomeAssistantIngress(handler) + } return &http.Server{ Addr: addr, Handler: handler, diff --git a/go/cmd/ftw/http_server_test.go b/go/cmd/ftw/http_server_test.go index 24a01a0d4..3b148845d 100644 --- a/go/cmd/ftw/http_server_test.go +++ b/go/cmd/ftw/http_server_test.go @@ -2,6 +2,7 @@ package main import ( "net/http" + "net/http/httptest" "testing" "time" ) @@ -21,3 +22,25 @@ func TestAPIHTTPServerSetsRequestTimeouts(t *testing.T) { t.Errorf("IdleTimeout = %v, want 60s", srv.IdleTimeout) } } + +func TestHomeAssistantBundleUsesAnIngressOnlyListener(t *testing.T) { + t.Setenv("FTW_BUNDLE", "home_assistant_addon") + t.Setenv("FTW_HA_INGRESS", "1") + server := newHTTPServer(":8080", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(204) })) + if server.Addr != ":18099" { + t.Fatal(server.Addr) + } + for _, path := range []string{"/", "/setup", "/api/config"} { + req := httptest.NewRequest("GET", "http://ftw.local"+path, nil) + req.RemoteAddr = "192.168.1.10:1234" + w := httptest.NewRecorder() + server.Handler.ServeHTTP(w, req) + if w.Code != 403 { + t.Fatal("LAN UI exposed", path, w.Code) + } + } + t.Setenv("FTW_BUNDLE", "") + if newHTTPServer(":8080", http.NotFoundHandler()).Addr != ":8080" { + t.Fatal("ordinary install changed") + } +} diff --git a/go/cmd/ftw/live_load_dark_test.go b/go/cmd/ftw/live_load_dark_test.go new file mode 100644 index 000000000..363f5043d --- /dev/null +++ b/go/cmd/ftw/live_load_dark_test.go @@ -0,0 +1,37 @@ +package main + +import ( + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/telemetry" +) + +func TestHistorySunDarknessForLiveLoad(t *testing.T) { + sun := historySunOf(59.33, 18.07, true) // Stockholm + cases := map[string]telemetry.Darkness{ + "2026-06-21T12:00:00+02:00": telemetry.DarknessUnknown, + "2026-10-21T02:00:00+02:00": telemetry.DarknessNight, + } + for at, want := range cases { + ts, _ := time.Parse(time.RFC3339, at) + if got := sun.darkness(ts); got != want { + t.Fatalf("%s: darkness %d, want %d", at, got, want) + } + } + if (historySun{}).darkness(time.Now()) != telemetry.DarknessUnknown { + t.Fatal("an unplaced site inferred darkness") + } + // Through an October evening: daylight, then twilight, then night, in order. + start, _ := time.Parse(time.RFC3339, "2026-10-21T15:00:00+02:00") + seen := []telemetry.Darkness{} + for m := 0; m <= 300; m++ { + d := sun.darkness(start.Add(time.Duration(m) * time.Minute)) + if len(seen) == 0 || seen[len(seen)-1] != d { + seen = append(seen, d) + } + } + if len(seen) != 3 || seen[0] != telemetry.DarknessUnknown || seen[1] != telemetry.DarknessTwilight || seen[2] != telemetry.DarknessNight { + t.Fatalf("evening sequence %v", seen) + } +} diff --git a/go/cmd/ftw/live_load_test.go b/go/cmd/ftw/live_load_test.go index f7535cb6b..c1f9a22f7 100644 --- a/go/cmd/ftw/live_load_test.go +++ b/go/cmd/ftw/live_load_test.go @@ -76,11 +76,12 @@ func TestLiveLoadAppAndLANShareSourcePowerAndValidity(t *testing.T) { want = 1500 case "negative": tel.Update("battery", telemetry.DerBattery, 2000, nil, nil) - valid = false + want = -1000 // shown as is: production Core cannot see case "offline_ev": tel.Update("charger", telemetry.DerEV, 500, nil, nil) tel.DriverHealthMut("charger").SetOffline() - valid = false + // Left out, as before #1592: Home use is an estimate, not blank. + want = 1000 case "idle_ev", "paused_ev": fields := map[string]any{"connected": scenario == "paused_ev", "charging": false, "control_power_observed_at": time.Now().Add(-time.Hour).Format(time.RFC3339Nano), @@ -136,11 +137,16 @@ func TestLiveLoadOptionsFollowConfigurationForAppLANAndCloud(t *testing.T) { snap := site.Snapshot() status := liveLoadStatus(t, tel, ctrl, options) _, cloudKnown := (&cloudSite{site: site}).Live().Fields[strconv.Itoa(int(appproto.FidLoadW))] - known := !batteryDeclared - if snap.LoadWKnown != known || (status["load_w"] != nil) != known || cloudKnown != known { + // A declared battery that never reported is left out: Home use stays + // known on every client and is marked estimated on the LAN. + if !snap.LoadWKnown || status["load_w"] == nil || !cloudKnown { t.Fatal("configured never-emitted flow differs between clients") } - if known && (snap.LoadW != 1000 || status["load_w"] != float64(1000)) { + wantQuality := map[bool]string{true: "estimated", false: "measured"}[batteryDeclared] + if status["load_quality"] != wantQuality { + t.Fatalf("load_quality = %v, want %s", status["load_quality"], wantQuality) + } + if snap.LoadW != 1000 || status["load_w"] != float64(1000) { t.Fatal("learning-only identity gate affected live load") } if batteryDeclared { diff --git a/go/cmd/ftw/main.go b/go/cmd/ftw/main.go index e62a67454..aea4e3c3e 100644 --- a/go/cmd/ftw/main.go +++ b/go/cmd/ftw/main.go @@ -41,6 +41,7 @@ import ( "github.com/srcfl/ftw/go/internal/api" "github.com/srcfl/ftw/go/internal/appuplink" "github.com/srcfl/ftw/go/internal/arp" + "github.com/srcfl/ftw/go/internal/backup" "github.com/srcfl/ftw/go/internal/battery" "github.com/srcfl/ftw/go/internal/cars" "github.com/srcfl/ftw/go/internal/components" @@ -448,6 +449,9 @@ func main() { backupDir = filepath.Join(dataDir, backupDir) } } + // A backup stopped with Core leaves its scratch; a large one in /dev/shm + // sends the next backup's copy to the data disk. + backup.RemoveStaleScratch(backupDir) if abs, err := filepath.Abs(coldDir); err == nil { coldDir = abs } @@ -858,7 +862,10 @@ func main() { } forecastSettings := newForecastSiteConfig(st) liveMeasurementOptions := func() telemetry.ForecastOptions { - return forecastSettings.Snapshot().HistoryOptions + site := forecastSettings.Snapshot() + opts := site.HistoryOptions + opts.PVDark = site.HistorySun.darkness + return opts } forecastSettings.identity = func(name string) (string, bool) { if id, ok := runningDeviceID(reg, name); ok { @@ -1782,7 +1789,7 @@ func main() { noBatteryToEV := !(ctrl.BatteryCoversEV || boostActive) ctrlMu.Unlock() charging, chargeDuration := lpMgr.ChargingPeriod(st.ID) - minW, maxW, steps := loadpoint.LimitPower(st, planningStepsForLoadpoint(st, cfg)) + minW, maxW, steps := planningLimitsForLoadpoint(st, cfg) specs = append(specs, &mpc.LoadpointSpec{ ID: st.ID, CapacityWh: capWh, @@ -1995,7 +2002,23 @@ func main() { lpController.SetFuseEVMax(func() (float64, bool) { ctrlMu.Lock() defer ctrlMu.Unlock() - return control.EVFuseBudget(tel, ctrl, lpMgr.States(), lpMgr.Configs(), ctrl.SiteFuseAmps*ctrl.SiteFuseVoltage*float64(ctrl.SiteFusePhases)) + // The cap is measured for the phases this tick's commands name. + return control.EVFuseBudget(tel, ctrl, lpMgr.States(), lpController.FuseConfigs(), ctrl.SiteFuseAmps*ctrl.SiteFuseVoltage*float64(ctrl.SiteFusePhases)) + }) + lpController.SetFusePhaseRoomA(func() (float64, bool) { + ctrlMu.Lock() + defer ctrlMu.Unlock() + return control.EVFusePhaseRoomA(ctrl) + }) + lpController.SetFuseLimitReason(func() string { + ctrlMu.Lock() + defer ctrlMu.Unlock() + return control.EVLimitReason(ctrl) + }) + lpController.SetSitePhases(func() (loadpoint.SitePhases, bool) { + ctrlMu.Lock() + defer ctrlMu.Unlock() + return control.SitePhases(tel, ctrl) }) // Every lease is preflighted and re-evaluated on each loadpoint tick. // This closure owns site/battery health because those concepts belong @@ -2003,7 +2026,7 @@ func main() { lpController.SetBatteryBoostSafety(func(id string, lease loadpoint.BatteryBoostLease) loadpoint.BatteryBoostStopReason { ctrlMu.Lock() mode := ctrl.Mode - fuseBlocked := ctrl.FuseSaturated + fuseBlocked, importBlocked := ctrl.FuseSaturated, ctrl.FuseSaturatedByImport _, batteryHeld := ctrl.GetBatteryManualHold(time.Now()) siteMeter := ctrl.SiteMeterDriver ctrlMu.Unlock() @@ -2013,6 +2036,9 @@ func main() { if batteryHeld { return loadpoint.BatteryBoostStoppedBatteryHold } + if fuseBlocked && importBlocked { + return loadpoint.BatteryBoostStoppedImportLimit + } if fuseBlocked { return loadpoint.BatteryBoostStoppedFuseSafety } @@ -2123,31 +2149,6 @@ func main() { go mpcSvc.ReplanWithReason(context.Background(), "loadpoint_battery_boost_stopped") slog.Info("battery boost stopped — replan requested", "lp", id, "reason", reason) }) - // Wire the live per-phase site-meter current reader. The EV hard - // limit (control.EVFuseBudget) cuts a worst-phase overage; this - // reader lets the loadpoint gate an increase on the phase a - // runtime 1p or auto charger may land on. Reads the same - // meter_lN_a metrics the fuse_over_limit notifier uses. - lpController.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { - cfgMu.RLock() - siteMeter := cfg.SiteMeterDriver() - phaseCount := cfg.Fuse.Phases - watchdogTimeout := time.Duration(cfg.Site.WatchdogTimeoutS) * time.Second - cfgMu.RUnlock() - if siteMeter == "" { - return 0, 0, 0, false - } - if watchdogTimeout <= 0 { - watchdogTimeout = 60 * time.Second - } - amps, available, fresh := sitePhaseCurrentsAt( - tel, siteMeter, phaseCount, watchdogTimeout, time.Now(), - ) - if !available || !fresh { - return 0, 0, 0, false - } - return amps[0], amps[1], amps[2], true - }) // Wire the matched-vehicle reader for auto-wake. When the // loadpoint is commanding power but the matched Tesla // reports `Stopped` / `Disconnected` / `Complete`, the @@ -2472,11 +2473,13 @@ func main() { // signed, readable copy of the house, off until an owner turns it on. // Both use the one FTW Cloud connection. var appTunnel *appuplink.Tunnel + var appExchange *appuplink.Exchange if appUplink != nil { appTunnel = appUplink.Tunnel() + appExchange = appUplink.Exchange() } cloudLink := startCloudLink(ctx, statePath, Version, st, appEnroll, - tel, ctrl, ctrlMu, controlRev, appLinkWatchdog, liveMeasurementOptions, priceSvc, appAPI, cloudNotify, appTunnel, carsRT, webPush) + tel, ctrl, ctrlMu, controlRev, appLinkWatchdog, liveMeasurementOptions, priceSvc, appAPI, cloudNotify, appTunnel, appExchange, carsRT, webPush) appCloud.bind(cloudLink) if cloudLink != nil { agents.home = cloudLink.Home @@ -2489,12 +2492,17 @@ func main() { // The bundled drivers dir is deliberately not passed: which drivers this // build ships with is compiled in, not discovered on disk. See // fleetCatalogue for what that buys. + migrateStatistics(cfg, cfgMu, st, cloudLink) + answer := statsAnswer{st: st, link: cloudLink} fleetPinger, fleetPingErr := startFleetPing( - ctx, cfg, cfgMu, st, Version, config.ManagedDriversDirOverride, + ctx, cfg, cfgMu, st, answer, tel, selfUpdater, Version, config.ManagedDriversDirOverride, ) if fleetPingErr != nil { slog.Warn("fleet ping unavailable", "err", fleetPingErr) } + // The anonymous daily energy report, under the same answer, through an + // independent relay so that FTW Cloud never learns which box sent it. + energyReport := startEnergyReport(ctx, cfg, cfgMu, st, answer) mutationPolicy := apiMutationPolicy() mutationPolicy.LANAuthEnabled = func() bool { @@ -2517,6 +2525,7 @@ func main() { AppEnroll: appEnrollForAPI(appEnroll, appUplink, st, appLinkEnabled, agents), Agents: agentsForAPI(agents), FleetPing: fleetPinger, + EnergyReport: energyReport, Cloud: cloudForAPI(cloudLink), CfgMu: cfgMu, Cfg: cfg, ConfigPath: *configPath, ConfigApplier: applyConfigChange, @@ -3169,6 +3178,8 @@ func main() { "ev_w", evW, "v2x_w", v2xW, "load_valid", live.Valid, + "load_quality", live.LoadQuality, + "load_reason", live.LoadReason, "self_tune_active", selfTuneActive, "self_tune_driver", selfTuneName, "self_tune_command_w", selfTuneCmd, @@ -4031,6 +4042,12 @@ func persistTelemetryTick(st *state.Store, tel *telemetry.Store, ctrl tickPersis Value: sm.Value, Unit: sm.Unit, } } + // Display load: the live balance exactly as clients show it. A record of + // what was on screen, never a training label; models read the household + // energy intervals the source writer closes (HouseholdIntervals). + if display, ok := displayLoadSample(tel, ctrl.SiteMeterDriver, nowMs, options, sun); ok { + stSamples = append(stSamples, display) + } energyObservations := buildEnergyObservations(st, tel, ctrl, hp, identity) if hp.Sources != nil { hp.Sources.Topology += historyDevices(st, tel, identity, energyObservations, dark) diff --git a/go/go.mod b/go/go.mod index 9df9d7ae9..e14af69fc 100644 --- a/go/go.mod +++ b/go/go.mod @@ -13,6 +13,7 @@ require ( github.com/quic-go/quic-go v0.63.0 github.com/shirou/gopsutil/v4 v4.26.8 github.com/simonvetter/modbus v1.6.4 + github.com/srcfl/device-simulator v0.5.1-0.20261023051213-736051fbcf91 github.com/yuin/gopher-lua v1.1.2 golang.org/x/crypto v0.57.0 golang.org/x/net v0.58.0 @@ -35,7 +36,7 @@ require ( github.com/andybalholm/brotli v1.2.0 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/ebitengine/purego v0.10.2 // indirect - github.com/go-ole/go-ole v1.2.6 // indirect + github.com/go-ole/go-ole v1.3.0 // indirect github.com/klauspost/compress v1.18.3 // indirect github.com/lorenzodonini/ocpp-go v0.19.0 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect diff --git a/go/go.sum b/go/go.sum index 62dd33714..6f807ddf0 100644 --- a/go/go.sum +++ b/go/go.sum @@ -20,8 +20,9 @@ github.com/eclipse/paho.mqtt.golang v1.5.1 h1:/VSOv3oDLlpqR2Epjn1Q7b2bSTplJIeV2I github.com/eclipse/paho.mqtt.golang v1.5.1/go.mod h1:1/yJCneuyOoCOzKSsOTUc0AJfpsItBGWvYpBLimhArU= github.com/fxamacker/cbor/v2 v2.9.4 h1:xwjVlxEMR3S605oUlgBjKLTTeGFciYPGYCtF/35LKGo= github.com/fxamacker/cbor/v2 v2.9.4/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ= -github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= +github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= +github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= github.com/go-playground/locales v0.12.1 h1:2FITxuFt/xuCNP1Acdhv62OzaCiviiE4kotfhkmOqEc= github.com/go-playground/locales v0.12.1/go.mod h1:IUMDtCfWo/w/mtMfIE/IG2K+Ey3ygWanZIBtBW0W2TM= github.com/go-playground/universal-translator v0.16.0 h1:X++omBR/4cE2MNg91AoC3rmGrCjJ8eAeUP/K/EKx4DM= @@ -95,6 +96,8 @@ github.com/shirou/gopsutil/v4 v4.26.8/go.mod h1:5O9FjBiXoTDFatIWjZZosqj4pV0DRtLx github.com/simonvetter/modbus v1.6.4 h1:E03lBz/JftDza/+Ue+vxwkNZ/WW1xiqyFCUQ4NhqHn0= github.com/simonvetter/modbus v1.6.4/go.mod h1:hh90ZaTaPLcK2REj6/fpTbiV0J6S7GWmd8q+GVRObPw= github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE= +github.com/srcfl/device-simulator v0.5.1-0.20261023051213-736051fbcf91 h1:86F3THGp+USxwiOIYxBqKlxLhitGZObZpzHIfPtyFHk= +github.com/srcfl/device-simulator v0.5.1-0.20261023051213-736051fbcf91/go.mod h1:KXJy4w3VMimpLm2LLm2Nr8lnKtBwUbxKAQRGKD1GnQk= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= @@ -135,6 +138,7 @@ golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20220804214406-8e32c043e418/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/tools v0.48.0 h1:3+hClM1aLL5mjMKm5ovokw9epgRXPuu2tILgismM6RE= diff --git a/go/internal/api/api.go b/go/internal/api/api.go index d9e2a9b4b..ecc4b96a1 100644 --- a/go/internal/api/api.go +++ b/go/internal/api/api.go @@ -41,6 +41,7 @@ import ( "github.com/srcfl/ftw/go/internal/coverage" "github.com/srcfl/ftw/go/internal/driverrepo" "github.com/srcfl/ftw/go/internal/drivers" + "github.com/srcfl/ftw/go/internal/energyreport" "github.com/srcfl/ftw/go/internal/evcloud" "github.com/srcfl/ftw/go/internal/events" "github.com/srcfl/ftw/go/internal/fleetping" @@ -115,6 +116,10 @@ type Deps struct { // embeddings; GET /api/fleet-ping then says so. FleetPing *fleetping.Pinger + // EnergyReport is the anonymous daily energy report, held for the same + // reason: Settings shows the report it would send. Nil leaves it out. + EnergyReport *energyreport.Sender + // Cloud is the box's FTW Cloud grant and uplink. Nil in tests and // minimal embeddings; the /api/cloud routes then say it is unavailable. Cloud CloudGrant diff --git a/go/internal/api/api_app_link.go b/go/internal/api/api_app_link.go index 4b025e2d5..c006963eb 100644 --- a/go/internal/api/api_app_link.go +++ b/go/internal/api/api_app_link.go @@ -1,6 +1,7 @@ package api import ( + "encoding/base64" "errors" "net" "net/http" @@ -39,11 +40,11 @@ type AppEnroller interface { // MintSpokenCode issues a box code: eight characters somebody can read // down a phone. Same single-use, same replacement of any live code. MintSpokenCode(role string) (string, time.Time, error) - // MintWatchCode issues a code that lets one watch in, bound to the owner - // phone named by parentID. It has a slot of its own, so it leaves a - // pending phone code alone. Returns ErrWatchNeedsOwner when that phone is - // not an owner now. - MintWatchCode(parentID string) ([]byte, time.Time, error) + // AddWatch stores a watch's Noise key under the owner phone named by + // parentID, and returns the watch's device id. Returns + // ErrWatchNeedsOwner when that phone is not an owner now, and + // ErrWatchAppKey for a key that cannot be a watch's. + AddWatch(parentID string, key []byte) (string, error) // EnrollmentURL is what goes in the QR. EnrollmentURL(code []byte, lanHint string) (string, error) // AuthorisedCount is how many phones may currently connect. Watches @@ -97,9 +98,11 @@ var ( // ErrWatchAppRole is the watch role asked for where it cannot be given: // a phone's code, or a role change to or from watch. ErrWatchAppRole = errors.New("api: a watch comes only from its phone") - // ErrWatchNeedsOwner is a watch code asked for by a phone that is not an - // owner. + // ErrWatchNeedsOwner is a watch added by a phone that is not an owner. ErrWatchNeedsOwner = errors.New("api: only an owner phone can add a watch") + // ErrWatchAppKey is a watch key the box cannot add: not 32 bytes, or a + // key it knows as something else. + ErrWatchAppKey = errors.New("api: that is not a watch's key") ) type appLinkStatus struct { @@ -126,8 +129,11 @@ type appLinkPairing struct { // gets read to the wrong person. Role string `json:"role"` // ExpiresAtMs is when the code stops working, so the UI can say so rather - // than leaving a stale square on screen. + // than leaving a stale square on screen. Zero for a watch added by its + // key, which needs no code. ExpiresAtMs int64 `json:"expires_at_ms"` + // DeviceID is the watch a key added, by the id the box lists it under. + DeviceID string `json:"device_id,omitempty"` } // appLinkPairingRequest is what a page or an app asks for. @@ -142,6 +148,9 @@ type appLinkPairingRequest struct { // Kind is "qr", "spoken", or "claim" for a code entered on the LAN. Kind string `json:"kind"` Code string `json:"code,omitempty"` + // Key is a watch's Noise static key, base64url without padding. With + // role watch it adds that watch at once, with no code. + Key string `json:"key,omitempty"` } // appLinkRoleRequest changes what one paired phone may do. @@ -249,20 +258,19 @@ func (s *Server) handleAppLinkPairing(w http.ResponseWriter, r *http.Request) { // appPairingKindSpoken asks for a code somebody can read aloud. const appPairingKindSpoken = "spoken" -// handleWatchPairing mints a watch code for the owner phone asking (ADR 0009, -// decision 4). +// handleWatchPairing adds a watch by its key for the owner phone asking (ADR +// 0009, decision 4, "Through FTW Cloud"). // // Three rules, each refused rather than bent. The request comes over a -// session, because the code is bound to the phone that asks and the LAN has +// session, because the watch is bound to the phone that asks and the LAN has // no phone to bind it to. The phone is an owner now, because a watch never // holds more than its phone and only an owner adds a device. And this very // request carried a passkey step-up: the grace window the passthrough keeps // for settings screens does not stand in for one, since a watch is a new key // that can charge the car from anywhere. // -// The answer is a QR payload like any other code's, with the box's static key, -// the rendezvous secret and the code. The phone hands it to its watch; nothing -// in it is new to the phone, and none of it passes a Sourceful service. +// A request without a key is refused: a watch code had no way in once the +// relay stopped, because FTW Cloud admits only what the roster lists. func (s *Server) handleWatchPairing(w http.ResponseWriter, r *http.Request, req appLinkPairingRequest) { caller, named := apiauth.FromRequest(r) if !named || caller.Kind != apiauth.KindApp { @@ -280,35 +288,41 @@ func (s *Server) handleWatchPairing(w http.ResponseWriter, r *http.Request, req "confirm with your passkey to add a watch.") return } - if req.Kind == appPairingKindSpoken { + if req.Key == "" { writeAppLinkError(w, http.StatusBadRequest, - "a watch code goes from the phone to the watch, never read aloud.") + "update the FTW app to add this watch: it is added by its key.") return } + s.addWatchByKey(w, r, req.Key) +} - code, expiresAt, err := s.deps.AppEnroll.MintWatchCode(callerDeviceID(r)) +// addWatchByKey adds the watch whose key the phone names, under the same +// three rules as a watch code (handleWatchPairing has checked them). The +// watch reaches the box through FTW Cloud, which admits only what the roster +// lists, so a code it would have to carry through the service has no place: +// the phone's own session vouches for the key, and the watch proves it holds +// the key in every handshake after. +func (s *Server) addWatchByKey(w http.ResponseWriter, r *http.Request, encoded string) { + key, err := base64.RawURLEncoding.DecodeString(encoded) + if err != nil { + writeAppLinkError(w, http.StatusBadRequest, "that is not a watch's key.") + return + } + id, err := s.deps.AppEnroll.AddWatch(callerDeviceID(r), key) switch { case err == nil: case errors.Is(err, ErrWatchNeedsOwner), errors.Is(err, ErrUnknownAppDevice): writeAppLinkError(w, http.StatusForbidden, "only an owner's phone can add a watch.") return - default: - writeAppLinkError(w, http.StatusInternalServerError, "could not mint a watch code") + case errors.Is(err, ErrWatchAppKey): + writeAppLinkError(w, http.StatusBadRequest, "that is not a watch's key.") return - } - // No LAN hint: the watch picks its own network and reaches the box - // through the relay's exchange. - url, err := s.deps.AppEnroll.EnrollmentURL(code, "") - if err != nil { - writeAppLinkError(w, http.StatusInternalServerError, "could not build the watch code") + default: + writeAppLinkError(w, http.StatusInternalServerError, "could not add the watch") return } - writeJSON(w, http.StatusOK, appLinkPairing{ - URL: url, - Role: apiauth.RoleWatch, - ExpiresAtMs: expiresAt.UnixMilli(), - }) + writeJSON(w, http.StatusOK, appLinkPairing{Role: apiauth.RoleWatch, DeviceID: id}) } func (s *Server) writeAppLinkMintError(w http.ResponseWriter, err error) { diff --git a/go/internal/api/api_app_link_session_test.go b/go/internal/api/api_app_link_session_test.go index 719b6a2b1..37adb61a5 100644 --- a/go/internal/api/api_app_link_session_test.go +++ b/go/internal/api/api_app_link_session_test.go @@ -56,9 +56,9 @@ func (e *realEnroller) MintSpokenCode(role string) (string, time.Time, error) { return code, expires, enrollError(err) } -func (e *realEnroller) MintWatchCode(parentID string) ([]byte, time.Time, error) { - code, expires, err := e.id.MintWatchCode(parentID, appenroll.WatchTTL) - return code, expires, enrollError(err) +func (e *realEnroller) AddWatch(parentID string, key []byte) (string, error) { + grant, err := e.id.AddWatch(parentID, key) + return grant.DeviceID, enrollError(err) } func (e *realEnroller) EnrollmentURL(code []byte, lanHint string) (string, error) { @@ -101,6 +101,8 @@ func enrollError(err error) error { return ErrWatchAppRole case errors.Is(err, appenroll.ErrWatchNeedsOwner): return ErrWatchNeedsOwner + case errors.Is(err, appenroll.ErrWatchKey): + return ErrWatchAppKey default: return err } @@ -495,10 +497,10 @@ func TestNoDoorButItsPhoneHandsOutAWatch(t *testing.T) { name string serve func(http.ResponseWriter) }{ - {"a scanned watch code from the LAN", func(w http.ResponseWriter) { + {"a watch from the LAN", func(w http.ResponseWriter) { s.handleAppLinkPairing(w, lan(http.MethodPost, "/api/app-link/pairing", `{"role":"watch"}`)) }}, - {"a spoken watch code from the LAN", func(w http.ResponseWriter) { + {"a spoken watch from the LAN", func(w http.ResponseWriter) { s.handleAppLinkPairing(w, lan(http.MethodPost, "/api/app-link/pairing", `{"role":"watch","kind":"spoken"}`)) }}, {"the guest changed into a watch from the LAN", func(w http.ResponseWriter) { @@ -514,12 +516,11 @@ func TestNoDoorButItsPhoneHandsOutAWatch(t *testing.T) { } } + // Over a session, from a phone this box does not know as an owner. rig := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) - status, refusal, body := rig.call(t, 1, appproto.APIReq{ - Method: appproto.APIPost, Path: "/api/app-link/pairing", - Body: []byte(`{"role":"watch"}`), StepUp: true}) + status, refusal, body := rig.call(t, 1, watchByKey(appKey(9), true)) if status != http.StatusForbidden { - t.Fatalf("a watch code over the session answered %d %q %s, want 403", status, refusal, body) + t.Fatalf("a watch over the session answered %d %q %s, want 403", status, refusal, body) } for _, d := range enrol.Devices() { diff --git a/go/internal/api/api_app_link_test.go b/go/internal/api/api_app_link_test.go index 7e82aba61..663f7e85f 100644 --- a/go/internal/api/api_app_link_test.go +++ b/go/internal/api/api_app_link_test.go @@ -28,7 +28,7 @@ type stubEnroller struct { spoken int watch int revoked int - // watchParent is the phone the last watch code was bound to. + // watchParent is the phone the last watch was added under. watchParent string mintedRole string roleSet string @@ -69,13 +69,13 @@ func (s *stubEnroller) MintSpokenCode(role string) (string, time.Time, error) { return "ABCD-EFGH", time.Now().Add(5 * time.Minute), nil } -func (s *stubEnroller) MintWatchCode(parentID string) ([]byte, time.Time, error) { +func (s *stubEnroller) AddWatch(parentID string, key []byte) (string, error) { if s.err != nil { - return nil, time.Time{}, s.err + return "", s.err } s.watch++ s.watchParent = parentID - return make([]byte, 16), time.Now().Add(10 * time.Minute), nil + return "wAtCh123", nil } func (s *stubEnroller) EnrollmentURL(code []byte, lanHint string) (string, error) { diff --git a/go/internal/api/api_app_link_watch_test.go b/go/internal/api/api_app_link_watch_test.go index 1cbdb8275..bfe3373e0 100644 --- a/go/internal/api/api_app_link_watch_test.go +++ b/go/internal/api/api_app_link_watch_test.go @@ -60,37 +60,27 @@ func watchPairing(stepUp bool) appproto.APIReq { } } -func TestAnOwnerPhoneAddsAWatchThroughTheApp(t *testing.T) { +// A watch is added by its key. A request without one comes from an app that +// would carry a code, which no watch can redeem once the relay stopped, so it +// is refused and adds nothing. +func TestAWatchWithoutAKeyIsRefused(t *testing.T) { enrol := newWatchEnrolment(t) rig := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) - status, refusal, body := rig.call(t, 1, watchPairing(true)) - if status != http.StatusOK { - t.Fatalf("adding a watch answered %d %q %s", status, refusal, body) - } - var answer appLinkPairing - if err := json.Unmarshal(body, &answer); err != nil { - t.Fatalf("decoding the watch code: %v (%q)", err, body) + status, _, body := rig.call(t, 1, watchPairing(true)) + if status != http.StatusBadRequest { + t.Fatalf("a watch without a key answered %d %s", status, body) } - if answer.Role != apiauth.RoleWatch || answer.URL == "" || answer.Code != "" { - t.Fatalf("the answer is %+v", answer) - } - if hint := lanHintIn(t, answer.URL); hint != "" { - t.Fatalf("the watch code carries a LAN hint %q", hint) - } - - grant, err := enrol.id.AuthoriseWatch(appKey(9), pairingCodeIn(t, answer.URL)) - if err != nil { - t.Fatalf("the watch could not join with the code its phone got: %v", err) - } - if grant.Role != apiauth.RoleWatch || grant.Parent != "aBcD1234" { - t.Fatalf("the box wrote down %+v", grant) + for _, d := range enrol.Devices() { + if d.Role == apiauth.RoleWatch { + t.Fatal("a watch joined") + } } } // The grace window a settings screen opens does not stand in for a step-up: // a watch is a new key that can charge the car from anywhere. -func TestAWatchCodeNeedsAStepUpOnTheRequestItself(t *testing.T) { +func TestAWatchNeedsAStepUpOnTheRequestItself(t *testing.T) { enrol := newWatchEnrolment(t) rig := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) @@ -105,9 +95,9 @@ func TestAWatchCodeNeedsAStepUpOnTheRequestItself(t *testing.T) { t.Fatalf("opening the window answered %d %q %s", status, refusal, body) } - status, refusal, body := rig.call(t, 2, watchPairing(false)) + status, refusal, body := rig.call(t, 2, watchByKey(appKey(9), false)) if status != http.StatusForbidden { - t.Fatalf("a watch code without a step-up answered %d %q %s", status, refusal, body) + t.Fatalf("a watch without a step-up answered %d %q %s", status, refusal, body) } var answer map[string]string if err := json.Unmarshal(body, &answer); err != nil { @@ -129,7 +119,7 @@ func TestOnlyAnOwnersPhoneAddsAWatch(t *testing.T) { enrol := newWatchEnrolment(t) viewer := newAppSession(t, apiauth.RoleViewer, withEnrolment(enrol)) - if status, refusal, _ := viewer.call(t, 1, watchPairing(true)); status == http.StatusOK { + if status, refusal, _ := viewer.call(t, 1, watchByKey(appKey(9), true)); status == http.StatusOK { t.Fatalf("a viewer added a watch (%q)", refusal) } @@ -154,7 +144,7 @@ func TestAddingAWatchLeavesAnInviteAlive(t *testing.T) { if err := json.Unmarshal(inviteBody, &invite); err != nil { t.Fatal(err) } - if status, refusal, body := rig.call(t, 2, watchPairing(true)); status != http.StatusOK { + if status, refusal, body := rig.call(t, 2, watchByKey(appKey(9), true)); status != http.StatusOK { t.Fatalf("adding a watch answered %d %q %s", status, refusal, body) } if _, err := enrol.id.Authorise(appKey(3), pairingCodeIn(t, invite.URL)); err != nil { @@ -165,11 +155,7 @@ func TestAddingAWatchLeavesAnInviteAlive(t *testing.T) { // The list names each watch's phone, and a watch keeps its role. func TestTheListShowsAWatchUnderItsPhone(t *testing.T) { enrol := newWatchEnrolment(t) - code, _, err := enrol.id.MintWatchCode("aBcD1234", appenroll.WatchTTL) - if err != nil { - t.Fatal(err) - } - watch, err := enrol.id.AuthoriseWatch(appKey(9), code) + watch, err := enrol.id.AddWatch("aBcD1234", appKey(9)) if err != nil { t.Fatal(err) } @@ -197,3 +183,108 @@ func TestTheListShowsAWatchUnderItsPhone(t *testing.T) { t.Fatalf("changing a watch's role answered %d %s", status, body) } } + +func watchByKey(key []byte, stepUp bool) appproto.APIReq { + return appproto.APIReq{ + Method: appproto.APIPost, + Path: "/api/app-link/pairing", + Body: []byte(`{"role":"watch","key":"` + base64.RawURLEncoding.EncodeToString(key) + `"}`), + StepUp: stepUp, + } +} + +// Through FTW Cloud a watch has no code to carry: the phone names its key +// over its own session, and the box lists the watch at once under that phone. +func TestAnOwnerPhoneAddsAWatchByItsKey(t *testing.T) { + enrol := newWatchEnrolment(t) + rig := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) + + status, refusal, body := rig.call(t, 1, watchByKey(appKey(9), true)) + if status != http.StatusOK { + t.Fatalf("adding a watch by its key answered %d %q %s", status, refusal, body) + } + var answer appLinkPairing + if err := json.Unmarshal(body, &answer); err != nil { + t.Fatal(err) + } + if answer.Role != apiauth.RoleWatch || answer.URL != "" || answer.Code != "" || answer.DeviceID == "" { + t.Fatalf("the answer is %+v", answer) + } + // The watch's first handshake needs no code. + grant, _, err := enrol.id.CheckWatch(appKey(9)) + if err != nil { + t.Fatalf("the watch could not open with its key alone: %v", err) + } + if grant.DeviceID != answer.DeviceID || grant.Role != apiauth.RoleWatch || grant.Parent != "aBcD1234" { + t.Fatalf("the box wrote down %+v for %q", grant, answer.DeviceID) + } + + // Asking again is the same watch, not a second one. + if status, refusal, body := rig.call(t, 2, watchByKey(appKey(9), true)); status != http.StatusOK { + t.Fatalf("adding the same watch again answered %d %q %s", status, refusal, body) + } + watches := 0 + for _, d := range enrol.Devices() { + if d.Role == apiauth.RoleWatch { + watches++ + } + } + if watches != 1 { + t.Fatalf("the box lists %d watches", watches) + } +} + +// A key goes through the same three rules as a code: a session, an owner, a +// step-up on the request. +func TestAWatchKeyMeetsTheSameRules(t *testing.T) { + enrol := newWatchEnrolment(t) + + // Without a step-up the session refuses before the route, or the route + // does when a settings screen opened the grace window; either way the + // app hears that it must ask for the passkey. + owner := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) + status, code, body := owner.call(t, 1, watchByKey(appKey(9), false)) + if status == http.StatusForbidden { + var answer map[string]string + _ = json.Unmarshal(body, &answer) + code = answer["code"] + } + if code != appproto.ErrNeedsStepUp { + t.Fatalf("a watch key without a step-up answered %d %q %s", status, code, body) + } + + viewer := newAppSession(t, apiauth.RoleViewer, withEnrolment(enrol)) + if status, refusal, _ := viewer.call(t, 1, watchByKey(appKey(9), true)); status == http.StatusOK { + t.Fatalf("a viewer added a watch by its key (%q)", refusal) + } + + s := New(&Deps{AppEnroll: enrol}) + w := httptest.NewRecorder() + req := pairingRequestRole("192.168.1.1", "127.0.0.1:1234", nil, apiauth.RoleWatch) + s.handleAppLinkPairing(w, req) + if w.Code != http.StatusForbidden { + t.Fatalf("the box's page added a watch: %d %s", w.Code, w.Body.String()) + } + + for _, d := range enrol.Devices() { + if d.Role == apiauth.RoleWatch { + t.Fatal("a watch joined") + } + } +} + +// A key that cannot be a watch's is refused: the wrong length, or the +// phone's own key. +func TestAKeyThatIsNoWatchIsRefused(t *testing.T) { + enrol := newWatchEnrolment(t) + rig := newAppSession(t, apiauth.RoleOwner, withEnrolment(enrol)) + // Each request on the session has its own id. + for i, c := range []struct { + name string + key []byte + }{{"short", appKey(9)[:16]}, {"the phone's own", sessionPhoneKey(t)}} { + if status, _, body := rig.call(t, uint32(i+1), watchByKey(c.key, true)); status != http.StatusBadRequest { + t.Fatalf("%s key answered %d %s", c.name, status, body) + } + } +} diff --git a/go/internal/api/api_backups.go b/go/internal/api/api_backups.go index c79fe6031..48a90ec81 100644 --- a/go/internal/api/api_backups.go +++ b/go/internal/api/api_backups.go @@ -5,6 +5,8 @@ import ( "encoding/json" "errors" "fmt" + "io" + "log/slog" "net/http" "os" "path/filepath" @@ -79,8 +81,27 @@ func withoutWriteDeadline(w http.ResponseWriter) { _ = http.NewResponseController(w).SetWriteDeadline(time.Time{}) } +func readBackupRequestBody(w http.ResponseWriter, r *http.Request) bool { + // HTTP/1 detects a disconnected client only after the request body has + // been read. Backup creation and verification take no body options. + // Consume it before starting work so a disconnect cancels the context. + if _, err := io.Copy(io.Discard, http.MaxBytesReader(w, r.Body, maxJSONBody)); err != nil { + status := http.StatusBadRequest + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + status = http.StatusRequestEntityTooLarge + } + writeJSON(w, status, map[string]string{"error": "could not read backup request body"}) + return false + } + return true +} + func (s *Server) handleBackupCreate(w http.ResponseWriter, r *http.Request) { withoutWriteDeadline(w) + if !readBackupRequestBody(w, r) { + return + } dir, ok := s.backupConfig(w) if !ok { return @@ -125,17 +146,25 @@ func (s *Server) handleBackupCreate(w http.ResponseWriter, r *http.Request) { writeJSON(w, 409, map[string]string{"error": "component update or rollback already in progress"}) return } + // The caller may be gone when this ends; the journal keeps the outcome. + started := time.Now() + slog.Info("full backup started", "dir", dir) + s.deps.State.ReportBackupProgress(state.BackupProgress{}) info, err := backup.Create(r.Context(), backup.CreateOptions{ ConfigPath: s.deps.ConfigPath, State: s.deps.State, StatePath: s.deps.StatePath, DataDir: s.deps.DataDir, OutputDir: dir, Components: s.backupComponentInventory(r.Context()), Maintenance: s.deps.DataMaintenanceMu, Progress: s.deps.State.ReportBackupProgress, }) + elapsed := time.Since(started).Round(time.Second) if err != nil { - s.deps.State.ReportBackupProgress(state.BackupProgress{Phase: "failed", Error: err.Error()}) + last := s.deps.State.BackupProgress() + slog.Error("full backup failed", "err", err, "phase", last.Phase, "table", last.Table, "rows_done", last.RowsDone, "elapsed", elapsed) + s.deps.State.ReportBackupProgress(state.BackupProgress{Phase: "failed", Table: last.Table, RowsDone: last.RowsDone, Error: err.Error()}) writeJSON(w, 500, map[string]string{"error": err.Error()}) return } + slog.Info("full backup created", "id", info.ID, "size_bytes", info.SizeBytes, "elapsed", elapsed) s.deps.State.ReportBackupProgress(state.BackupProgress{Phase: "complete", CompletedBytes: info.SizeBytes, TotalBytes: info.SizeBytes}) warning := "" if err := writeVerification(info.Path, info); err != nil { @@ -151,6 +180,9 @@ func (s *Server) handleBackupCreate(w http.ResponseWriter, r *http.Request) { func (s *Server) handleBackupVerify(w http.ResponseWriter, r *http.Request) { withoutWriteDeadline(w) + if !readBackupRequestBody(w, r) { + return + } dir, ok := s.backupConfig(w) if !ok { return diff --git a/go/internal/api/api_backups_cancel_test.go b/go/internal/api/api_backups_cancel_test.go new file mode 100644 index 000000000..bde9922ef --- /dev/null +++ b/go/internal/api/api_backups_cancel_test.go @@ -0,0 +1,122 @@ +package api + +import ( + "fmt" + "net" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/state" +) + +func TestBackupClientDisconnectCancelsBeforePublishing(t *testing.T) { + for _, body := range []string{"", "{}"} { + t.Run(fmt.Sprintf("body_bytes_%d", len(body)), func(t *testing.T) { + root := t.TempDir() + st, err := state.Open(filepath.Join(root, "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + maintenance := &sync.Mutex{} + maintenance.Lock() + server := New(&Deps{ + State: st, StatePath: filepath.Join(root, "state.db"), DataDir: root, + BackupDir: filepath.Join(root, "backups"), DataMaintenanceMu: maintenance, + }) + httpServer := httptest.NewServer(server.Handler()) + t.Cleanup(httpServer.Close) + // Unlock before closing the server even when a regression leaves the + // handler waiting. No test cleanup needs a Core restart. + t.Cleanup(maintenance.Unlock) + conn, err := net.DialTimeout("tcp", strings.TrimPrefix(httpServer.URL, "http://"), time.Second) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = conn.Close() }) + _, err = fmt.Fprintf(conn, "POST /api/backups HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: %d\r\n\r\n%s", len(body), body) + if err != nil { + t.Fatal(err) + } + waitBackupCondition(t, func() bool { return st.BackupProgress().Phase == "waiting_for_maintenance" }) + if err := conn.Close(); err != nil { + t.Fatal(err) + } + waitBackupCondition(t, func() bool { return st.BackupProgress().Phase == "failed" }) + if got := st.BackupProgress().Error; !strings.Contains(got, "context canceled") { + t.Fatalf("backup failure after disconnect = %q", got) + } + waitBackupCondition(t, func() bool { + if !server.backupMu.TryLock() { + return false + } + server.backupMu.Unlock() + if !server.versionUpdateMu.TryLock() { + return false + } + server.versionUpdateMu.Unlock() + return true + }) + archives, err := filepath.Glob(filepath.Join(root, "backups", "*.ftwbak")) + if err != nil || len(archives) != 0 { + t.Fatalf("archive published after disconnect: %v (error %v)", archives, err) + } + }) + } +} + +func TestBackupCreateRefusesOversizedIgnoredBody(t *testing.T) { + root := t.TempDir() + st, err := state.Open(filepath.Join(root, "state.db")) + if err != nil { + t.Fatal(err) + } + defer st.Close() + server := New(&Deps{State: st, StatePath: filepath.Join(root, "state.db"), DataDir: root, BackupDir: filepath.Join(root, "backups")}) + r := httptest.NewRequest(http.MethodPost, "/api/backups", strings.NewReader(strings.Repeat(" ", maxJSONBody+1))) + r.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + server.Handler().ServeHTTP(w, r) + if w.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("oversized backup body: status %d", w.Code) + } + if st.BackupProgress().Phase != "" { + t.Fatal("oversized request started a backup") + } + if _, err := os.Stat(filepath.Join(root, "backups")); !os.IsNotExist(err) { + t.Fatalf("oversized request created backup directory: %v", err) + } +} + +func TestBackupVerifyRefusesOversizedIgnoredBody(t *testing.T) { + root := t.TempDir() + server := New(&Deps{BackupDir: filepath.Join(root, "backups")}) + r := httptest.NewRequest(http.MethodPost, "/api/backups/ftw-full-backup-fixture.ftwbak/verify", strings.NewReader(strings.Repeat(" ", maxJSONBody+1))) + r.Header.Set("Content-Type", "application/json") + w := httptest.NewRecorder() + server.Handler().ServeHTTP(w, r) + if w.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("oversized verification body: status %d", w.Code) + } + if _, err := os.Stat(filepath.Join(root, "backups")); !os.IsNotExist(err) { + t.Fatalf("oversized verification created backup directory: %v", err) + } +} + +func waitBackupCondition(t *testing.T, condition func() bool) { + t.Helper() + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if condition() { + return + } + time.Sleep(5 * time.Millisecond) + } + t.Fatal("backup condition did not hold before the deadline") +} diff --git a/go/internal/api/api_backups_test.go b/go/internal/api/api_backups_test.go index 118d6dab6..011188ceb 100644 --- a/go/internal/api/api_backups_test.go +++ b/go/internal/api/api_backups_test.go @@ -1,7 +1,10 @@ package api import ( + "bytes" + "context" "encoding/json" + "log/slog" "net/http" "net/http/httptest" "os" @@ -127,3 +130,56 @@ func TestBackupIDRejectsTraversal(t *testing.T) { } } } + +// The home box's backup on 16 October 2026 failed after its caller had left +// and logged nothing. The journal and GET /api/backups must keep the reason. +func TestFailedBackupLogsAndReportsItsReason(t *testing.T) { + dataDir := t.TempDir() + statePath := filepath.Join(dataDir, "state.db") + st, err := state.Open(statePath) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + backupDir := filepath.Join(dataDir, "backups") + srv := New(&Deps{State: st, StatePath: statePath, DataDir: dataDir, BackupDir: backupDir, Version: "v1.3.1"}) + var logs bytes.Buffer + previous := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil))) + defer slog.SetDefault(previous) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + create := httptest.NewRecorder() + srv.Handler().ServeHTTP(create, httptest.NewRequest(http.MethodPost, "/api/backups", nil).WithContext(ctx)) + if create.Code != http.StatusInternalServerError || !strings.Contains(create.Body.String(), "context canceled") { + t.Fatalf("create = %d %s", create.Code, create.Body.String()) + } + if !strings.Contains(logs.String(), `msg="full backup failed" err="context canceled"`) { + t.Fatalf("journal lacks the reason: %s", logs.String()) + } + list := httptest.NewRecorder() + srv.Handler().ServeHTTP(list, httptest.NewRequest(http.MethodGet, "/api/backups", nil)) + var listed struct { + Progress state.BackupProgress `json:"progress"` + } + if err := json.Unmarshal(list.Body.Bytes(), &listed); err != nil { + t.Fatal(err) + } + if listed.Progress.Phase != "failed" || listed.Progress.Error != "context canceled" { + t.Fatalf("progress = %+v", listed.Progress) + } + + // A run that fails before its first phase must not inherit the last + // run's table. + st.ReportBackupProgress(state.BackupProgress{Phase: state.BackupPhaseCopying, Table: "ts_buckets", RowsDone: 4_700_000}) + notDir := filepath.Join(dataDir, "backups-file") + if err := os.WriteFile(notDir, nil, 0o600); err != nil { + t.Fatal(err) + } + srv = New(&Deps{State: st, StatePath: statePath, DataDir: dataDir, BackupDir: notDir, Version: "v1.3.1"}) + srv.Handler().ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodPost, "/api/backups", nil)) + if p := st.BackupProgress(); p.Phase != "failed" || p.Table != "" || p.RowsDone != 0 || !strings.Contains(p.Error, "create output dir") { + t.Fatalf("early failure = %+v", p) + } +} diff --git a/go/internal/api/api_cloud.go b/go/internal/api/api_cloud.go index 331d9a3bc..2d7b86126 100644 --- a/go/internal/api/api_cloud.go +++ b/go/internal/api/api_cloud.go @@ -189,7 +189,7 @@ func (s *Server) handleCloudStats(w http.ResponseWriter, r *http.Request) { writeJSON(w, http.StatusOK, link.Status()) case errors.Is(err, cloudlink.ErrGrantRequest): writeAppLinkError(w, http.StatusBadRequest, err.Error()) - case errors.Is(err, cloudlink.ErrNotGranted), errors.Is(err, cloudlink.ErrPurgePending): + case errors.Is(err, cloudlink.ErrNotGranted), errors.Is(err, cloudlink.ErrPurgePending), errors.Is(err, cloudlink.ErrCloudTermsFirst): writeAppLinkError(w, http.StatusConflict, err.Error()) case errors.Is(err, cloudlink.ErrNoService): writeAppLinkError(w, http.StatusServiceUnavailable, err.Error()) diff --git a/go/internal/api/api_cloud_test.go b/go/internal/api/api_cloud_test.go index e4c064bc1..1f37153de 100644 --- a/go/internal/api/api_cloud_test.go +++ b/go/internal/api/api_cloud_test.go @@ -430,20 +430,28 @@ func TestTheConsentRoundTripShowsThePurposes(t *testing.T) { return st } + current := `{"policy":"` + cloudwire.StatsPolicy + `"}` call(s.handleCloudStats, http.MethodPost, "/api/cloud/stats", statsBody, http.StatusConflict) call(s.handleCloudGrant, http.MethodPost, "/api/cloud/grant", grantBody, http.StatusOK) - call(s.handleCloudStats, http.MethodPost, "/api/cloud/stats", statsBody, http.StatusOK) - if st := call(s.handleCloudStatus, http.MethodGet, "/api/cloud", "", http.StatusOK); st.State != "on" || - strings.Join(st.Purposes, ",") != "serve,stats" { - t.Fatalf("status with statistics on: %+v", st) + // Older terms did not include statistics: the current ones come first. + if st := call(s.handleCloudStatus, http.MethodGet, "/api/cloud", "", http.StatusOK); !st.StatisticsNeedTerms || st.Statistics { + t.Fatalf("status under older terms: %+v", st) } - if st := call(s.handleCloudStopStats, http.MethodDelete, "/api/cloud/stats", "", http.StatusOK); strings.Join(st.Purposes, ",") != "serve" { - t.Fatalf("status with statistics off: %+v", st) + call(s.handleCloudStats, http.MethodPost, "/api/cloud/stats", current, http.StatusConflict) + call(s.handleCloudGrant, http.MethodPost, "/api/cloud/grant", `{"scopes":["ftw.live.read"],"policy":"`+cloudwire.CloudTermsPolicy+`"}`, http.StatusOK) + if st := call(s.handleCloudStatus, http.MethodGet, "/api/cloud", "", http.StatusOK); st.State != "on" || + strings.Join(st.Purposes, ",") != "serve,stats" || !st.Statistics || !st.StatisticsRequired { + t.Fatalf("status under the current terms: %+v", st) } + call(s.handleCloudStopStats, http.MethodDelete, "/api/cloud/stats", "", http.StatusConflict) if st := call(s.handleCloudRevoke, http.MethodDelete, "/api/cloud/grant", "", http.StatusOK); st.State != "revoking" { t.Fatalf("status after FTW Cloud off: %+v", st) } + // While the copy is deleted, the box's own answer can still change. call(s.handleCloudStats, http.MethodPost, "/api/cloud/stats", statsBody, http.StatusConflict) + if st := call(s.handleCloudStats, http.MethodPost, "/api/cloud/stats", current, http.StatusOK); !st.Statistics { + t.Fatalf("status after the local yes: %+v", st) + } } type cloudSiteStub struct{} diff --git a/go/internal/api/api_control_feedback.go b/go/internal/api/api_control_feedback.go index 03265872c..3a4b739b9 100644 --- a/go/internal/api/api_control_feedback.go +++ b/go/internal/api/api_control_feedback.go @@ -2,6 +2,7 @@ package api import ( "encoding/json" + "fmt" "math" "sort" "time" @@ -32,11 +33,13 @@ type ControlFeedback struct { SiteConfirmation string `json:"site_confirmation"` SiteMeter string `json:"site_meter,omitempty"` SiteEvidence *ControlComparison `json:"site_evidence,omitempty"` + SiteHeadroom *EVSiteHeadroom `json:"site_headroom,omitempty"` ToleranceW *float64 `json:"tolerance_w"` // Internal steps of the verdict; the API exposes their result above. State string `json:"-"` VerificationTier *int `json:"-"` VerificationLost bool `json:"-"` + Escalated bool `json:"-"` // not following long enough to be an alarm Response string `json:"-"` LimitedBy string `json:"-"` SiteBeforeW *float64 `json:"-"` @@ -56,9 +59,13 @@ type ControlFeedback struct { OfferedA *float64 `json:"offered_a"` DeviceLimitA *float64 `json:"device_limit_a"` DeviceReason string `json:"device_reason,omitempty"` - SinceMs int64 `json:"since_ms,omitempty"` - CommandAtMs int64 `json:"command_at_ms,omitempty"` - ObservedAtMs int64 `json:"observed_at_ms,omitempty"` + // ChargingPhase names the one phase (L1-L3) a one-phase car charges on + // under a three-phase order, and ChargingPhaseMaxW the most it takes. + ChargingPhase string `json:"charging_phase,omitempty"` + ChargingPhaseMaxW *float64 `json:"charging_phase_max_w,omitempty"` + SinceMs int64 `json:"since_ms,omitempty"` + CommandAtMs int64 `json:"command_at_ms,omitempty"` + ObservedAtMs int64 `json:"observed_at_ms,omitempty"` } type feedbackReading struct { @@ -104,8 +111,10 @@ func (s *Server) controlFeedback(now time.Time) []ControlFeedback { } observe, disabled := map[string]bool{}, map[string]bool{} configuredBattery := map[string]bool{} + var site loadpoint.SiteFuse if s.deps.Cfg != nil && s.deps.CfgMu != nil { s.deps.CfgMu.RLock() + site = loadpoint.SiteFuse{MaxAmps: s.deps.Cfg.Fuse.MaxAmps, Voltage: s.deps.Cfg.Fuse.Voltage, PhaseCnt: s.deps.Cfg.Fuse.Phases} observe = config.ObserveOnlyDriverSet(s.deps.Cfg) for _, d := range s.deps.Cfg.Drivers { disabled[d.Name] = d.Disabled @@ -206,6 +215,18 @@ func (s *Server) controlFeedback(now time.Time) []ControlFeedback { if f.RequestedW != nil && lp.Phases > 0 && lp.VoltageV > 0 { f.RequestedA = watts(*f.RequestedW / float64(lp.Phases) / lp.VoltageV) } + // A one-phase car takes a third of a three-phase order. + if lp.CarPhases == 1 && lp.CarPhase > 0 { + f.ChargingPhase = fmt.Sprintf("L%d", lp.CarPhase) + maxW := loadpoint.OnePhaseCarMaxW(loadpoint.Config{MaxChargeW: lp.MaxChargeW}, site) + if lp.CarMaxW > 0 { + maxW = math.Min(maxW, lp.CarMaxW) + } + f.ChargingPhaseMaxW = watts(maxW) + if f.RequestedW != nil && lp.Phases == 3 { + f.RequestedW = watts(*f.RequestedW / 3) + } + } if lp.PowerUnavailable { fresh = false f.ActualW = nil @@ -280,6 +301,7 @@ func (s *Server) controlFeedback(now time.Time) []ControlFeedback { f.Reason = "readings_lost" } f.ReadingsFresh = fresh + f.SiteHeadroom = s.evSiteHeadroom(f, lps[rd.Driver], meter, now) out = append(out, f) } } @@ -339,7 +361,9 @@ func setControlStatus(f *ControlFeedback) { "vehicle_complete", "vehicle_limit_completion", "vehicle_not_requesting": status = "following" case "battery_full", "battery_nearly_full", "battery_nearly_empty", "core_limit", - "fuse_limit", "fuse_cooldown", "charger_limit", "load_balancer_limit": + "fuse_limit", "fuse_cooldown", "charger_limit", "load_balancer_limit", "one_phase": + status = "limited" + case "import_limit": status = "limited" case "device_limit", "offered_current_lower": status, severity = "limited", "warning" @@ -360,6 +384,9 @@ func setControlStatus(f *ControlFeedback) { if status == "following" && (f.VerificationTier == nil || *f.VerificationTier < 1) { status = "waiting" } + if status == "not_following" && f.Escalated { + severity = "alarm" + } f.Status, f.Severity = status, severity f.Evidence = "none" if f.VerificationTier != nil { @@ -390,6 +417,15 @@ func controlVerificationLost(f ControlFeedback, cmd telemetry.CommandEvidence, n return !cmd.Since.IsZero() && now.Sub(cmd.Since) >= grace && now.Sub(cmd.LastMeasuredAt) >= grace } +// evMismatchWait is how long a charger's fresh readings must disagree with +// the orders in force before it is not following. evOverdrawAlarm is how +// much longer one that draws more than ordered may go on before the warning +// becomes an alarm. +const ( + evMismatchWait = 2 * time.Minute + evOverdrawAlarm = 10 * time.Minute +) + func classifyControlFeedback(f *ControlFeedback, cmd telemetry.CommandEvidence, commanded, fresh bool, coreReason string, connected *bool, now time.Time) { set := func(state, reason, severity string) { f.State, f.Reason, f.Severity = state, reason, severity } if !fresh { @@ -420,8 +456,9 @@ func classifyControlFeedback(f *ControlFeedback, cmd telemetry.CommandEvidence, // A positive clamp explains the lower offer, not prolonged zero delivery. // An explicit load-balancer reason can explain zero delivery. Keep known // safety pauses and unconfirmed/stale commands authoritative. - evMismatchObserved := !cmd.PowerMismatchSince.IsZero() && cmd.LastObservation.Sub(cmd.PowerMismatchSince) >= 2*time.Minute - noPowerObserved := evMismatchObserved || !cmd.NoPowerSince.IsZero() && cmd.NoPowerAt.Sub(cmd.NoPowerSince) >= 2*time.Minute + evMismatchObserved := !cmd.PowerMismatchSince.IsZero() && cmd.LastObservation.Sub(cmd.PowerMismatchSince) >= evMismatchWait + noPowerObserved := evMismatchObserved || !cmd.NoPowerSince.IsZero() && cmd.NoPowerAt.Sub(cmd.NoPowerSince) >= evMismatchWait + if f.Kind == "ev" && commanded && cmd.Result == "accepted" && f.LimitedBy != "load_balancer" && coreReason != "site_meter_stale" && coreReason != "site_phase_currents_stale" && coreReason != "fuse_cooldown" && @@ -434,7 +471,11 @@ func classifyControlFeedback(f *ControlFeedback, cmd telemetry.CommandEvidence, set("limited", "device_limit", "warning") return } - if coreReason == "site_meter_stale" || coreReason == "site_phase_currents_stale" || coreReason == "fuse_cooldown" || coreReason == "fuse_limit" || coreReason == "charger_limit" { + // A limit explains a lower order, not a draw that differs from it. A + // charger whose fresh readings have disagreed with the order past its + // response time is judged below, as under any other order. + limit := coreReason == "site_meter_stale" || coreReason == "site_phase_currents_stale" || coreReason == "fuse_cooldown" || coreReason == "fuse_limit" || coreReason == "import_limit" || coreReason == "charger_limit" + if limit && !(f.Kind == "ev" && evMismatchObserved) { set("limited", coreReason, "warning") return } @@ -469,6 +510,14 @@ func classifyControlFeedback(f *ControlFeedback, cmd telemetry.CommandEvidence, set("waiting", "waiting_response", "info") return } + // A one-phase car drawing a third of a three-phase order does what it + // can: the order is its current per phase, on its one phase. A limit, a + // changed setpoint or a response still due says more, and comes first. + if f.ChargingPhase != "" && cmd.Result == "accepted" && f.SentW != nil && *f.SentW > 100 && + f.ActualW != nil && math.Abs(*f.ActualW-*f.SentW) <= telemetry.ControlToleranceW(*f.SentW) { + set("limited", "one_phase", "info") + return + } if f.SentW == nil || f.ActualW == nil { set("unknown", "response_unknown", "info") return @@ -509,6 +558,11 @@ func classifyControlFeedback(f *ControlFeedback, cmd telemetry.CommandEvidence, reason = "power_below_target" } set("warning", reason, "warning") + // FTW cannot stop a charger that draws more than it is told. A car + // that takes less only charges slower. + if f.Kind == "ev" && (reason == "power_while_idle" || reason == "power_above_target") { + f.Escalated = !cmd.PowerOverSince.IsZero() && cmd.LastObservation.Sub(cmd.PowerOverSince) >= evMismatchWait+evOverdrawAlarm + } return } if coreReason == "core_limit" { diff --git a/go/internal/api/api_ev_headroom.go b/go/internal/api/api_ev_headroom.go new file mode 100644 index 000000000..dc512d43e --- /dev/null +++ b/go/internal/api/api_ev_headroom.go @@ -0,0 +1,105 @@ +package api + +import ( + "encoding/json" + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" + "math" + "time" +) + +// EVSiteHeadroom is a conditional projection, never a charger diagnosis or +// dispatch input. Current site loads are held constant in the calculation. +type EVSiteHeadroom struct { + Evidence string `json:"evidence"` + FuseA float64 `json:"fuse_a"` + MarginA float64 `json:"margin_a"` + ProjectedMaxA float64 `json:"projected_max_a"` + BatteryChargeW float64 `json:"battery_charge_w"` +} + +func projectEVHeadroom(f ControlFeedback, phases int, voltage float64, fuse config.Fuse, currents, powers []float64, meterAt, now time.Time) *EVSiteHeadroom { + // Only balanced, explicitly three-phase charging is supported. Single-phase + // and automatic switching need a verified phase mapping, not an assumed L1. + if f.Kind != "ev" || f.Reason != "power_below_target" || !f.ReadingsFresh || + f.SentW == nil || f.ActualW == nil || *f.ActualW < 100 || phases != 3 || fuse.Phases != 3 || + len(currents) != 3 || len(powers) != 3 || voltage <= 0 || fuse.MaxAmps <= 0 || math.IsNaN(voltage) || math.IsInf(voltage, 0) { + return nil + } + age := now.Sub(meterAt) + skew := meterAt.Sub(time.UnixMilli(f.ObservedAtMs)) + if age < 0 || age > 15*time.Second || skew < -10*time.Second || skew > 10*time.Second { + return nil + } + gap := *f.SentW - *f.ActualW + if math.IsNaN(gap) || math.IsInf(gap, 0) || gap <= telemetry.ControlToleranceW(*f.SentW) { + return nil + } + margin := fuse.EffectiveSafetyMarginA() + if math.IsNaN(margin) || math.IsInf(margin, 0) || margin < 0 || margin >= fuse.MaxAmps { + return nil + } + worst := 0.0 + for i, a := range currents { + // Meters may report current magnitudes during export. Signed power must + // establish import on every phase before adding the missing EV load. + if math.IsNaN(a) || math.IsInf(a, 0) || a < 0 || + math.IsNaN(powers[i]) || math.IsInf(powers[i], 0) || powers[i] <= 0 { + return nil + } + worst = math.Max(worst, a+gap/(3*voltage)) + } + if worst <= fuse.MaxAmps-margin+0.5 { + return nil + } + return &EVSiteHeadroom{Evidence: "calculated", FuseA: fuse.MaxAmps, MarginA: margin, ProjectedMaxA: worst} +} + +func (s *Server) evSiteHeadroom(f ControlFeedback, lp loadpoint.State, meter string, now time.Time) *EVSiteHeadroom { + if s.deps.Cfg == nil || s.deps.CfgMu == nil || s.deps.Tel == nil || meter == "" { + return nil + } + s.deps.CfgMu.RLock() + fuse := s.deps.Cfg.Fuse + explicit3P := false + for _, c := range s.deps.Cfg.Loadpoints { + if c.DriverName == f.Driver && c.PhaseMode == "3p" { + explicit3P = true + } + } + s.deps.CfgMu.RUnlock() + if !explicit3P { + return nil + } + rd := s.deps.Tel.Get(meter, telemetry.DerMeter) + h := s.deps.Tel.DriverHealth(meter) + if rd == nil || h == nil || !h.TelemetryLive() || h.DeviceFault { + return nil + } + var phase struct { + L1 *float64 `json:"l1_a"` + L2 *float64 `json:"l2_a"` + L3 *float64 `json:"l3_a"` + W1 *float64 `json:"l1_w"` + W2 *float64 `json:"l2_w"` + W3 *float64 `json:"l3_w"` + } + if json.Unmarshal(rd.Data, &phase) != nil || phase.L1 == nil || phase.L2 == nil || phase.L3 == nil || + phase.W1 == nil || phase.W2 == nil || phase.W3 == nil { + return nil + } + result := projectEVHeadroom(f, lp.Phases, lp.VoltageV, fuse, + []float64{*phase.L1, *phase.L2, *phase.L3}, []float64{*phase.W1, *phase.W2, *phase.W3}, rd.UpdatedAt, now) + if result == nil { + return nil + } + for _, obs := range s.deps.Tel.OnlineBatteryControlPowers(now, 15*time.Second) { + skew := rd.UpdatedAt.Sub(obs.Seen()) + if skew >= -10*time.Second && skew <= 10*time.Second && obs.PowerW > 100 { + result.BatteryChargeW += obs.PowerW + } + } + + return result +} diff --git a/go/internal/api/api_ev_headroom_test.go b/go/internal/api/api_ev_headroom_test.go new file mode 100644 index 000000000..3c48b3f15 --- /dev/null +++ b/go/internal/api/api_ev_headroom_test.go @@ -0,0 +1,118 @@ +package api + +import ( + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" + "math" + "sync" + "testing" + "time" +) + +func TestEVSiteHeadroomProjection(t *testing.T) { + now := time.Now() + for _, tc := range []struct { + name string + modify func(*ControlFeedback, *config.Fuse, *[]float64, *int, *time.Time) + want bool + }{ + {"actual household case", nil, true}, + {"ample headroom", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { *a = []float64{8, 9, 8} }, false}, + {"asymmetric busiest phase", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + *a = []float64{10, 21, 10} + }, true}, + {"stale meter", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + *at = now.Add(-16 * time.Second) + }, false}, + {"future meter", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + *at = now.Add(time.Second) + }, false}, + {"misaligned EV", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + f.ObservedAtMs = now.Add(-11 * time.Second).UnixMilli() + }, false}, + {"stale EV", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { f.ReadingsFresh = false }, false}, + {"single phase mapping unknown", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { *p = 1 }, false}, + {"missing phase", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { *a = []float64{21, 20} }, false}, + {"nonfinite phase", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { (*a)[0] = math.NaN() }, false}, + {"mixed export", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { (*a)[0] = -3 }, false}, + {"failed command precedence", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + f.Reason = "command_failed" + }, false}, + {"device reason precedence", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + f.Reason = "load_balancer_limit" + }, false}, + {"one phase reason precedence", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { f.Reason = "one_phase" }, false}, + {"hourly import reason precedence", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + f.Reason = "import_limit" + }, false}, + {"zero delivery not explained", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { f.ActualW = watts(0) }, false}, + {"invalid safety margin", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + c.SafetyMarginA = watts(26) + }, false}, + {"safety margin binds", func(f *ControlFeedback, c *config.Fuse, a *[]float64, p *int, at *time.Time) { + c.SafetyMarginA = watts(2) + *a = []float64{15, 15, 15} + }, true}, + } { + t.Run(tc.name, func(t *testing.T) { + f := ControlFeedback{Kind: "ev", Reason: "power_below_target", ReadingsFresh: true, SentW: watts(10350), ActualW: watts(4155), ObservedAtMs: now.UnixMilli()} + c := config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230} + a := []float64{19.65, 20.52, 19.46} + p := 3 + at := now + if tc.modify != nil { + tc.modify(&f, &c, &a, &p, &at) + } + got := projectEVHeadroom(f, p, 230, c, a, []float64{4519.5, 4719.6, 4475.8}, at, now) + if (got != nil) != tc.want { + t.Fatalf("projection=%+v want present=%v", got, tc.want) + } + if got != nil && tc.name == "actual household case" && math.Abs(got.ProjectedMaxA-29.498) > 0.01 { + t.Fatalf("unexpected projection %+v", got) + } + }) + } +} + +func TestEVSiteHeadroomServerContext(t *testing.T) { + tel := telemetry.NewStore() + tel.Update("meter", telemetry.DerMeter, 13843, nil, []byte(`{"l1_a":19.65,"l2_a":20.52,"l3_a":19.46,"l1_w":4519.5,"l2_w":4719.6,"l3_w":4475.8}`)) + tel.Update("battery", telemetry.DerBattery, 9383, nil, nil) + tel.DriverHealthMut("meter").RecordSuccess() + tel.DriverHealthMut("battery").RecordSuccess() + cfg := &config.Config{Fuse: config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230}, Loadpoints: []config.Loadpoint{{DriverName: "ev", PhaseMode: "3p"}}} + srv := New(&Deps{Tel: tel, Cfg: cfg, CfgMu: &sync.RWMutex{}}) + now := time.Now() + f := ControlFeedback{Driver: "ev", Kind: "ev", Reason: "power_below_target", ReadingsFresh: true, SentW: watts(10350), ActualW: watts(4155), ObservedAtMs: now.UnixMilli()} + lp := loadpoint.State{Phases: 3, VoltageV: 230} + got := srv.evSiteHeadroom(f, lp, "meter", now) + if got == nil || got.BatteryChargeW != 9383 { + t.Fatalf("missing site/battery context %+v", got) + } + tel.DriverHealthMut("battery").SetDeviceFault(true, "test") + got = srv.evSiteHeadroom(f, lp, "meter", now) + if got == nil || got.BatteryChargeW != 0 { + t.Fatalf("faulted battery contributed %+v", got) + } + cfg.Loadpoints[0].PhaseMode = "auto" + if got = srv.evSiteHeadroom(f, lp, "meter", now); got != nil { + t.Fatalf("unknown phase mapping %+v", got) + } + cfg.Loadpoints[0].PhaseMode = "3p" + tel.DriverHealthMut("meter").SetDeviceFault(true, "test") + if got = srv.evSiteHeadroom(f, lp, "meter", now); got != nil { + t.Fatalf("faulted meter contributed %+v", got) + } +} + +func TestEVSiteHeadroomRejectsInvalidPhasePower(t *testing.T) { + now := time.Now() + f := ControlFeedback{Kind: "ev", Reason: "power_below_target", ReadingsFresh: true, SentW: watts(10350), ActualW: watts(4155), ObservedAtMs: now.UnixMilli()} + fuse := config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230} + for _, powers := range [][]float64{nil, {4600, 4600}, {math.NaN(), 4600, 4600}, {4600, math.Inf(1), 4600}, {4600, 4600, math.Inf(-1)}} { + if got := projectEVHeadroom(f, 3, 230, fuse, []float64{20, 20, 20}, powers, now, now); got != nil { + t.Fatalf("invalid phase power produced explanation: %+v", got) + } + } +} diff --git a/go/internal/api/api_ev_not_following_test.go b/go/internal/api/api_ev_not_following_test.go new file mode 100644 index 000000000..2592f6100 --- /dev/null +++ b/go/internal/api/api_ev_not_following_test.go @@ -0,0 +1,172 @@ +package api + +import ( + "fmt" + "testing" + "testing/synctest" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// evTrace is an Easee Cloud charger on a loadpoint: a command and a poll +// every five seconds, as Core and the driver send them. The cloud stamps a +// power value when it changes and each poll confirms it (easee_cloud.lua). +type evTrace struct { + tel *telemetry.Store + mgr *loadpoint.Manager + srv *Server + changedAt time.Time + drawW float64 +} + +func newEVTrace(drawW float64) *evTrace { + tel := telemetry.NewStore() + tel.DriverHealthMut("easee") + mgr := loadpoint.NewManager() + mgr.Load([]loadpoint.Config{{ID: "garage", DriverName: "easee", MinChargeW: 4140, MaxChargeW: 11040}}) + return &evTrace{tel: tel, mgr: mgr, srv: New(&Deps{Tel: tel, Loadpoints: mgr}), changedAt: time.Now(), drawW: drawW} +} + +// tick orders w for reason, polls the charger and waits five seconds. +func (e *evTrace) tick(w float64, reason string) { + c := e.tel.BeginCommand("easee", []byte(fmt.Sprintf(`{"action":"ev_set_current","power_w":%g}`, w)), time.Now()) + e.tel.CompleteCommand(c, "accepted") + e.mgr.SetCommanded("garage", w, reason) + e.tel.Update("easee", telemetry.DerEV, e.drawW, nil, []byte(fmt.Sprintf( + `{"connected":true,"control_power_observed_at":%q,"control_power_confirmed":true,"power_max_age_s":180}`, + e.changedAt.Format(time.RFC3339Nano)))) + e.mgr.Observe("garage", true, e.drawW, 0, true) + time.Sleep(5 * time.Second) +} + +func (e *evTrace) verdict(t *testing.T) ControlFeedback { + t.Helper() + for _, f := range e.srv.controlFeedback(time.Now()) { + if f.Kind == "ev" { + return f + } + } + t.Fatal("no EV verdict") + return ControlFeedback{} +} + +// Field report, 22 Oct 2026, v0.155.0-beta.1, Easee Cloud: the charger drew +// 3.13 kW, Core cut its order to 0 W for the main fuse at 06:32 and the +// charger kept drawing. The verdict said "Limited by the main fuse" for as +// long as it drew. A limit explains a lower order, never a draw above it. +func TestChargerDrawingAgainstAFuseStopIsNotFollowing(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + e := newEVTrace(3130) + for range 60 * 60 / 5 { + e.tick(7590, "plan") + } + cut := time.Now() + var first, alarm time.Duration + for range 20 * 60 / 5 { + e.tick(0, "fuse_limit") + f, since := e.verdict(t), time.Since(cut) + if f.Status == "not_following" && first == 0 { + first = since + if f.Reason != "power_while_idle" || f.Severity != "warning" || f.Evidence != "measured" { + t.Fatalf("first not following at %v: %s/%s/%s", since, f.Reason, f.Severity, f.Evidence) + } + } + if f.Severity == "alarm" && alarm == 0 { + alarm = since + if f.Status != "not_following" { + t.Fatalf("alarm at %v is %s/%s", since, f.Status, f.Reason) + } + } + if first != 0 && f.Status != "not_following" { + t.Fatalf("at %v the verdict went back to %s/%s while the charger still draws", since, f.Status, f.Reason) + } + } + t.Logf("not following from %v after the cut, alarm from %v", first, alarm) + // Easee declares 180 s for its power to show; Core waits two minutes + // for an order to take hold and two more of measured disagreement. + if first < 3*time.Minute || first > 5*time.Minute { + t.Fatalf("not following after %v of drawing against a stop, want within 3-5 min", first) + } + if alarm < first+10*time.Minute || alarm > first+11*time.Minute { + t.Fatalf("alarm after %v (not following from %v), want ten minutes later", alarm, first) + } + }) +} + +// A charger that obeys the cut within its response time stays Limited, +// including Easee Cloud at the end of its declared 180 s. +func TestChargerFollowingAFuseStopStaysLimited(t *testing.T) { + for _, after := range []time.Duration{100 * time.Second, 175 * time.Second} { + synctest.Test(t, func(t *testing.T) { + e := newEVTrace(7500) + for range 10 * 60 / 5 { + e.tick(7590, "plan") + } + cut := time.Now() + for range 15 * 60 / 5 { + if time.Since(cut) >= after && e.drawW > 0 { + e.drawW, e.changedAt = 0, time.Now() + } + e.tick(0, "fuse_limit") + if f := e.verdict(t); f.Status != "limited" || f.Reason != "fuse_limit" || f.Severity == "alarm" { + t.Fatalf("stopped after %v: at %v the verdict is %s/%s/%s", after, time.Since(cut), f.Status, f.Reason, f.Severity) + } + } + }) + } +} + +// Time spent drawing below the order is not time drawing above it. After +// 13 minutes at 3 kW against a 7 kW order, a car that rises to 9 kW is +// drawing more than it is told: a warning now, an alarm only ten minutes +// after that wait has passed for the overdraw itself. +func TestOverdrawAfterAShortfallAlarmsOnItsOwnTime(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + e := newEVTrace(3000) + for range 13 * 60 / 5 { + e.tick(7000, "plan") + } + if f := e.verdict(t); f.Reason != "power_below_target" || f.Severity != "warning" { + t.Fatalf("3 kW against 7 kW reads %s/%s", f.Reason, f.Severity) + } + e.drawW, e.changedAt = 9000, time.Now() + over := time.Now() + var alarm time.Duration + for range 15 * 60 / 5 { + e.tick(7000, "plan") + f := e.verdict(t) + if f.Reason != "power_above_target" { + t.Fatalf("at %v 9 kW against 7 kW reads %s", time.Since(over), f.Reason) + } + if f.Severity == "alarm" && alarm == 0 { + alarm = time.Since(over) + } + } + if alarm < 12*time.Minute || alarm > 13*time.Minute { + t.Fatalf("overdraw alarmed after %v, want two minutes plus ten", alarm) + } + }) +} + +// A car that takes less than a fuse-limited order is not the fuse's doing. +// It reads as it would under a plan order: not following, a warning. +func TestCarBelowAFuseLimitedOrderIsNotFollowing(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + e := newEVTrace(3130) + for range 10 * 60 / 5 { + e.tick(8280, "fuse_limit") + } + f := e.verdict(t) + if f.Status != "not_following" || f.Reason != "power_below_target" || f.Severity != "warning" { + t.Fatalf("3.1 kW against an 8.3 kW fuse-limited order reads %s/%s/%s", f.Status, f.Reason, f.Severity) + } + for range 30 * 60 / 5 { + e.tick(8280, "fuse_limit") + } + if f := e.verdict(t); f.Severity != "warning" { + t.Fatalf("a car drawing less than offered escalated to %s", f.Severity) + } + }) +} diff --git a/go/internal/api/api_ev_one_phase_car_test.go b/go/internal/api/api_ev_one_phase_car_test.go new file mode 100644 index 000000000..856c059b5 --- /dev/null +++ b/go/internal/api/api_ev_one_phase_car_test.go @@ -0,0 +1,89 @@ +package api + +import ( + "context" + "encoding/json" + "fmt" + "math" + "sync" + "testing" + "testing/synctest" + "time" + + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// A one-phase car (a plug-in hybrid) on a three-phase charger, the field +// site's table and a plan for 4.8 kW, through a night. It takes the order's +// current per phase on L1. Once the meter shows that, the verdict says so +// and the car is never "not following" for doing what it can. +func TestOnePhaseCarThroughANight(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + tel := telemetry.NewStore() + tel.DriverHealthMut("easee") + mgr := loadpoint.NewManager() + steps := []float64{0, 1600, 1900, 2100, 2400, 2600, 2800, 3100, 3300, 3500, 3800, + 4800, 5500, 6200, 6900, 7600, 8300, 9000, 9700, 10400, 11100} + mgr.Load([]loadpoint.Config{{ID: "garage", DriverName: "easee", MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: "auto", AllowedStepsW: steps}}) + mgr.SetSchedule("garage", loadpoint.Schedule{SoC: .8}) + var orderW float64 + var orderAt time.Time + drawW := func() float64 { + if orderW <= 0 || time.Since(orderAt) < 40*time.Second { + return 0 + } + return math.Min(math.Floor(orderW/690+.5), 16) * 230 + } + plan := func(now time.Time) (loadpoint.Directive, bool) { + slot := now.Truncate(15 * time.Minute) + return loadpoint.Directive{SlotStart: slot, SlotEnd: slot.Add(15 * time.Minute), LoadpointEnergyWh: map[string]float64{"garage": 4800.0 / 4}}, true + } + sample := func(string) (loadpoint.EVSample, bool) { + return loadpoint.EVSample{Connected: true, RequestActive: true, PowerW: drawW(), PowerAt: time.Now()}, true + } + send := func(_ context.Context, driver string, payload []byte) error { + c := tel.BeginCommand(driver, payload, time.Now()) + tel.CompleteCommand(c, "accepted") + var cmd struct { + PowerW float64 `json:"power_w"` + } + if err := json.Unmarshal(payload, &cmd); err != nil { + return err + } + if cmd.PowerW != orderW { + orderW, orderAt = cmd.PowerW, time.Now() + } + return nil + } + ctrl := loadpoint.NewController(mgr, plan, sample, send) + ctrl.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3}) + ctrl.SetSitePhases(func() (loadpoint.SitePhases, bool) { + return loadpoint.SitePhases{A: [3]float64{1.6 + drawW()/230, 1.7, 2.3}, At: time.Now()}, true + }) + srv := New(&Deps{Tel: tel, Loadpoints: mgr, Cfg: &config.Config{Fuse: config.Fuse{MaxAmps: 20, Voltage: 230, Phases: 3}}, CfgMu: &sync.RWMutex{}}) + changedAt := time.Now() + last := -1.0 + var f ControlFeedback + for range 8 * 3600 / 5 { + ctrl.Tick(context.Background(), time.Now()) + if w := drawW(); w != last { + last, changedAt = w, time.Now() + } + tel.Update("easee", telemetry.DerEV, drawW(), nil, []byte(fmt.Sprintf( + `{"connected":true,"control_power_observed_at":%q,"control_power_confirmed":true,"power_max_age_s":180}`, + changedAt.Format(time.RFC3339Nano)))) + time.Sleep(5 * time.Second) + f = srv.controlFeedback(time.Now())[0] + if f.Status == "not_following" { + t.Fatalf("at %v a one-phase car drawing %.0f W of a %.0f W order reads %s/%s", time.Since(changedAt), drawW(), orderW, f.Status, f.Reason) + } + } + st, _ := mgr.State("garage") + if st.CarPhases != 1 || f.Reason != "one_phase" || f.Severity != "info" || f.ChargingPhase != "L1" || + f.ChargingPhaseMaxW == nil || *f.ChargingPhaseMaxW != 3680 || drawW() != 3680 { + t.Fatalf("after a night: car phases %d, draw %.0f W, verdict %s/%s/%s on %q", st.CarPhases, drawW(), f.Status, f.Reason, f.Severity, f.ChargingPhase) + } + }) +} diff --git a/go/internal/api/api_fleet_ping.go b/go/internal/api/api_fleet_ping.go index cc203a6d2..a07e3fb72 100644 --- a/go/internal/api/api_fleet_ping.go +++ b/go/internal/api/api_fleet_ping.go @@ -3,7 +3,9 @@ package api import ( "net/http" + "github.com/srcfl/ftw/go/internal/cloudlink" "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/energyreport" ) // What this box would add to FTW's daily fleet totals. @@ -19,8 +21,18 @@ type fleetPingView struct { // FTWRelay lets Settings state FTW's retention rules only when the running // process is using the endpoint those rules govern. FTWRelay bool `json:"ftw_relay"` + // ThroughRelay is true when the report goes to FTW Cloud through the + // Oblivious HTTP relay, so FTW Cloud never sees the box's address. + ThroughRelay bool `json:"through_relay"` + // WaitsForRelay is a box that shares by default, never having said yes: + // its report goes to FTW Cloud only through the relay, and this build + // names none, so it waits. + WaitsForRelay bool `json:"waits_for_relay"` // Payload is the whole message, exactly as it would be posted. Payload any `json:"payload"` + // Energy is the daily energy report under the same answer, built by the + // call its sender uses. Absent on a box without one. + Energy *energyreport.Preview `json:"energy,omitempty"` } func (s *Server) handleFleetPing(w http.ResponseWriter, r *http.Request) { @@ -31,18 +43,29 @@ func (s *Server) handleFleetPing(w http.ResponseWriter, r *http.Request) { return } - // The switch is live, but the endpoint is fixed when the pinger starts. - // Report that active endpoint rather than a newly saved value that needs a - // restart before it can be used. - s.deps.CfgMu.RLock() - enabled := s.deps.Cfg.FleetPing.On() - s.deps.CfgMu.RUnlock() + // Enabled is whether the next report goes, read the way the sender reads + // it: the box's one statistics answer, and no direct post to FTW Cloud + // from a box that never said yes. The endpoint is fixed when the pinger + // starts. Report that active endpoint rather than a newly saved value + // that needs a restart before it can be used. + enabled := s.deps.FleetPing.Enabled() endpoint := s.deps.FleetPing.Endpoint() - writeJSON(w, http.StatusOK, fleetPingView{ - Enabled: enabled, - Endpoint: endpoint, - FTWRelay: endpoint == config.DefaultFleetPingEndpoint, - Payload: s.deps.FleetPing.Payload(), - }) + view := fleetPingView{ + Enabled: enabled, + Endpoint: endpoint, + FTWRelay: endpoint == config.DefaultFleetPingEndpoint || s.deps.FleetPing.ThroughRelay(), + ThroughRelay: s.deps.FleetPing.ThroughRelay(), + Payload: s.deps.FleetPing.Payload(), + } + shares := s.deps.State != nil && cloudlink.SharesStatistics(s.deps.State) + if s.deps.Cloud != nil { + shares = s.deps.Cloud.Status().Statistics + } + view.WaitsForRelay = !enabled && !view.ThroughRelay && endpoint == config.DefaultFleetPingEndpoint && shares + if s.deps.EnergyReport != nil { + p := s.deps.EnergyReport.Preview(r.Context()) + view.Energy = &p + } + writeJSON(w, http.StatusOK, view) } diff --git a/go/internal/api/api_fleet_ping_test.go b/go/internal/api/api_fleet_ping_test.go index 306137d5b..ed543683d 100644 --- a/go/internal/api/api_fleet_ping_test.go +++ b/go/internal/api/api_fleet_ping_test.go @@ -5,12 +5,16 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "path/filepath" "sync" "testing" "time" + "github.com/srcfl/ftw/go/internal/cloudlink" "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/energyreport" "github.com/srcfl/ftw/go/internal/fleetping" + "github.com/srcfl/ftw/go/internal/state" ) type silentProvider string @@ -24,7 +28,13 @@ func fleetPingServer(t *testing.T, cfg *config.Config) *Server { func fleetPingServerAt(t *testing.T, cfg *config.Config, endpoint string) *Server { t.Helper() + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) pinger, err := fleetping.New(fleetping.Options{ + Enabled: func() bool { return cloudlink.SharesStatistics(st) }, Provider: silentProvider(endpoint), Facts: func() fleetping.Facts { return fleetping.FactsFromConfig(cfg, "v1.4.0", @@ -35,7 +45,12 @@ func fleetPingServerAt(t *testing.T, cfg *config.Config, endpoint string) *Serve if err != nil { t.Fatalf("fleetping.New: %v", err) } - return New(&Deps{CfgMu: &sync.RWMutex{}, Cfg: cfg, FleetPing: pinger}) + // The box's one statistics answer, written from this config the way + // Core writes it once at start. + if err := cloudlink.MigrateStatistics(st, !cfg.FleetPing.On()); err != nil { + t.Fatal(err) + } + return New(&Deps{CfgMu: &sync.RWMutex{}, Cfg: cfg, FleetPing: pinger, State: st}) } func TestFleetPingShowsTheRealPayload(t *testing.T) { @@ -149,3 +164,50 @@ func TestFleetPingSaysSoWhenItIsNotRunning(t *testing.T) { t.Fatalf("status %d, want 503", w.Code) } } + +// dayLedger is one site meter that recorded every bucket of every day. +type dayLedger struct{} + +func (dayLedger) EnergyAssets() ([]state.EnergyAsset, error) { + return []state.EnergyAsset{{AssetID: "m/grid_meter", Kind: state.AssetGridMeter, FirstSeenMS: 0, LastSeenMS: 1 << 62}}, nil +} + +func (dayLedger) LoadEnergyHistoryContext(_ context.Context, q state.EnergyHistoryQuery) ([]state.EnergyLedgerPoint, bool, error) { + var out []state.EnergyLedgerPoint + for at := q.SinceMS; at < q.UntilMS; at += state.EnergyLedgerBucketMS { + out = append(out, state.EnergyLedgerPoint{Flow: state.FlowGridImport, BucketStartMS: at, BucketLenMS: state.EnergyLedgerBucketMS, EnergyWh: 50, Quality: "measured"}) + } + return out, false, nil +} + +// The daily energy report rides under the same switch, so Settings shows it +// beside the fleet report, built by the call its sender uses. A build with no +// relay says it sends nothing. +func TestFleetPingShowsTheEnergyReportItWouldSend(t *testing.T) { + cfg := &config.Config{FleetPing: &config.FleetPing{Enabled: true}} + srv := fleetPingServer(t, cfg) + sender, err := energyreport.New(energyreport.Options{Ledger: dayLedger{}, Enabled: func() bool { return true }, + Now: func() time.Time { return time.Date(2026, 10, 24, 9, 0, 0, 0, time.UTC) }}) + if err != nil { + t.Fatal(err) + } + srv.deps.EnergyReport = sender + + w := httptest.NewRecorder() + srv.Handler().ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/api/fleet-ping", nil)) + var got struct { + Energy struct { + Relay string `json:"relay"` + Sending bool `json:"sending"` + Day string `json:"day"` + Payload map[string]any `json:"payload"` + } `json:"energy"` + } + if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil { + t.Fatalf("decode: %v", err) + } + e := got.Energy + if e.Relay != "" || e.Sending || e.Day != "2026-10-23" || e.Payload["grid_import_kwh"] != 14.0 || e.Payload["date"] != "2026-10-23" || len(e.Payload) != 8 { + t.Fatalf("energy = %+v", e) + } +} diff --git a/go/internal/api/api_history_quality.go b/go/internal/api/api_history_quality.go index a53a7c21c..063299a51 100644 --- a/go/internal/api/api_history_quality.go +++ b/go/internal/api/api_history_quality.go @@ -18,9 +18,6 @@ func (s *Server) qualifyStatusHistory(resp map[string]any, siteMeter string) { opts = s.deps.HistoryMeasurementSources() } r := s.deps.Tel.LiveMeasurementNow(siteMeter, opts) - // Interval metadata remains separate from the current live balance. - opts.UseWatchdogTimeout, opts.UseControlPower = true, true - resp["load_estimate"] = s.deps.Tel.HomeUseEstimate(time.Now(), siteMeter, opts) var missing state.HistoryMissing for kind, flag := range map[telemetry.DerType]state.HistoryMissing{telemetry.DerMeter: state.MissingGrid, telemetry.DerPV: state.MissingPV, telemetry.DerBattery: state.MissingBattery, telemetry.DerEV: state.MissingEV, telemetry.DerV2X: state.MissingV2X} { if !r.Known[kind] { @@ -37,14 +34,26 @@ func (s *Server) qualifyStatusHistory(resp map[string]any, siteMeter string) { missing |= state.BalanceMismatch } resp["grid_w"] = missing.Value(state.MissingGrid, r.GridW) - resp["pv_w"] = missing.Value(state.MissingPV, r.PVW) + // Live power is not history: a dark inverter is 0 W and Home use stays + // known while the site meter is read (load_quality says how). + if r.PVLiveKnown && opts.PVInvalidReason == "" { + resp["pv_w"] = r.PVW + } else { + resp["pv_w"] = nil + } resp["bat_w"] = missing.Value(state.MissingBattery, r.BatteryW) resp["ev_w"] = missing.Value(state.MissingEV, r.EVW) if missing&state.MissingEV != 0 { resp["ev_charging_w"] = nil } resp["v2x_w"] = missing.Value(state.MissingV2X, r.V2XW) - resp["load_w"] = missing.Value(state.MissingLoad, r.HouseholdW) + if r.LoadKnown { + resp["load_w"] = r.LoadW + } else { + resp["load_w"] = nil + } + resp["load_quality"] = r.LoadQuality + resp["load_reason"] = r.LoadReason resp["history_quality"] = missing.Quality() resp["history_reason"] = r.Reason drivers, _ := resp["drivers"].(map[string]any) diff --git a/go/internal/api/api_history_quality_test.go b/go/internal/api/api_history_quality_test.go index 92bfd4e49..9822e7a58 100644 --- a/go/internal/api/api_history_quality_test.go +++ b/go/internal/api/api_history_quality_test.go @@ -77,13 +77,15 @@ func TestLiveHistoryQualitySeparatesOfflineFromInvalidBalance(t *testing.T) { }}) resp := map[string]any{} srv.qualifyStatusHistory(resp, "meter") - if resp["grid_w"] != float64(1200) || resp["bat_w"] != nil || resp["load_w"] != nil || resp["history_quality"] != "incomplete" { + // Live Home use leaves the offline battery out; history stays incomplete. + if resp["grid_w"] != float64(1200) || resp["bat_w"] != nil || resp["load_w"] != float64(1200) || resp["load_quality"] != "estimated" || resp["history_quality"] != "incomplete" { t.Fatal(resp) } tel.Update("battery", telemetry.DerBattery, 2000, nil, nil) tel.RecordDriverSuccess("battery") srv.qualifyStatusHistory(resp, "meter") - if resp["bat_w"] != float64(2000) || resp["load_w"] != nil || resp["history_quality"] != "balance_mismatch" { + // Negative balance: shown as is, history still records the mismatch. + if resp["bat_w"] != float64(2000) || resp["load_w"] != float64(-800) || resp["load_reason"] != "invalid_balance" || resp["history_quality"] != "balance_mismatch" { t.Fatal(resp) } tel.Update("battery", telemetry.DerBattery, 0, nil, nil) @@ -130,7 +132,50 @@ func TestLiveHistoryQualityConfiguredPVWithoutMeasurementIsUnknown(t *testing.T) }}) resp := map[string]any{} srv.qualifyStatusHistory(resp, "meter") - if resp["pv_w"] != nil || resp["load_w"] != nil { - t.Fatal("unmeasured configured PV reported as zero", resp) + if resp["pv_w"] != nil || resp["load_w"] != float64(1200) || resp["load_quality"] != "estimated" || resp["history_quality"] != "incomplete" { + t.Fatal("unmeasured configured PV reported as zero, or Home use blanked", resp) + } +} + +// Anders (Discord, v0.154.1): Growatt offline after sunset, Pixii meter and +// battery read. /api/status showed pv_w and load_w null. +func TestStatusAndersNightAndDay(t *testing.T) { + for _, tc := range []struct { + name string + dark telemetry.Darkness + pv any + quality string + loadWant any + }{ + {"night", telemetry.DarknessNight, float64(0), "measured", float64(980)}, + {"day", telemetry.DarknessUnknown, nil, "estimated", float64(980)}, + } { + t.Run(tc.name, func(t *testing.T) { + tel := telemetry.NewStore() + for _, name := range []string{"pixii", "growatt"} { + tel.EnsureDriverHealth(name) + } + tel.Update("pixii", telemetry.DerMeter, 780, nil, nil) + tel.Update("pixii", telemetry.DerBattery, -200, nil, nil) + tel.RecordDriverSuccess("pixii") + tel.Update("growatt", telemetry.DerPV, -40, nil, nil) + tel.DriverHealthMut("growatt").SetOffline() + dark := tc.dark + srv := New(&Deps{Tel: tel, HistoryMeasurementSources: func() telemetry.ForecastOptions { + return telemetry.ForecastOptions{PVDark: func(time.Time) telemetry.Darkness { return dark }} + }}) + resp := map[string]any{"drivers": map[string]any{"growatt": map[string]any{}}} + srv.qualifyStatusHistory(resp, "pixii") + if resp["grid_w"] != float64(780) || resp["pv_w"] != tc.pv || resp["load_w"] != tc.loadWant || resp["load_quality"] != tc.quality { + t.Fatal(resp) + } + // History quality stays separate from live Home use. + if resp["history_quality"] != "incomplete" || resp["history_reason"] != "offline:growatt:pv" { + t.Fatal("live display changed history quality", resp) + } + if tc.dark == telemetry.DarknessNight && resp["drivers"].(map[string]any)["growatt"].(map[string]any)["pv_w"] == nil { + t.Fatal("dark inverter not shown as 0 W") + } + }) } } diff --git a/go/internal/api/api_home_estimate_test.go b/go/internal/api/api_home_estimate_test.go index 02ad03018..6984af761 100644 --- a/go/internal/api/api_home_estimate_test.go +++ b/go/internal/api/api_home_estimate_test.go @@ -2,7 +2,6 @@ package api import ( "encoding/json" - "math" "testing" "time" @@ -10,7 +9,9 @@ import ( "github.com/srcfl/ftw/go/internal/telemetry" ) -func TestStatusHomeEstimateIsSeparateFromInstantaneousPower(t *testing.T) { +// Status sends one Home use: the live balance and its quality. A second, +// energy-averaged number made every client show the live value as an estimate. +func TestStatusSendsOneHomeUse(t *testing.T) { tel := telemetry.NewStore() tel.EnsureDriverHealth("site") base := time.Now().Add(-20 * time.Second) @@ -22,17 +23,16 @@ func TestStatusHomeEstimateIsSeparateFromInstantaneousPower(t *testing.T) { resp := map[string]any{} for i := 0; i < 3; i++ { srv.qualifyStatusHistory(resp, "site") - e, ok := resp["load_estimate"].(telemetry.HomeEstimate) - if !ok || e.Watts == nil || math.Abs(*e.Watts-1500) > 1e-6 || e.AgeMS < 15000 || e.AgeMS > 20000 || e.Method != "reported_energy_balance_v3" || e.TimingBasis != "source" || e.EnergyBasis != "power_integral" || e.EnergyWh == nil || e.EndAgeMS < 10000 { - t.Fatal(e) + if _, ok := resp["load_estimate"]; ok { + t.Fatal("status sent a second Home use") } - if resp["load_w"] != float64(2000) || resp["grid_w"] != float64(2000) || resp["history_quality"] != "complete" { + if resp["load_w"] != float64(2000) || resp["load_quality"] != telemetry.LoadMeasured || resp["grid_w"] != float64(2000) || resp["history_quality"] != "complete" { t.Fatal(resp) } } tel.DriverHealthMut("site").SetOffline() srv.qualifyStatusHistory(resp, "site") - if resp["load_w"] != nil || resp["load_estimate"].(telemetry.HomeEstimate).Watts != nil { + if resp["load_w"] != nil || resp["load_quality"] != telemetry.LoadUnknown { t.Fatal(resp) } } @@ -60,19 +60,17 @@ func TestStatusKeepsPowerBalanceWhenSessionEnergyCannotFormAnInterval(t *testing if resp["load_w"] != float64(1234) || resp["history_quality"] != "complete" { t.Fatal("valid power balance disappeared") } - if e := resp["load_estimate"].(telemetry.HomeEstimate); e.Watts != nil || e.Reason != "incomplete_source_window" { - t.Fatal("old session was presented as an energy interval") - } tel.DriverHealthMut("charger").SetOffline() srv.qualifyStatusHistory(resp, "site") - if resp["load_w"] != nil || resp["load_estimate"].(telemetry.HomeEstimate).Watts != nil { - t.Fatal("offline charger became zero load") + // The offline charger is left out of live Home use, which says so. + if resp["load_w"] != float64(1234) || resp["load_quality"] != "estimated" { + t.Fatal("offline charger blanked Home use") } tel.RecordDriverSuccess("charger") tel.Update("charger", telemetry.DerEV, 6000, nil, nil) srv.qualifyStatusHistory(resp, "site") - if resp["load_w"] != nil || resp["history_reason"] != "invalid_balance" { - t.Fatal("negative residual became a zero household balance") + if resp["load_w"] != float64(1234-6000) || resp["history_reason"] != "invalid_balance" { + t.Fatal("negative residual was hidden or zeroed") } } @@ -96,12 +94,13 @@ func TestStatusHomePowerUsesPixiiACWithoutChangingRawHistoryInputs(t *testing.T) } tel.Update("battery", telemetry.DerBattery, -2500, nil, json.RawMessage(`{"control_power_available":false}`)) srv.qualifyStatusHistory(resp, "site") - if resp["load_w"] != nil || resp["bat_w"] != nil { - t.Fatal("missing AC measurement became DC power or zero") + // Unknown AC power leaves the battery out; the balance is shown as is. + if resp["load_w"] != float64(5000-6000) || resp["bat_w"] != nil || resp["load_quality"] != "estimated" { + t.Fatal("missing AC measurement became DC power or hid the balance") } tel.Update("battery", telemetry.DerBattery, -2500, nil, json.RawMessage(`{"control_power_w":-500,"control_power_available":true}`)) srv.qualifyStatusHistory(resp, "site") - if resp["load_w"] != nil || resp["history_reason"] != "invalid_balance" { + if resp["load_w"] != float64(5000-(-500)-6000) || resp["history_reason"] != "invalid_balance" { t.Fatal("negative AC balance became zero or a DC estimate") } } diff --git a/go/internal/api/api_loadpoint_manual.go b/go/internal/api/api_loadpoint_manual.go index 22380cacc..a63134d59 100644 --- a/go/internal/api/api_loadpoint_manual.go +++ b/go/internal/api/api_loadpoint_manual.go @@ -249,14 +249,16 @@ func (s *Server) handleLoadpointManualHoldGet(w http.ResponseWriter, r *http.Req // slider (range from min/max charge W ÷ phases ÷ voltage) and reflect // whether a manual hold is currently pinned. Phases come from the // loadpoint's phase_mode (1p/3p) or the site fuse; voltage from the -// site fuse. Mutates states in place. +// site fuse. A charger that cannot run three phases reports one unless +// pinned to 3p, as Core runs it. Mutates states in place. func (s *Server) decorateLoadpointsWithManual(states []loadpoint.State) { var fusePhases int - var voltage float64 + var voltage, maxAmps float64 if s.deps.CfgMu != nil && s.deps.Cfg != nil { s.deps.CfgMu.RLock() fusePhases = s.deps.Cfg.Fuse.Phases voltage = s.deps.Cfg.Fuse.Voltage + maxAmps = s.deps.Cfg.Fuse.MaxAmps s.deps.CfgMu.RUnlock() } if fusePhases <= 0 { @@ -266,10 +268,11 @@ func (s *Server) decorateLoadpointsWithManual(states []loadpoint.State) { voltage = 230 } - phaseModeByID := map[string]string{} + site := loadpoint.SiteFuse{MaxAmps: maxAmps, Voltage: voltage, PhaseCnt: fusePhases} + cfgByID := map[string]loadpoint.Config{} if s.deps.Loadpoints != nil { for _, c := range s.deps.Loadpoints.Configs() { - phaseModeByID[c.ID] = c.PhaseMode + cfgByID[c.ID] = c } } @@ -277,11 +280,14 @@ func (s *Server) decorateLoadpointsWithManual(states []loadpoint.State) { chargers := s.chargerReadings() for i := range states { phases := fusePhases - switch phaseModeByID[states[i].ID] { - case "1p": + cfg, known := cfgByID[states[i].ID] + switch { + case cfg.PhaseMode == "1p": phases = 1 - case "3p": + case cfg.PhaseMode == "3p": phases = 3 + case known && !loadpoint.ThreePhaseCapable(cfg, site): + phases = 1 } states[i].Phases = phases states[i].VoltageV = voltage diff --git a/go/internal/api/api_loadpoint_phases_test.go b/go/internal/api/api_loadpoint_phases_test.go new file mode 100644 index 000000000..722fe5924 --- /dev/null +++ b/go/internal/api/api_loadpoint_phases_test.go @@ -0,0 +1,53 @@ +package api + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "sync" + "testing" + + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/loadpoint" +) + +// The app sends Charge now with the phases the API reports. A charger whose +// table stops below 6 A on three phases runs on one, so it reports one +// unless pinned to 3p; a loadpoint that can run three reports three. +func TestLoadpointsReportTheChargersPhases(t *testing.T) { + onePhase := []float64{0, 1380, 2300, 3680} + mixed := []float64{0, 1380, 2300, 3680, 4140, 6900, 11040} + for _, tc := range []struct { + name string + mode string + steps []float64 + maxW float64 + want int + }{ + {"32 A one-phase table, auto", "auto", onePhase, 7360, 1}, + {"one-phase table, unset", "", onePhase, 7360, 1}, + {"one-phase table pinned 3p", "3p", onePhase, 7360, 3}, + {"mixed table, auto", "auto", mixed, 11040, 3}, + {"no table, unset", "", nil, 11000, 3}, + {"pinned 1p", "1p", mixed, 11040, 1}, + } { + t.Run(tc.name, func(t *testing.T) { + mgr := loadpoint.NewManager() + mgr.Load([]loadpoint.Config{{ID: "garage", DriverName: "easee", PhaseMode: tc.mode, AllowedStepsW: tc.steps, + MinChargeW: 1380, MaxChargeW: tc.maxW}}) + cfg := &config.Config{Fuse: config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230}} + srv := New(&Deps{Loadpoints: mgr, Cfg: cfg, CfgMu: &sync.RWMutex{}}) + rr := httptest.NewRecorder() + srv.Handler().ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/api/loadpoints", nil)) + var body struct { + Loadpoints []loadpoint.State `json:"loadpoints"` + } + if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil || len(body.Loadpoints) != 1 { + t.Fatalf("GET /api/loadpoints = %d: %s", rr.Code, rr.Body.String()) + } + if got := body.Loadpoints[0].Phases; got != tc.want { + t.Fatalf("phases %d, want %d", got, tc.want) + } + }) + } +} diff --git a/go/internal/api/api_myuplink_oauth.go b/go/internal/api/api_myuplink_oauth.go index 535d32262..b118832fc 100644 --- a/go/internal/api/api_myuplink_oauth.go +++ b/go/internal/api/api_myuplink_oauth.go @@ -64,7 +64,11 @@ func validMyUplinkCallback(raw string) bool { if u.Host == "" { return false } - return u.Path == myUplinkCallbackPath && u.RawQuery == "" && u.Fragment == "" + callbackPath := u.Path == myUplinkCallbackPath + if prefix, ok := strings.CutSuffix(u.Path, myUplinkCallbackPath); ok && ingressPath.MatchString(prefix) { + callbackPath = true + } + return callbackPath && u.RawQuery == "" && u.Fragment == "" } // oauthPending is an in-flight authorization-code request, created by /start @@ -113,6 +117,10 @@ func pruneOAuthStateLocked(now time.Time) { // byte-for-byte across authorize + token exchange + portal registration, so // it is derived once and stored with the pending state. func requestOrigin(r *http.Request) string { + if HomeAssistantIngress(r) { + base, _ := r.Context().Value(homeAssistantURLKey{}).(string) + return base + } scheme := "http" if r.TLS != nil { scheme = "https" diff --git a/go/internal/api/api_one_phase_order_test.go b/go/internal/api/api_one_phase_order_test.go new file mode 100644 index 000000000..9e153204b --- /dev/null +++ b/go/internal/api/api_one_phase_order_test.go @@ -0,0 +1,71 @@ +package api + +import ( + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// For a one-phase car the command's power is a third of the order. +// "Charging on one phase" explains a draw that matches it; a limit, a +// changed setpoint or a response still due says more and comes first, and +// a draw beyond tolerance is judged as for any car. +func TestOnePhaseVerdictComesAfterLimits(t *testing.T) { + now := time.Now() + connected := true + for _, tc := range []struct { + name string + marked bool + core string + sent, actual float64 + requestedA, offeredA float64 + mismatch, over bool // since 12 minutes + gap float64 + observedBeforeCommand bool + readbackMismatch bool + state, reason string + }{ + {"fuse-limited order, car takes its third", true, "fuse_limit", 1380, 1380, 6, 6, false, false, 0, false, false, "limited", "fuse_limit"}, + {"same, unmarked car taking the order", false, "fuse_limit", 4140, 4140, 6, 6, false, false, 0, false, false, "limited", "fuse_limit"}, + {"import-limited, car takes its third", true, "import_limit", 1610, 1600, 7, 7, false, false, 0, false, false, "limited", "import_limit"}, + {"fuse cooldown, car takes its third", true, "fuse_cooldown", 1380, 1380, 6, 6, false, false, 0, false, false, "limited", "fuse_cooldown"}, + {"meter stale, car takes its third", true, "site_meter_stale", 1380, 1380, 6, 6, false, false, 0, false, false, "limited", "site_meter_stale"}, + {"setpoint changed by another app, draw matches", true, "", 3680, 3680, 16, 16, false, false, 0, false, true, "warning", "setpoint_changed"}, + {"reading older than the command", true, "", 2300, 2300, 10, 10, false, false, 0, true, false, "waiting", "waiting_response"}, + {"plan order, car takes its third", true, "", 3680, 3650, 16, 16, false, false, 0, false, false, "limited", "one_phase"}, + {"overdraw: charger ignores a cut 16->10 A", true, "", 2300, 3680, 10, 16, true, true, 1380, false, false, "warning", "power_above_target"}, + {"overdraw under fuse_limit", true, "fuse_limit", 2300, 3680, 10, 16, true, true, 1380, false, false, "warning", "power_above_target"}, + {"shortfall: car takes 1.5 of 3.7 kW", true, "", 3680, 1500, 16, 16, true, false, 2180, false, false, "warning", "power_below_target"}, + {"wrong mark: 3p car draws the full order", true, "", 3680, 11040, 16, 16, true, true, 7360, false, false, "warning", "power_above_target"}, + {"overdraw inside tolerance (+300 W on 3.68 kW)", true, "", 3680, 3980, 16, 16, true, true, 300, false, false, "limited", "one_phase"}, + } { + t.Run(tc.name, func(t *testing.T) { + c := telemetry.CommandEvidence{Result: "accepted", Since: now.Add(-15 * time.Minute), LastObservation: now} + if tc.mismatch { + c.PowerMismatchSince = now.Add(-12 * time.Minute) + c.LastGapW = &tc.gap + } else { + c.PowerMatchSince = now.Add(-time.Minute) + } + if tc.over { + c.PowerOverSince = now.Add(-12 * time.Minute) + } + if tc.readbackMismatch { + c.ReadbackMismatchSince = now.Add(-time.Minute) + } + f := ControlFeedback{Kind: "ev", SentW: watts(tc.sent), ActualW: watts(tc.actual), ReadbackW: watts(tc.sent), + ObservedAtMs: now.UnixMilli(), Response: "device_reported", RequestedA: watts(tc.requestedA), OfferedA: watts(tc.offeredA)} + if tc.observedBeforeCommand { + f.ObservedAtMs = now.Add(-20 * time.Minute).UnixMilli() + } + if tc.marked { + f.ChargingPhase = "L1" + } + classifyControlFeedback(&f, c, true, true, tc.core, &connected, now) + if f.State != tc.state || f.Reason != tc.reason { + t.Fatalf("%s/%s, want %s/%s", f.State, f.Reason, tc.state, tc.reason) + } + }) + } +} diff --git a/go/internal/api/api_support_live_load_test.go b/go/internal/api/api_support_live_load_test.go index 1a08d7845..76ab304e8 100644 --- a/go/internal/api/api_support_live_load_test.go +++ b/go/internal/api/api_support_live_load_test.go @@ -70,7 +70,7 @@ func TestSupportLiveLoadMatchesLANSourcePowerAndKnownZero(t *testing.T) { } func TestSupportUnknownLoadRendersUnavailableWithoutForecastClaims(t *testing.T) { - for _, scenario := range []string{"negative_balance", "declared_never_emitted", "stale_power"} { + for _, scenario := range []string{"stale_power"} { t.Run(scenario, func(t *testing.T) { s, ctrl, tel := reportTestServer(t) switch scenario { @@ -159,3 +159,28 @@ func TestSupportUnknownLoadNeverBlamesEitherForecast(t *testing.T) { }) } } + +func TestSupportEstimatedLoadRendersEstimateWithoutForecastClaims(t *testing.T) { + s, ctrl, _ := reportTestServer(t) + s.deps.HistoryMeasurementSources = func() telemetry.ForecastOptions { + return telemetry.ForecastOptions{ExpectedFlows: []telemetry.ForecastFlow{{Driver: "battery", DerType: telemetry.DerBattery}}} + } + if st := supportLoadStatus(t, s); st["load_w"] == nil || st["load_quality"] != "estimated" { + t.Fatal("left-out battery blanked LAN Home use", st) + } + snap := s.liveNow(*ctrl, time.Now()) + if snap.HaveLoad || !snap.LoadEstimated { + t.Fatal("estimated load counted as measured") + } + snap.PredictedLd = 50000 + var b strings.Builder + writeRightNow(&b, *ctrl, snap, nil, nil, control.SlotEnergySnapshot{}, time.Now()) + if !strings.Contains(b.String(), "(estimated: `missing:battery:battery`)") { + t.Fatal("estimate not rendered as one", b.String()) + } + for _, f := range s.collectFindings(*ctrl, snap, nil, nil, nil, nil, control.SlotEnergySnapshot{}, time.Now()) { + if strings.Contains(f.Title, "load forecast") { + t.Fatal("estimated load caused a forecast claim") + } + } +} diff --git a/go/internal/api/api_support_report.go b/go/internal/api/api_support_report.go index c78564349..9741f1f08 100644 --- a/go/internal/api/api_support_report.go +++ b/go/internal/api/api_support_report.go @@ -155,11 +155,15 @@ type liveSnapshot struct { EVW float64 V2XW float64 LoadW float64 - HaveLoad bool - SoCPct float64 - HaveSoC bool - PredictedPV float64 - PredictedLd float64 + // HaveLoad: a measured balance, fit for forecast comparisons. + // LoadEstimated: the site meter is read but a flow was left out. + HaveLoad bool + LoadEstimated bool + LoadReason string + SoCPct float64 + HaveSoC bool + PredictedPV float64 + PredictedLd float64 } func (s *Server) liveNow(ctrl control.State, now time.Time) liveSnapshot { @@ -172,7 +176,8 @@ func (s *Server) liveNow(ctrl control.State, now time.Time) liveSnapshot { GridW: live.GridW, HaveGrid: live.Known[telemetry.DerMeter], PVW: live.PVW, HavePV: live.Known[telemetry.DerPV], BatW: live.BatteryW, HaveBattery: live.Known[telemetry.DerBattery], - EVW: live.EVW, V2XW: live.V2XW, LoadW: live.HouseholdW, HaveLoad: live.Valid, + EVW: live.EVW, V2XW: live.V2XW, LoadW: live.LoadW, HaveLoad: live.LoadKnown && live.LoadQuality == telemetry.LoadMeasured, + LoadEstimated: live.LoadKnown && live.LoadQuality == telemetry.LoadEstimated, LoadReason: live.LoadReason, } var socSum, socWeight float64 for _, r := range s.deps.Tel.ReadingsByType(telemetry.DerBattery) { @@ -316,7 +321,11 @@ func writeRightNow( b.WriteString("| Grid | **no site meter reading** |\n") } fmt.Fprintf(b, "| Solar | %s |\n", fmtReportLiveW(snap.PVW, snap.HavePV)) - fmt.Fprintf(b, "| House load | %s |\n", fmtReportLiveW(snap.LoadW, snap.HaveLoad)) + if snap.LoadEstimated { + fmt.Fprintf(b, "| House load | ≈ %s (estimated: `%s`) |\n", fmtReportW(snap.LoadW), snap.LoadReason) + } else { + fmt.Fprintf(b, "| House load | %s |\n", fmtReportLiveW(snap.LoadW, snap.HaveLoad)) + } fmt.Fprintf(b, "| Battery | %s |\n", fmtReportLiveW(snap.BatW, snap.HaveBattery)) if snap.EVW != 0 { fmt.Fprintf(b, "| EV | %s |\n", fmtReportW(snap.EVW)) @@ -711,7 +720,11 @@ func (s *Server) collectFindings( "guesswork until the meter reports again."}) } - if !snap.HaveLoad { + if snap.LoadEstimated { + out = append(out, finding{sevNote, "Household load is an estimate", + fmt.Sprintf("The site meter is read, but `%s` is left out of the balance. "+ + "Load forecast comparisons are skipped until every flow is read.", snap.LoadReason)}) + } else if !snap.HaveLoad { out = append(out, finding{sevWarning, "Household load is unavailable", "The required power measurements do not form a valid balance. " + "Load forecast comparisons are skipped until that balance is available."}) diff --git a/go/internal/api/api_test.go b/go/internal/api/api_test.go index 303875b54..45f972fca 100644 --- a/go/internal/api/api_test.go +++ b/go/internal/api/api_test.go @@ -216,8 +216,8 @@ func TestHandleStatusKeepsOfflineFlowsUnavailable(t *testing.T) { // balance. This policy also applies when the config callback is absent. for field, want := range map[string]any{ "grid_w": float64(8000), "ev_w": float64(1000), - "pv_w": nil, "bat_w": nil, "load_w": nil, "bat_soc": nil, - "history_quality": "incomplete", + "pv_w": nil, "bat_w": nil, "load_w": float64(10000), "bat_soc": nil, + "history_quality": "incomplete", "load_quality": "estimated", } { if resp[field] != want { t.Fatalf("%s differs from the qualified live value", field) diff --git a/go/internal/api/cloud_statistics.go b/go/internal/api/cloud_statistics.go index 5f47fea75..3769f2d63 100644 --- a/go/internal/api/cloud_statistics.go +++ b/go/internal/api/cloud_statistics.go @@ -140,6 +140,6 @@ func (s *Server) handleCloudStatsPolicy(w http.ResponseWriter, r *http.Request) "retention": "48 hours. Leaving Cloud or stopping local sharing deletes unpublished household rows; published group totals remain.", "publication_minimum_homes": cloudwire.StatsMinimumHomes, "publication_max_home_share": 0.3, - "controller": "sourceful", "local_default": "ask_yes", "local_optional": true, "cloud_required": true, + "controller": "sourceful", "local_default": "on", "local_optional": true, "cloud_required": true, }) } diff --git a/go/internal/api/ev_headroom_export_regression_test.go b/go/internal/api/ev_headroom_export_regression_test.go new file mode 100644 index 000000000..f5de0722c --- /dev/null +++ b/go/internal/api/ev_headroom_export_regression_test.go @@ -0,0 +1,40 @@ +package api + +import ( + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +func TestEVHeadroomRequiresPhaseImportDirection(t *testing.T) { + for _, tc := range []struct { + name string + grid float64 + data string + }{ + {"missing phase power", 13800, `{"l1_a":20,"l2_a":20,"l3_a":20}`}, + {"missing one phase power", 13800, `{"l1_a":20,"l2_a":20,"l3_a":20,"l1_w":4600,"l2_w":4600}`}, + {"null phase power", 13800, `{"l1_a":20,"l2_a":20,"l3_a":20,"l1_w":null,"l2_w":4600,"l3_w":4600}`}, + {"zero phase power leaves direction unknown", 9200, `{"l1_a":20,"l2_a":20,"l3_a":20,"l1_w":0,"l2_w":4600,"l3_w":4600}`}, + {"all phases export with magnitude currents", -13800, `{"l1_a":20,"l2_a":20,"l3_a":20,"l1_w":-4600,"l2_w":-4600,"l3_w":-4600}`}, + {"mixed flow with net import and magnitude currents", 920, `{"l1_a":20,"l2_a":12,"l3_a":12,"l1_w":-4600,"l2_w":2760,"l3_w":2760}`}, + } { + t.Run(tc.name, func(t *testing.T) { + tel := telemetry.NewStore() + tel.Update("meter", telemetry.DerMeter, tc.grid, nil, []byte(tc.data)) + tel.DriverHealthMut("meter").RecordSuccess() + cfg := &config.Config{Fuse: config.Fuse{MaxAmps: 25, Phases: 3, Voltage: 230}, Loadpoints: []config.Loadpoint{{DriverName: "ev", PhaseMode: "3p"}}} + srv := New(&Deps{Tel: tel, Cfg: cfg, CfgMu: &sync.RWMutex{}}) + now := time.Now() + f := ControlFeedback{Driver: "ev", Kind: "ev", Reason: "power_below_target", ReadingsFresh: true, SentW: watts(10350), ActualW: watts(4155), ObservedAtMs: now.UnixMilli()} + got := srv.evSiteHeadroom(f, loadpoint.State{Phases: 3, VoltageV: 230}, "meter", now) + if got != nil { + t.Fatalf("unqualified phase direction produced an import-headroom explanation: %+v", got) + } + }) + } +} diff --git a/go/internal/api/home_assistant.go b/go/internal/api/home_assistant.go new file mode 100644 index 000000000..3a7b99893 --- /dev/null +++ b/go/internal/api/home_assistant.go @@ -0,0 +1,145 @@ +package api + +import ( + "bytes" + "context" + "net/http" + "regexp" + "strings" + + "github.com/srcfl/ftw/go/internal/apiauth" +) + +// Supervisor authenticates the HA user before forwarding ingress requests. +// Never infer that trust from a forwarded header on the ordinary LAN listener. +type homeAssistantContextKey struct{} +type homeAssistantURLKey struct{} + +func HomeAssistantIngress(r *http.Request) bool { + trusted, _ := r.Context().Value(homeAssistantContextKey{}).(bool) + return trusted +} + +var ingressPath = regexp.MustCompile(`^/api/hassio_ingress/[A-Za-z0-9_-]+$`) + +// WithHomeAssistantIngress belongs only on the HA bundle's ingress listener. +// It admits Supervisor, plus loopback GET /api/health for Docker's healthcheck. +// API responses and streaming requests pass through without body rewriting. +func WithHomeAssistantIngress(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if isLoopbackClient(r.RemoteAddr) && r.Method == http.MethodGet && r.URL.Path == "/api/health" { + next.ServeHTTP(w, r) + return + } + prefix := r.Header.Get("X-Ingress-Path") + if remoteHost(r) != "172.30.32.2" || !ingressPath.MatchString(prefix) || r.Header.Get("X-Remote-User-Id") == "" { + http.Error(w, "Open FTW through Home Assistant", http.StatusForbidden) + return + } + authority, err := parseAuthority(r.Host) + if err != nil || (requiresMutationProtection(r) && validateFetchMetadata(r, authority) != nil) { + http.Error(w, "invalid Home Assistant request origin", http.StatusForbidden) + return + } + externalBase := requestOrigin(r) + prefix + ctx := context.WithValue(r.Context(), homeAssistantURLKey{}, externalBase) + r = r.Clone(context.WithValue(ctx, homeAssistantContextKey{}, true)) + // This is an authenticated local owner, including when an existing install + // still has a house password. Preserve the Supervisor peer for local gates. + r = r.WithContext(withLANSecret(r.Context(), true)) + r = r.WithContext(apiauth.WithCaller(r.Context(), apiauth.Caller{ + Subject: "ha:" + r.Header.Get("X-Remote-User-Id"), Kind: apiauth.KindLAN, + Role: apiauth.RoleOwner, Scopes: apiauth.EveryScope(), + })) + // Check HA's origin above, then present the authenticated local request to + // existing API guards. Ordinary proxies never reach this normalization. + r.Host = "ftw.local" + if r.Header.Get("Origin") != "" { + r.Header.Set("Origin", "http://ftw.local") + } + for name := range r.Header { + lower := strings.ToLower(name) + if lower == "forwarded" || lower == "x-real-ip" || strings.HasPrefix(lower, "x-forwarded-") { + r.Header.Del(name) + } + } + if strings.HasPrefix(r.URL.Path, "/api/") { + next.ServeHTTP(w, r) + return + } + // The on-box UI uses root URLs. Adapt only its static HTML/JS/CSS to HA's + // installation-specific prefix; never rewrite user data, API JSON or files. + r.Header.Del("Range") + r.Header.Del("If-Modified-Since") + r.Header.Del("If-None-Match") + response := &ingressResponse{headers: make(http.Header)} + next.ServeHTTP(response, r) + for name, values := range response.Header() { + w.Header()[name] = values + } + if location := w.Header().Get("Location"); strings.HasPrefix(location, "/") && !strings.HasPrefix(location, "//") { + w.Header().Set("Location", prefix+location) + } + body := response.body.Bytes() + kind := w.Header().Get("Content-Type") + if strings.HasPrefix(kind, "text/html") || strings.Contains(kind, "javascript") || strings.HasPrefix(kind, "text/css") { + body = ingressAsset(body, prefix, strings.HasPrefix(kind, "text/html")) + w.Header().Del("Content-Length") + w.Header().Del("Etag") + w.Header().Set("Cache-Control", "no-store") + } + w.WriteHeader(response.statusCode()) + if r.Method != http.MethodHead { + _, _ = w.Write(body) + } + }) +} + +// Match complete root URL starts, not arbitrary slashes (split('/'), API path +// suffixes such as '/draft', protocol-relative URLs, or data in API replies). +var ingressAssetURL = regexp.MustCompile("([\"'`(])(/(?:api/|components/|settings/|vendor/|setup(?:[?\"'`#]|$)|[A-Za-z0-9_-]+\\.(?:js|mjs|css|html|svg|png|ico|woff2?)))") + +func ingressAsset(body []byte, prefix string, html bool) []byte { + body = ingressAssetURL.ReplaceAll(body, []byte("${1}"+prefix+"${2}")) + for _, quote := range []string{"'", "\""} { + for _, start := range []string{"window.location.href = ", "href="} { + body = bytes.ReplaceAll(body, []byte(start+quote+"/"+quote), []byte(start+quote+prefix+"/"+quote)) + } + body = bytes.ReplaceAll(body, []byte(quote+"/?"), []byte(quote+prefix+"/?")) + } + if html { + script := `` + body = bytes.Replace(body, []byte(""), []byte(script+""), 1) + } + return body +} + +// Only static responses use this buffer. API handlers keep the original +// ResponseWriter, including its streaming and WebSocket interfaces. +type ingressResponse struct { + headers http.Header + body bytes.Buffer + status int +} + +func (w *ingressResponse) Header() http.Header { return w.headers } +func (w *ingressResponse) WriteHeader(status int) { + if w.status == 0 { + w.status = status + } +} +func (w *ingressResponse) Write(body []byte) (int, error) { + if w.status == 0 { + w.status = http.StatusOK + } + if w.headers.Get("Content-Type") == "" { + w.headers.Set("Content-Type", http.DetectContentType(body)) + } + return w.body.Write(body) +} +func (w *ingressResponse) statusCode() int { + if w.status == 0 { + return http.StatusOK + } + return w.status +} diff --git a/go/internal/api/home_assistant_test.go b/go/internal/api/home_assistant_test.go new file mode 100644 index 000000000..9e38641f3 --- /dev/null +++ b/go/internal/api/home_assistant_test.go @@ -0,0 +1,185 @@ +package api + +import ( + "github.com/srcfl/ftw/go/internal/apiauth" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +const testIngressPrefix = "/api/hassio_ingress/valid_token" + +func ingressRequest(method, path string) *http.Request { + r := httptest.NewRequest(method, "https://ha.example"+path, nil) + r.RemoteAddr = "172.30.32.2:1234" + r.Header.Set("X-Ingress-Path", testIngressPrefix) + r.Header.Set("X-Remote-User-Id", "owner-id") + r.Header.Set("Origin", "https://ha.example") + r.Header.Set("X-Forwarded-For", "198.51.100.1") + return r +} + +func TestIngressOnlyTrustsSupervisor(t *testing.T) { + next := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if !HomeAssistantIngress(r) { + t.Fatal("missing trusted context") + } + caller, _ := apiauth.FromRequest(r) + if caller.Role != apiauth.RoleOwner || caller.Subject != "ha:owner-id" { + t.Fatalf("caller: %+v", caller) + } + if hasForwardingHeader(r.Header) { + t.Fatal("forwarding metadata reached local gates") + } + w.WriteHeader(204) + }) + policy := MutationPolicy{RequireTokenForRemote: true, LANAuthEnabled: func() bool { return true }, VerifyLANSecret: func(string) bool { return false }} + handler := WithHomeAssistantIngress(WithSecurityHeaders(Authenticate(next, policy))) + for _, tc := range []struct { + name string + change func(*http.Request) + status int + }{ + {"HA owner with an existing password", func(r *http.Request) {}, 204}, + {"LAN spoofing all headers", func(r *http.Request) { r.RemoteAddr = "192.168.1.50:1234" }, 403}, + {"loopback cannot impersonate HA", func(r *http.Request) { r.RemoteAddr = "127.0.0.1:1234" }, 403}, + {"missing user", func(r *http.Request) { r.Header.Del("X-Remote-User-Id") }, 403}, + {"missing prefix", func(r *http.Request) { r.Header.Del("X-Ingress-Path") }, 403}, + {"invalid prefix", func(r *http.Request) { r.Header.Set("X-Ingress-Path", `/api/hassio_ingress/evil">SetupHome` + + "fetch('/api/config'); import('/components/index.js'); window.location.href = '/'; path.split('/'); url + '/draft'; '//external';" + result := string(ingressAsset([]byte(asset), testIngressPrefix, true)) + for _, path := range []string{"/app.js", "/setup", "/api/config", "/components/index.js", "/home-assistant.js"} { + if !strings.Contains(result, testIngressPrefix+path) { + t.Errorf("missing prefix for %s: %s", path, result) + } + } + for _, unchanged := range []string{"path.split('/')", "url + '/draft'", "'//external'"} { + if !strings.Contains(result, unchanged) { + t.Errorf("changed non-root URL: %s", unchanged) + } + } + original := `{"name":"/api/config","location":"/setup"}` + handler := WithHomeAssistantIngress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(original)) + })) + w := httptest.NewRecorder() + handler.ServeHTTP(w, ingressRequest("GET", "/api/config")) + if w.Body.String() != original { + t.Fatal("API data rewritten") + } +} + +func TestIngressRedirectStaysUnderHA(t *testing.T) { + handler := WithHomeAssistantIngress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/setup", 302) })) + w := httptest.NewRecorder() + handler.ServeHTTP(w, ingressRequest("GET", "/")) + if w.Header().Get("Location") != testIngressPrefix+"/setup" { + t.Fatal(w.Header()) + } +} + +func TestIngressKeepsAnExistingStoredPassword(t *testing.T) { + resetLANGuesses(t) + server := newLANAuthServer(t) + enableStoredLANAuth(t, server) + before := server.deps.Cfg.LANPasswordHash + if before == "" { + t.Fatal("test needs a real stored password") + } + handler := WithHomeAssistantIngress(server.Handler()) + for _, path := range []string{"/api/config", "/api/auth/status"} { + w := httptest.NewRecorder() + handler.ServeHTTP(w, ingressRequest("GET", path)) + if w.Code != 200 { + t.Fatalf("%s: %d %s", path, w.Code, w.Body.String()) + } + } + if server.deps.Cfg.LANPasswordHash != before { + t.Fatal("ingress changed the saved password") + } +} + +func TestIngressVendorCallbackKeepsTheExternalHAURL(t *testing.T) { + handler := WithHomeAssistantIngress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + callback := requestOrigin(r) + myUplinkCallbackPath + if callback != "https://ha.example"+testIngressPrefix+myUplinkCallbackPath { + t.Fatal("wrong callback", callback) + } + if !validMyUplinkCallback(callback) { + t.Fatal("HA callback refused") + } + })) + req := ingressRequest("GET", "/api/oauth/myuplink/start") + req.Header.Set("X-Forwarded-Proto", "https") + handler.ServeHTTP(httptest.NewRecorder(), req) + for _, bad := range []string{"/setup", "?code=wrong", "#fragment"} { + if validMyUplinkCallback("https://ha.example" + testIngressPrefix + myUplinkCallbackPath + bad) { + t.Fatal("invalid callback accepted") + } + } +} + +func TestIngressAllowsTheNormalVendorOAuthReturn(t *testing.T) { + // An OAuth browser navigation arrives from the vendor's origin. The nonce + // protects the callback; the existing mutation policy exempts this GET. + req := ingressRequest("GET", myUplinkCallbackPath+"?code=example&state=example") + req.Header.Del("Origin") + req.Header.Set("Sec-Fetch-Site", "cross-site") + if requiresMutationProtection(req) { + t.Fatal("callback no longer uses its own state proof") + } + handler := WithHomeAssistantIngress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(204) })) + w := httptest.NewRecorder() + handler.ServeHTTP(w, req) + if w.Code != 204 { + t.Fatalf("vendor return blocked: %d", w.Code) + } +} diff --git a/go/internal/api/lan_auth.go b/go/internal/api/lan_auth.go index 4b8cd579d..18cb830df 100644 --- a/go/internal/api/lan_auth.go +++ b/go/internal/api/lan_auth.go @@ -291,7 +291,7 @@ func parseLANPasswordHash(encoded string) (salt, key []byte, timeCost, memory ui return salt, key, uint32(t), uint32(m), uint8(p), uint32(len(key)), true } -func (s *Server) handleAuthStatus(w http.ResponseWriter, _ *http.Request) { +func (s *Server) handleAuthStatus(w http.ResponseWriter, r *http.Request) { lanAuth := false if s.deps.Cfg != nil && s.deps.CfgMu != nil { s.deps.CfgMu.RLock() @@ -299,8 +299,9 @@ func (s *Server) handleAuthStatus(w http.ResponseWriter, _ *http.Request) { s.deps.CfgMu.RUnlock() } writeJSON(w, http.StatusOK, map[string]any{ - "lan_auth": lanAuth, - "configured": lanPasswordConfigured(s.deps.State), + "lan_auth": lanAuth, + "home_assistant": HomeAssistantIngress(r), + "configured": lanPasswordConfigured(s.deps.State), }) } diff --git a/go/internal/api/security.go b/go/internal/api/security.go index f4f38fa98..0c45e0d2a 100644 --- a/go/internal/api/security.go +++ b/go/internal/api/security.go @@ -34,6 +34,10 @@ func WithSecurityHeaders(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Content-Security-Policy", "frame-ancestors 'none'") w.Header().Set("X-Frame-Options", "DENY") + if HomeAssistantIngress(r) { + w.Header().Set("Content-Security-Policy", "frame-ancestors 'self'") + w.Header().Set("X-Frame-Options", "SAMEORIGIN") + } w.Header().Set("X-Content-Type-Options", "nosniff") w.Header().Set("Referrer-Policy", "no-referrer") next.ServeHTTP(w, r) diff --git a/go/internal/appenroll/agents_test.go b/go/internal/appenroll/agents_test.go index aab2a835f..e11762e16 100644 --- a/go/internal/appenroll/agents_test.go +++ b/go/internal/appenroll/agents_test.go @@ -259,7 +259,7 @@ func TestAnAgentComesOnlyThroughTheTunnel(t *testing.T) { if _, _, err := b.id.Check(key, nil); !errors.Is(err, ErrNoPairing) { t.Errorf("the relay admitted an agent: %v", err) } - if _, err := b.id.AuthoriseWatch(key, nil); !errors.Is(err, ErrNoPairing) { + if _, _, err := b.id.CheckWatch(key); !errors.Is(err, ErrNoPairing) { t.Errorf("the watch door admitted an agent: %v", err) } grant, commit, err := b.id.CheckTunnel(key, nil) diff --git a/go/internal/appenroll/enroll.go b/go/internal/appenroll/enroll.go index 109027710..8f9fcfae9 100644 --- a/go/internal/appenroll/enroll.go +++ b/go/internal/appenroll/enroll.go @@ -215,10 +215,7 @@ type Identity struct { prepareInviteHome func() (string, error) onInviteClaimed func(protocol.AgentInvite, protocol.AgentRow) pairing *pairingCode - // watchPairing is the watch code, in a slot of its own so that adding a - // watch never cancels a phone's pending code. See MintWatchCode. - watchPairing *pairingCode - now func() time.Time + now func() time.Time // randRead is the entropy source. Injectable so a test can prove the // failure path instead of only the happy one. randRead func([]byte) (int, error) @@ -298,9 +295,6 @@ type pairingCode struct { kind string // attempts counts wrong guesses against this code. See Authorise. attempts int - // parent is the full key of the phone a watch code was minted for, and - // empty on a phone's code. - parent string } // Kinds of pairing code. @@ -654,7 +648,7 @@ func knownRole(role string) error { } // phoneRole refuses a role a phone cannot hold: one the registry does not -// define, watch, which only MintWatchCode hands out, and an agent's, which +// define, watch, which only an owner's phone hands out by key (AddWatch), and an agent's, which // only an owner's signed grant hands out. func phoneRole(role string) error { if err := knownRole(role); err != nil { @@ -1143,10 +1137,6 @@ func (i *Identity) Revoke(id string, from Presence) ([]Removed, error) { for _, key := range gone { delete(i.authorised, key) } - // A watch code minted by this phone dies with it, as its watches do. - if i.watchPairing != nil && i.watchPairing.parent == fullKey { - i.watchPairing = nil - } i.mu.Unlock() err := i.save() diff --git a/go/internal/appenroll/watch.go b/go/internal/appenroll/watch.go index 2aab4e80c..45e24c488 100644 --- a/go/internal/appenroll/watch.go +++ b/go/internal/appenroll/watch.go @@ -1,105 +1,103 @@ package appenroll import ( - "crypto/subtle" "encoding/base64" - "fmt" - "time" + "errors" "github.com/srcfl/ftw/go/internal/apiauth" ) -// WatchTTL is how long a watch code stays usable. The phone hands the code to -// its watch at once, so the life of an owner's code is ample. -const WatchTTL = PairingTTL - -// MintWatchCode issues a single-use code that lets one watch in, bound to the -// owner phone that asked for it (ADR 0009, decision 4). -// -// The code has its own slot. Minting it replaces a pending watch code and -// leaves a phone's pending code alone, so adding a watch never cancels an -// invite somebody is about to scan. +// CheckWatch decides whether a finished exchange handshake may become a +// watch's session, and changes nothing until commit runs. A watch the owner's +// phone added by its key (AddWatch) is admitted; anything else, a phone's key +// among them, answers as a stranger: phones have their stream. There is no +// code: a watch reaches the box through FTW Cloud, which admits only what +// the roster lists. // -// The phone must be an owner now. The API checks that too, and only over a -// session with a fresh passkey step-up; the check here keeps the rule a -// property of the box rather than of one route. -func (i *Identity) MintWatchCode(parentID string, ttl time.Duration) ([]byte, time.Time, error) { - code := make([]byte, PairingCodeBytes) - if _, err := i.randRead(code); err != nil { - return nil, time.Time{}, fmt.Errorf("appenroll: minting a watch code: %w", err) - } - expires := i.now().Add(ttl) - - i.mu.Lock() - defer i.mu.Unlock() - parentKey, parent := i.findLocked(parentID) - if parent == nil { - return nil, time.Time{}, ErrUnknownDevice - } - if parent.role != apiauth.RoleOwner { - return nil, time.Time{}, ErrWatchNeedsOwner - } - i.watchPairing = &pairingCode{ - code: code, expiresAt: expires, role: apiauth.RoleWatch, kind: PairingKindQR, parent: parentKey, - } - return append([]byte(nil), code...), expires, nil -} - -// AuthoriseWatch decides whether a finished exchange handshake may become a -// watch's session. It is Authorise for the watch's door. -// -// A watch already on the list needs no code. Anything else needs the live -// watch code, and spending it stores the watch's key with role watch and the -// phone that minted the code as its parent. A phone's key answers as a -// stranger: phones have their stream. -// -// The code is spent, timed and counted as a phone's code is. The parent must -// still be an owner when the code is spent; a phone demoted or removed since -// it asked burns the code instead. -func (i *Identity) AuthoriseWatch(watchStatic, code []byte) (Grant, error) { +// The exchange commits once the session has proved itself with its first +// transport message, so a message 1 replayed by whoever carries it stamps +// nothing and writes nothing. commit returns the grant as it stands then, and +// fails if the watch or its phone was removed meanwhile. +func (i *Identity) CheckWatch(watchStatic []byte) (Grant, func() (Grant, error), error) { key := base64.RawURLEncoding.EncodeToString(watchStatic) - i.mu.Lock() - if meta := i.authorised[key]; meta != nil { - if meta.role != apiauth.RoleWatch || i.authorised[meta.parent] == nil { + meta := i.authorised[key] + if meta == nil || meta.role != apiauth.RoleWatch || i.authorised[meta.parent] == nil { + i.mu.Unlock() + return Grant{}, nil, ErrNoPairing + } + grant := i.grantLocked(key, meta) + i.mu.Unlock() + return grant, func() (Grant, error) { + i.mu.Lock() + meta := i.authorised[key] + if meta == nil || meta.role != apiauth.RoleWatch || i.authorised[meta.parent] == nil { i.mu.Unlock() return Grant{}, ErrNoPairing } meta.lastSeenMs = i.now().UnixMilli() grant := i.grantLocked(key, meta) i.mu.Unlock() + // A stamp that could not be persisted must not block a session. _ = i.save() return grant, nil + }, nil +} + +// HasWatch reports whether a device id names a watch whose phone is still +// paired. It costs a map walk and no cryptography, so the exchange asks it +// before a handshake's Diffie-Hellman. +func (i *Identity) HasWatch(deviceID string) bool { + i.mu.Lock() + defer i.mu.Unlock() + _, meta := i.findLocked(deviceID) + return meta != nil && meta.role == apiauth.RoleWatch && i.authorised[meta.parent] != nil +} + +// WatchKeyBytes is the length of a watch's Noise static key. +const WatchKeyBytes = 32 + +// ErrWatchKey is a watch key that is not one: the wrong length, or a key the +// box already knows as a phone or an agent. +var ErrWatchKey = errors.New("appenroll: that is not a watch key this box can add") + +// AddWatch stores a watch's key with role watch and the owner phone named by +// parentID as its parent, with no code (ADR 0009, decision 4, as FTW Cloud +// carries it). The phone vouches for the key over its own session, with a +// passkey step-up the API checks; the watch proves it holds the key in every +// handshake after. No code crosses FTW Cloud, and the next roster lists the +// watch, so its first request is admitted like any other. +// +// Adding the same watch again changes nothing. The parent must be an owner +// now, as for a watch code. +func (i *Identity) AddWatch(parentID string, watchStatic []byte) (Grant, error) { + if len(watchStatic) != WatchKeyBytes { + return Grant{}, ErrWatchKey } - if len(code) == 0 { + key := base64.RawURLEncoding.EncodeToString(watchStatic) + + i.mu.Lock() + parentKey, parent := i.findLocked(parentID) + switch { + case parent == nil: i.mu.Unlock() - return Grant{}, ErrNoPairing - } - pairing := i.watchPairing - if pairing == nil || !i.now().Before(pairing.expiresAt) { + return Grant{}, ErrUnknownDevice + case parent.role != apiauth.RoleOwner: i.mu.Unlock() - return Grant{}, ErrBadPairing + return Grant{}, ErrWatchNeedsOwner } - if subtle.ConstantTimeCompare(code, pairing.code) != 1 { - pairing.attempts++ - if pairing.attempts >= MaxPairingAttempts { - i.watchPairing = nil + if meta := i.authorised[key]; meta != nil { + if meta.role != apiauth.RoleWatch || meta.parent != parentKey { + i.mu.Unlock() + return Grant{}, ErrWatchKey } + grant := i.grantLocked(key, meta) i.mu.Unlock() - return Grant{}, ErrBadPairing - } - if parent := i.authorised[pairing.parent]; parent == nil || parent.role != apiauth.RoleOwner { - i.watchPairing = nil - i.mu.Unlock() - return Grant{}, ErrBadPairing + return grant, nil } - now := i.now().UnixMilli() - meta := &appMeta{ - addedAtMs: now, lastSeenMs: now, role: apiauth.RoleWatch, epoch: firstEpoch, parent: pairing.parent, - } + meta := &appMeta{addedAtMs: now, role: apiauth.RoleWatch, epoch: firstEpoch, parent: parentKey} i.authorised[key] = meta - i.watchPairing = nil grant := i.grantLocked(key, meta) i.mu.Unlock() diff --git a/go/internal/appenroll/watch_test.go b/go/internal/appenroll/watch_test.go index 60b169f28..b72ef4067 100644 --- a/go/internal/appenroll/watch_test.go +++ b/go/internal/appenroll/watch_test.go @@ -14,22 +14,27 @@ import ( "github.com/srcfl/ftw/go/internal/apiauth" ) -// addWatch walks one watch through a code its phone minted and returns the -// watch's key and the grant the box stamped on it. +// addWatch adds one watch by its key, as its phone does over a session, and +// returns the watch's key and the grant the box stamped on it. func addWatch(t *testing.T, id *Identity, phone Grant) ([]byte, Grant) { t.Helper() - code, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatalf("MintWatchCode: %v", err) - } key := appKey(t) - grant, err := id.AuthoriseWatch(key, code) + grant, err := id.AddWatch(phone.DeviceID, key) if err != nil { t.Fatalf("adding a watch: %v", err) } return key, grant } +// checkWatch is the exchange's door, decided and committed at once. +func checkWatch(id *Identity, key []byte) (Grant, error) { + _, commit, err := id.CheckWatch(key) + if err != nil { + return Grant{}, err + } + return commit() +} + func TestAnOwnerPhoneAddsAWatchThatNamesItAsParent(t *testing.T) { id, _ := newIdentity(t) phone := pair(t, id, apiauth.RoleOwner) @@ -42,8 +47,8 @@ func TestAnOwnerPhoneAddsAWatchThatNamesItAsParent(t *testing.T) { t.Fatalf("the watch's parent is %q (%q), want %q (owner)", watch.Parent, watch.ParentRole, phone.DeviceID) } - // Its next handshake needs no code, as a phone's does not. - again, err := id.AuthoriseWatch(key, nil) + // Its handshake needs no code: the box knows its key. + again, err := checkWatch(id, key) if err != nil { t.Fatalf("the watch's second handshake was refused: %v", err) } @@ -53,64 +58,6 @@ func TestAnOwnerPhoneAddsAWatchThatNamesItAsParent(t *testing.T) { } // A viewer cannot add a watch, and neither can a phone the box does not know. -func TestOnlyAnOwnerPhoneMintsAWatchCode(t *testing.T) { - id, _ := newIdentity(t) - pair(t, id, apiauth.RoleOwner) - viewer := pair(t, id, apiauth.RoleViewer) - - if _, _, err := id.MintWatchCode(viewer.DeviceID, WatchTTL); !errors.Is(err, ErrWatchNeedsOwner) { - t.Fatalf("a viewer minted a watch code: %v", err) - } - if _, _, err := id.MintWatchCode("nosuchid", WatchTTL); !errors.Is(err, ErrUnknownDevice) { - t.Fatalf("a stranger minted a watch code: %v", err) - } -} - -// The watch code has a slot of its own: adding a watch leaves an invite that -// somebody is about to scan alone, and the invite leaves the watch code alone. -func TestAWatchCodeDoesNotCancelAPhoneCode(t *testing.T) { - id, _ := newIdentity(t) - phone := pair(t, id, apiauth.RoleOwner) - - invite, _, err := id.MintPairingCode(apiauth.RoleViewer, InviteTTL) - if err != nil { - t.Fatal(err) - } - watchCode, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } - if _, err := id.Authorise(appKey(t), invite); err != nil { - t.Fatalf("the invite died when a watch code was minted: %v", err) - } - if _, err := id.AuthoriseWatch(appKey(t), watchCode); err != nil { - t.Fatalf("the watch code died when the invite was spent: %v", err) - } -} - -// A watch code is spent once, and a phone's door never spends it. -func TestAWatchCodeWorksOnceAndOnlyAtTheWatchDoor(t *testing.T) { - id, _ := newIdentity(t) - phone := pair(t, id, apiauth.RoleOwner) - code, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } - - if _, err := id.Authorise(appKey(t), code); !errors.Is(err, ErrBadPairing) { - t.Fatalf("a watch code opened a phone's stream: %v", err) - } - if _, err := id.AuthoriseWatch(appKey(t), code); err != nil { - t.Fatalf("the watch code was spent by the stream: %v", err) - } - if _, err := id.AuthoriseWatch(appKey(t), code); !errors.Is(err, ErrBadPairing) { - t.Fatalf("a watch code let a second watch in: %v", err) - } - if n := id.AuthorisedCount(); n != 2 { - t.Fatalf("%d enrolments, want the phone and one watch", n) - } -} - // Each door admits its own kind and answers the other as a stranger. func TestAWatchHasNoStreamAndAPhoneNoWatchDoor(t *testing.T) { id, _ := newIdentity(t) @@ -128,62 +75,11 @@ func TestAWatchHasNoStreamAndAPhoneNoWatchDoor(t *testing.T) { if _, err := id.Authorise(watchKey, nil); !errors.Is(err, ErrNoPairing) { t.Fatalf("a watch opened a stream: %v", err) } - if _, err := id.AuthoriseWatch(phoneKey, nil); !errors.Is(err, ErrNoPairing) { + if _, err := checkWatch(id, phoneKey); !errors.Is(err, ErrNoPairing) { t.Fatalf("a phone came through the watch's door: %v", err) } } -func TestAnExpiredWatchCodeIsRefused(t *testing.T) { - id, _ := newIdentity(t) - phone := pair(t, id, apiauth.RoleOwner) - now := time.UnixMilli(1_760_000_000_000) - id.now = func() time.Time { return now } - - code, expires, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } - now = expires.Add(time.Millisecond) - if _, err := id.AuthoriseWatch(appKey(t), code); !errors.Is(err, ErrBadPairing) { - t.Fatalf("an expired watch code was redeemed: %v", err) - } -} - -func TestAWatchCodeIsBurnedByGuessing(t *testing.T) { - id, _ := newIdentity(t) - phone := pair(t, id, apiauth.RoleOwner) - code, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } - wrong := make([]byte, PairingCodeBytes) - for i := 0; i < MaxPairingAttempts; i++ { - if _, err := id.AuthoriseWatch(appKey(t), wrong); !errors.Is(err, ErrBadPairing) { - t.Fatalf("guess %d: %v", i, err) - } - } - if _, err := id.AuthoriseWatch(appKey(t), code); !errors.Is(err, ErrBadPairing) { - t.Fatalf("the right code still worked after %d wrong guesses: %v", MaxPairingAttempts, err) - } -} - -// The phone must still be an owner when the watch spends the code. -func TestAWatchCodeDiesWithItsPhonesOwnerRole(t *testing.T) { - id, _ := newIdentity(t) - pair(t, id, apiauth.RoleOwner) - phone := pair(t, id, apiauth.RoleOwner) - code, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } - if err := id.SetRole(phone.DeviceID, apiauth.RoleViewer); err != nil { - t.Fatal(err) - } - if _, err := id.AuthoriseWatch(appKey(t), code); !errors.Is(err, ErrBadPairing) { - t.Fatalf("a watch joined under a phone that is no longer an owner: %v", err) - } -} - // Removing a phone removes its watches, and their keys come back so the // caller can end their sessions. Removing a watch touches nothing else. func TestRemovalRunsFromPhoneToWatchAndNeverBack(t *testing.T) { @@ -206,14 +102,10 @@ func TestRemovalRunsFromPhoneToWatchAndNeverBack(t *testing.T) { if _, ok := id.GrantFor(second.DeviceID); !ok { t.Fatal("removing one watch removed another") } - if _, err := id.AuthoriseWatch(firstKey, nil); !errors.Is(err, ErrNoPairing) { + if _, err := checkWatch(id, firstKey); !errors.Is(err, ErrNoPairing) { t.Fatalf("a removed watch's next message 1 was answered: %v", err) } - pending, _, err := id.MintWatchCode(phone.DeviceID, WatchTTL) - if err != nil { - t.Fatal(err) - } removed, err = id.Revoke(phone.DeviceID, OverASession) if err != nil { t.Fatal(err) @@ -224,7 +116,7 @@ func TestRemovalRunsFromPhoneToWatchAndNeverBack(t *testing.T) { if _, ok := id.GrantFor(second.DeviceID); ok { t.Fatal("a watch outlived its phone") } - if _, err := id.AuthoriseWatch(secondKey, nil); !errors.Is(err, ErrNoPairing) { + if _, err := checkWatch(id, secondKey); !errors.Is(err, ErrNoPairing) { t.Fatalf("a watch whose phone was removed was answered: %v", err) } for _, d := range id.Roster() { @@ -232,9 +124,6 @@ func TestRemovalRunsFromPhoneToWatchAndNeverBack(t *testing.T) { t.Fatal("a removed watch is still in the roster") } } - if _, err := id.AuthoriseWatch(appKey(t), pending); !errors.Is(err, ErrBadPairing) { - t.Fatalf("a watch code outlived the phone that asked for it: %v", err) - } } // A watch is a watch until it is removed; its reach follows its phone. @@ -342,7 +231,7 @@ func TestAWatchAndItsParentSurviveARestart(t *testing.T) { if err != nil { t.Fatal(err) } - grant, err := again.AuthoriseWatch(key, nil) + grant, err := checkWatch(again, key) if err != nil { t.Fatalf("the watch was forgotten across a restart: %v", err) } @@ -387,3 +276,99 @@ func TestAWatchWithoutItsPhoneIsDroppedAtLoad(t *testing.T) { t.Fatalf("%d enrolments after loading an orphan", n) } } + +// A watch added by its key is the same watch a code adds: the owner phone is +// its parent, it opens with no code, a viewer cannot add one, and it goes +// with its phone. +func TestAWatchAddedByItsKey(t *testing.T) { + id, _ := newIdentity(t) + phone := pair(t, id, apiauth.RoleOwner) + viewer := pair(t, id, apiauth.RoleViewer) + key := appKey(t) + + if _, err := id.AddWatch(viewer.DeviceID, key); !errors.Is(err, ErrWatchNeedsOwner) { + t.Fatalf("a viewer added a watch: %v", err) + } + if _, err := id.AddWatch("nobody00", key); !errors.Is(err, ErrUnknownDevice) { + t.Fatalf("an unknown phone added a watch: %v", err) + } + if _, err := id.AddWatch(phone.DeviceID, key[:31]); !errors.Is(err, ErrWatchKey) { + t.Fatalf("a short key was taken: %v", err) + } + + watch, err := id.AddWatch(phone.DeviceID, key) + if err != nil { + t.Fatal(err) + } + if watch.Role != apiauth.RoleWatch || watch.Parent != phone.DeviceID || watch.ParentRole != apiauth.RoleOwner { + t.Fatalf("the box wrote down %+v", watch) + } + opened, err := checkWatch(id, key) + if err != nil || opened.DeviceID != watch.DeviceID { + t.Fatalf("the watch could not open with its key alone: %+v %v", opened, err) + } + if again, err := id.AddWatch(phone.DeviceID, key); err != nil || again.DeviceID != watch.DeviceID { + t.Fatalf("adding it again gave %+v %v", again, err) + } + + // Under another phone, or as a phone's key, it is no watch to add. + other := pair(t, id, apiauth.RoleOwner) + if _, err := id.AddWatch(other.DeviceID, key); !errors.Is(err, ErrWatchKey) { + t.Fatalf("another phone took the watch: %v", err) + } + + if _, err := id.Revoke(phone.DeviceID, OverASession); err != nil { + t.Fatal(err) + } + if _, err := checkWatch(id, key); err == nil { + t.Fatal("the watch outlived its phone") + } +} + +// The exchange's check writes nothing: a message 1 that FTW Cloud replays +// stamps no visit until the session proves itself and commits. +func TestCheckingAWatchWritesNothingUntilItCommits(t *testing.T) { + id, _ := newIdentity(t) + phone := pair(t, id, apiauth.RoleOwner) + key, watch := addWatch(t, id, phone) + seen := func() int64 { + for _, d := range id.Devices() { + if d.ID == watch.DeviceID { + return d.LastSeenMs + } + } + t.Fatal("the watch is not listed") + return 0 + } + + for i := 0; i < 3; i++ { + if _, _, err := id.CheckWatch(key); err != nil { + t.Fatal(err) + } + } + if seen() != 0 { + t.Fatal("a check stamped the watch's visit") + } + _, commit, err := id.CheckWatch(key) + if err != nil { + t.Fatal(err) + } + if _, err := commit(); err != nil || seen() == 0 { + t.Fatalf("a commit did not stamp the visit: %v", err) + } + if !id.HasWatch(watch.DeviceID) || id.HasWatch(phone.DeviceID) || id.HasWatch("nobody00") { + t.Fatal("HasWatch names the wrong devices") + } + + // A watch removed between the check and the commit is not let in. + _, commit, err = id.CheckWatch(key) + if err != nil { + t.Fatal(err) + } + if _, err := id.Revoke(watch.DeviceID, OverASession); err != nil { + t.Fatal(err) + } + if _, err := commit(); !errors.Is(err, ErrNoPairing) { + t.Fatalf("a removed watch committed: %v", err) + } +} diff --git a/go/internal/appproto/cloud_tile_test.go b/go/internal/appproto/cloud_tile_test.go new file mode 100644 index 000000000..a86127d70 --- /dev/null +++ b/go/internal/appproto/cloud_tile_test.go @@ -0,0 +1,43 @@ +package appproto + +import ( + "bytes" + "context" + "testing" + + "github.com/srcfl/ftw/go/internal/cloudlink" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// ledgerSite is a history provider as the cloud link reads it. +type ledgerSite struct{ HistoryProvider } + +func (s ledgerSite) HistoryWindow(ctx context.Context, names []string, stepMs, fromMs, toMs int64) (map[string][]HistorySample, error) { + return s.Window(ctx, names, stepMs, fromMs, toMs) +} + +// A closed tile FTW Cloud gets is the tile the box serves over Noise for the +// same series: the same id, etag and bytes, holes included, read through the +// two real paths. +func TestACloudTileIsTheNoiseTile(t *testing.T) { + start := tileStartFor(ResN5m, 1_760_000_000_000-24*3_600_000) + span := ResolutionTileSpanMs[ResN5m] + ledger := &fakeLedger{ + values: map[string]float64{"grid_w": 1200.4, "pv_w": -800, "battery_w": 300, "load_w": 1700.6}, + holes: map[int64]bool{start + 5*300_000: true, start + 90*300_000: true}, + } + h := &Handler{cfg: Config{History: ledger}} + plan := planHistoryQuery(ResN5m, start, start+span, defaultMaxPoints) + noise, err := h.buildTile(context.Background(), cloudwire.TileSeries, plan, plan.tiles[0], start+2*span) + if err != nil { + t.Fatal(err) + } + cloud, err := cloudlink.PackTile(context.Background(), ledgerSite{ledger}, start, start, start+span) + if err != nil { + t.Fatal(err) + } + if noise.TileID != cloud.TileID || noise.Etag != cloud.Etag || !bytes.Equal(noise.Data, cloud.Data) || + noise.Partial || cloud.Partial || noise.StepMs != cloud.StepMs { + t.Fatalf("Noise %s %s, FTW Cloud %s %s", noise.TileID, noise.Etag, cloud.TileID, cloud.Etag) + } +} diff --git a/go/internal/appproto/exchange.go b/go/internal/appproto/exchange.go index b4153f962..ed43839e7 100644 --- a/go/internal/appproto/exchange.go +++ b/go/internal/appproto/exchange.go @@ -4,6 +4,7 @@ import ( "context" "encoding/hex" "github.com/srcfl/ftw/go/protocol" + "slices" "sort" "github.com/srcfl/ftw/go/internal/units" @@ -12,17 +13,27 @@ import ( // A watch's exchange session (ADR 0009, decision 2). // // A watch speaks HTTPS and nothing lower, so each of its requests is one Noise -// message through the relay and each answer one HTTP body. The session takes -// hello, cmd and now. It never streams: there is no socket to stream on. Every -// answer is two lane 0 frames of one size whatever the op or the outcome, so -// the relay cannot tell a read from a command or a refusal from a success by -// length. appuplink pads an answer to two frames with Pad. +// message that FTW Cloud carries and each answer one HTTP body. The session +// takes hello, cmd and now. It never streams: there is no socket to stream on. +// Every answer is two lane 0 frames of one size whatever the op or the +// outcome, so the service cannot tell a read from a command or a refusal from +// a success by length. appuplink pads an answer to two frames with Pad. + +// ExchangeOps are the commands a watch's exchange takes: the charger's hold, +// which Charge now, Pause and Back to the plan send. A watch reads and +// charges and does nothing else (ADR 0009, decision B); its role's +// ftw.dispatch.write reaches further, and the exchange does not. +var ExchangeOps = []string{OpLoadpointHold} func (h *Handler) dispatchExchange(ctx context.Context, env Envelope) error { switch env.T { case MsgHello: return h.onExchangeHello(env) case MsgCmd: + var cmd Cmd + if err := Unmarshal(env.B, &cmd); err == nil && !slices.Contains(ExchangeOps, cmd.Op) { + return h.sendError(env.ID, ErrorBody{Code: protocol.ErrOpNotGranted, Args: map[string]any{"op": cmd.Op}}) + } return h.onCmd(ctx, env, "") case MsgNow: return h.onNow() @@ -92,8 +103,8 @@ func (h *Handler) onExchangeHello(env Envelope) error { } if bucket := bulkBucketFor(size); bucket == 0 || bucket > ExchangeHelloMaxBucket { // A house with too many sources to name in 1 KiB says less. The - // watch then reads now on the exchange, as with FTW Cloud off. - h.log.Warn("exchange hello_ok left FTW Cloud out to fit the relay's answer", "bytes", size) + // watch then keeps the names its phone gave it. + h.log.Warn("exchange hello_ok left FTW Cloud out to fit the exchange's answer", "bytes", size) body.Cloud = nil } } @@ -101,9 +112,8 @@ func (h *Handler) onExchangeHello(env Envelope) error { } // ExchangeHelloMaxBucket is the largest bucket an exchange hello_ok may take: -// the smallest one. The relay answers a watch with at most 4 KiB behind its -// id, and handshake message 2 adds 48 bytes to the frame, so the next bucket, -// 4 KiB, never fits. +// the smallest one. FTW Cloud carries an exchange message of at most 2 KiB +// (cloudwire.XDataMax), so the next bucket, 4 KiB, never fits. const ExchangeHelloMaxBucket = 1024 func encodedSize(t string, body any) (int, error) { diff --git a/go/internal/appproto/exchange_test.go b/go/internal/appproto/exchange_test.go index e483856cc..7c738915e 100644 --- a/go/internal/appproto/exchange_test.go +++ b/go/internal/appproto/exchange_test.go @@ -45,10 +45,10 @@ func TestAnExchangeHelloNamesTheRevAndNoCatalogue(t *testing.T) { } } -// The relay answers a watch with at most 4 KiB, so hello_ok never takes the -// 4 KiB bucket. A house too big for 1 KiB leaves FTW Cloud out, and the watch -// reads now instead. -func TestAnExchangeHelloTooBigForTheRelayLeavesCloudOut(t *testing.T) { +// FTW Cloud carries an exchange message of at most 2 KiB, so hello_ok never +// takes the 4 KiB bucket. A house too big for 1 KiB leaves FTW Cloud out, and +// the watch keeps the names its phone gave it. +func TestAnExchangeHelloTooBigForItsAnswerLeavesCloudOut(t *testing.T) { h, _, _, rec := newExchangeRig(t) sources := map[string]string{} for i := 0; i < 80; i++ { @@ -80,6 +80,22 @@ func TestAnExchangeCommandPushesNoPlan(t *testing.T) { } } +// A watch charges and does nothing else: every other op its role's scope +// would reach is refused before it runs, and the charger hears nothing. +func TestAnExchangeTakesTheChargersHoldAlone(t *testing.T) { + for _, op := range []string{OpLoadpointBoost, OpLoadpointSoCSet, OpLoadpointSurplusOnlySet, OpCarGoalSet, OpCarSoCSet, OpSessionCarSet, OpSetMode} { + h, lp, _, rec := newExchangeRig(t) + deliver(t, h, MsgCmd, nil, cmdLoadpoint(op, "x-"+op, holdArgs(), 7, 200_000)) + e := body[ErrorBody](t, rec.only(t, MsgError)) + if e.Code != protocol.ErrOpNotGranted { + t.Fatalf("%s through the exchange answered %q", op, e.Code) + } + if lp.holdCalls+lp.boostCalls+lp.socCalls+lp.surplusCalls != 0 { + t.Fatalf("%s through the exchange reached the charger", op) + } + } +} + // now is one lane 0 frame. What does not fit is left out and flagged, never // sent in a bigger frame. func TestNowFitsLaneZeroOrSaysWhatItLeftOut(t *testing.T) { diff --git a/go/internal/appuplink/client.go b/go/internal/appuplink/client.go index a48216432..922a6e185 100644 --- a/go/internal/appuplink/client.go +++ b/go/internal/appuplink/client.go @@ -130,8 +130,8 @@ type Options struct { Handler HandlerBuilder // ExchangeHandler builds one handler per watch session, with - // appproto.Config.Exchange set (ADR 0009). Nil leaves the exchange - // socket closed: the box takes no watch, and nothing else changes. + // appproto.Config.Exchange set (ADR 0009). Nil opens no exchange: the + // box takes no x from FTW Cloud, and nothing else changes. ExchangeHandler HandlerBuilder Logger *slog.Logger @@ -190,10 +190,10 @@ type Uplink struct { liveMu sync.Mutex live []*liveLink - // exchanges is the box's table of watch sessions, nil without an - // ExchangeHandler. One for the box rather than one per connection, so a + // exchange is the watches' door through FTW Cloud and their table of + // sessions, nil without an ExchangeHandler. One for the box, so a // watch's session survives the hour turning under it. - exchanges *exchanges + exchange *Exchange // slots holds each device's one session, across the relay's // connections and the tunnel through FTW Cloud. @@ -231,8 +231,8 @@ func (u *Uplink) DropSessionsByAppKey(pub []byte) int { for _, l := range live { dropped += l.sessions.dropByAppKey(pub) } - if u.exchanges != nil { - dropped += u.exchanges.dropByAppKey(pub) + if u.exchange != nil { + dropped += u.exchange.table.dropByAppKey(pub) } dropped += u.tunnel.dropByAppKey(pub) return dropped @@ -296,14 +296,14 @@ func New(opts Options) (*Uplink, error) { u.slots = &slots{} u.tunnel = newTunnel(opts.Handler, opts.Enroll, u.slots, opts.Arrivals, u.log) if opts.ExchangeHandler != nil { - u.exchanges = &exchanges{ + u.exchange = newExchange(&exchanges{ build: opts.ExchangeHandler, enroll: opts.Enroll, static: opts.Enroll.StaticKey(), log: u.log, now: time.Now, cmdFloor: ExchangeCmdFloor, - } + }, u.log) } return u, nil } @@ -321,9 +321,12 @@ func productionDialer() *websocket.Dialer { } } -// Run serves phone sessions carried by FTW Cloud until the context ends. -// It opens no connection to the retired relay. +// Run serves phone sessions and watch exchanges carried by FTW Cloud until +// the context ends. It opens no connection to the retired relay. func (u *Uplink) Run(ctx context.Context) error { + if u.exchange != nil { + go u.exchange.Run(ctx) + } u.tunnel.Run(ctx) return ctx.Err() } @@ -414,11 +417,6 @@ func (u *Uplink) start(ctx context.Context, conn *websocket.Conn, epoch int64, h l := &link{epoch: epoch, stop: stop, done: make(chan linkEnd, 1)} go func() { defer stop() - // The exchange socket rides beside this connection, under the same - // epoch and handle, and ends with it. - if u.exchanges != nil { - go u.keepExchange(ctx, epoch, handle) - } code, reason, err := u.serve(ctx, conn) l.done <- linkEnd{code: code, reason: reason, err: err} }() diff --git a/go/internal/appuplink/cloudtunnel_test.go b/go/internal/appuplink/cloudtunnel_test.go index 3f9cb4f32..54087d702 100644 --- a/go/internal/appuplink/cloudtunnel_test.go +++ b/go/internal/appuplink/cloudtunnel_test.go @@ -13,6 +13,7 @@ import ( "net/http" "net/http/httptest" "path/filepath" + "slices" "sync" "sync/atomic" "testing" @@ -42,8 +43,10 @@ type miniCloud struct { conn *websocket.Conn home string roster *cloudwire.Roster + caps []string got []string toApp chan cloudwire.Tun + toWatch chan cloudwire.X rostered chan struct{} wmu sync.Mutex } @@ -51,7 +54,8 @@ type miniCloud struct { func newMiniCloud(t *testing.T) *miniCloud { t.Helper() key, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) - c := &miniCloud{t: t, key: key, toApp: make(chan cloudwire.Tun, 64), rostered: make(chan struct{}, 8)} + c := &miniCloud{t: t, key: key, toApp: make(chan cloudwire.Tun, 64), toWatch: make(chan cloudwire.X, 64), + rostered: make(chan struct{}, 8)} c.srv = httptest.NewUnstartedServer(http.HandlerFunc(c.serve)) c.srv.TLS = &tls.Config{MinVersion: tls.VersionTLS13} c.srv.StartTLS() @@ -113,7 +117,7 @@ func (c *miniCloud) serve(w http.ResponseWriter, r *http.Request) { return } c.mu.Lock() - c.conn, c.home = conn, home + c.conn, c.home, c.caps = conn, home, hello.Caps c.mu.Unlock() for { _, msg, err := conn.ReadMessage() @@ -139,6 +143,11 @@ func (c *miniCloud) serve(w http.ResponseWriter, r *http.Request) { if cloudwire.Unmarshal(env.B, &t) == nil { c.toApp <- t } + case cloudwire.MsgX: + var x cloudwire.X + if cloudwire.Unmarshal(env.B, &x) == nil { + c.toWatch <- x + } case cloudwire.MsgRoster: var ro cloudwire.Roster if cloudwire.Verify(pub, home, env) == nil && cloudwire.Unmarshal(env.B, &ro) == nil { @@ -169,6 +178,35 @@ func (c *miniCloud) fromApp(key []byte, sid uint32, d []byte) bool { return false } +// fromWatch is a watch's x, as /v1/app/once hands it on: the service finds +// the roster entry by the key its own handshake proved, needs tun on it, and +// names the device from it. Only a box that said x gets one. +func (c *miniCloud) fromWatch(key []byte, id uint32, d []byte) bool { + c.mu.Lock() + conn, home, ro, caps := c.conn, c.home, c.roster, c.caps + c.mu.Unlock() + if conn == nil || ro == nil || !slices.Contains(caps, cloudwire.CapX) { + return false + } + for _, dev := range ro.Devices { + if dev.Tun && bytes.Equal(dev.Key, key) { + return c.write(conn, nil, home, cloudwire.MsgX, cloudwire.X{Dev: dev.ID, ID: id, D: d}) == nil + } + } + return false +} + +func (c *miniCloud) toWatchNext(t *testing.T) cloudwire.X { + t.Helper() + select { + case m := <-c.toWatch: + return m + case <-time.After(5 * time.Second): + t.Fatal("no x from the box") + return cloudwire.X{} + } +} + // forward sends a tun to the box as it is, with no roster check: a service // that forwards what it should not. func (c *miniCloud) forward(dev string, sid uint32, d []byte) { @@ -412,3 +450,124 @@ wait: t.Fatalf("hello_ok.cloud %+v", ok.Cloud) } } + +// The watch's half of the one door, end to end: a watch its phone added by +// its key charges through FTW Cloud with the readable copy off. The box says +// x, the roster lists the watch with tun, and each exchange message rides one +// x each way through the real cloud link. +func TestAWatchCommandThroughFTWCloud(t *testing.T) { + enroll, err := appenroll.LoadOrCreate(filepath.Join(t.TempDir(), "nova.key")) + if err != nil { + t.Fatal(err) + } + charger := &fakeCharger{} + cmds := appproto.NewCmdLog() + uplink, err := New(Options{ + Enroll: enroll, Handler: handlerWith(false, charger, cmds), ExchangeHandler: handlerWith(true, charger, cmds), + Logger: slog.New(slog.DiscardHandler), + }) + if err != nil { + t.Fatal(err) + } + cloud := newMiniCloud(t) + link, err := cloudlink.New(cloudlink.Options{ + Site: quietSite{}, Roster: CloudRoster{Enroll: enroll, Tunnel: true}, KV: &memState{m: map[string]string{}}, + KeyPath: filepath.Join(t.TempDir(), "cloud.key"), Build: "test", + Tunnel: uplink.Tunnel(), Exchange: uplink.Exchange(), + Endpoint: cloud.endpoint(), ServiceKeys: []*ecdsa.PublicKey{&cloud.key.PublicKey}, RootCAs: cloud.roots(), + Tick: 20 * time.Millisecond, Logger: slog.New(slog.DiscardHandler), + }) + if err != nil { + t.Fatal(err) + } + uplink.Tunnel().Attach(link) + uplink.Exchange().Attach(link) + enroll.OnChange(link.RosterChanged) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + go func() { _ = uplink.Run(ctx) }() + go func() { _ = link.Run(ctx) }() + + code, _, err := enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) + if err != nil { + t.Fatal(err) + } + app, err := newAppClient() + if err != nil { + t.Fatal(err) + } + phone, err := enroll.Authorise(app.staticPublic(), code) + if err != nil { + t.Fatal(err) + } + w, err := newWatchClient() + if err != nil { + t.Fatal(err) + } + if _, err := enroll.AddWatch(phone.DeviceID, w.static.Public()); err != nil { + t.Fatal(err) + } + link.RosterChanged() + deadline := time.After(10 * time.Second) + for { + select { + case <-cloud.rostered: + case <-deadline: + t.Fatal("the box never sent a roster with the watch") + } + cloud.mu.Lock() + listed := false + if cloud.roster != nil { + for _, d := range cloud.roster.Devices { + listed = listed || d.ID == w.dev() && d.Tun && d.Role == apiauth.RoleWatch + } + } + cloud.mu.Unlock() + if listed { + break + } + } + + message1, err := w.open(enroll.StaticKey().Public(), nil) + if err != nil { + t.Fatal(err) + } + if !cloud.fromWatch(w.static.Public(), 1, message1) { + t.Fatal("the service would not carry the watch's handshake; the box must say x") + } + reply := cloud.toWatchNext(t) + if reply.Dev != w.dev() || reply.ID != 1 { + t.Fatalf("the handshake's answer is %+v", reply) + } + if _, _, err := w.opened(reply.D); err != nil { + t.Fatal(err) + } + request, err := w.request(appproto.MsgCmd, chargeNow("through-cloud")) + if err != nil { + t.Fatal(err) + } + if !cloud.fromWatch(w.static.Public(), 2, request) { + t.Fatal("the service would not carry the command") + } + reply = cloud.toWatchNext(t) + frames, err := w.answer(reply.D) + if err != nil { + t.Fatal(err) + } + if kinds := types(frames); reply.ID != 2 || len(kinds) != 2 || kinds[1] != appproto.MsgCmdResult { + t.Fatalf("the answer %d carries %v", reply.ID, kinds) + } + if decodeAs[appproto.CmdResult](t, frames[1]).State != appproto.CmdApplied || charger.count() != 1 { + t.Fatalf("the charge did not apply: %d holds", charger.count()) + } + + // The readable copy stayed off: only the roster, the notice and the + // exchange left the box. + cloud.mu.Lock() + defer cloud.mu.Unlock() + for _, typ := range cloud.got { + if typ != cloudwire.MsgRoster && typ != cloudwire.MsgX && typ != cloudwire.MsgNotice { + t.Fatalf("the box sent %s with the copy off", typ) + } + } +} diff --git a/go/internal/appuplink/exchange.go b/go/internal/appuplink/exchange.go index c2ceb903d..112142651 100644 --- a/go/internal/appuplink/exchange.go +++ b/go/internal/appuplink/exchange.go @@ -10,34 +10,35 @@ import ( "sync" "time" - "github.com/gorilla/websocket" "github.com/srcfl/ftw/go/internal/apiauth" "github.com/srcfl/ftw/go/internal/appenroll" "github.com/srcfl/ftw/go/internal/appproto" "github.com/srcfl/ftw/go/protocol/appwire" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) // The exchange: a watch's one-shot door to the box (ADR 0009, decisions 1 -// and 2; the relay's half is srcfl/ftw-webapp docs/watch-exchange.md). +// and 2; docs/protocol-cloud.md, "Exchange"). // // watchOS gives an app HTTPS and nothing lower, so a watch cannot hold a -// stream. It posts one Noise message to the relay, and the relay hands it to -// the box on a second socket, /r///exchange, as -// id(16) ‖ message. The box answers id ‖ reply on the same socket. The id is -// the relay's own random number; the box copies it back and reads nothing -// into it. +// stream. It posts one Noise message to FTW Cloud inside a one-shot request, +// and the service hands it to the box on the uplink as x {dev, id, d}. The +// box answers x {dev, id, d} with the same dev and id. dev is the device the +// service's own handshake proved, which the box checks against the key its +// handshake proves; id is the service's number, which the box copies back +// and reads nothing into. // -// Exchange sessions are a class of their own: a table of at most eight, apart -// from the four stream places, so a watch never pushes a phone out. Losing -// the exchange socket changes nothing for the uplink, local control or the -// planner. +// Exchange sessions are a class of their own: a table of at most eight, +// apart from the phones' places, so a watch never pushes a phone out. They +// live on the box, not on the uplink, so a session survives the uplink +// reconnecting under it. Losing FTW Cloud changes nothing for local control +// or the planner. const ( - // MaxExchangeSessions is how many watch sessions the box holds at once. - MaxExchangeSessions = 8 - - // ExchangeIDBytes is the relay's id in front of every message. - ExchangeIDBytes = 16 + // MaxExchangeSessions is how many watch sessions the box holds at once, + // and MaxExchangeSessionsPerWatch how many of them one watch may hold. + MaxExchangeSessions = 8 + MaxExchangeSessionsPerWatch = 2 // exchangeFrameBytes is the one size every frame takes but hello_ok: // lane 0's 512 B bucket. @@ -58,8 +59,16 @@ const ( exchangeLifetime = 3 * time.Minute // exchangeOpensPerMinute bounds handshakes, each of which costs the box - // up to four X25519 operations. The rest are dropped unanswered. + // up to four X25519 operations, and exchangeOpensPerWatch one watch's + // share, so one watch's replays never spend another's. The rest are + // dropped unanswered. exchangeOpensPerMinute = 60 + exchangeOpensPerWatch = 12 + + // exchangeRequestsPerMinute bounds what the box takes off its link in + // all, handshakes, requests and junk alike. Past it a request is + // dropped unanswered and runs nothing. + exchangeRequestsPerMinute = 120 // exchangeSettle is how long the box holds an answer for a command's // result. Core settles a charger's result in the pass that takes the @@ -72,16 +81,16 @@ const ( // from when the request arrived (ADR 0009, decision F; Fredrik, 14 // October 2026). A refusal is ready in a millisecond and an applied // charge waits for the EV save, so without a floor the time to answer - // would tell the relay which happened. A command that takes longer, an + // would tell FTW Cloud which happened. A command that takes longer, an // EV save of up to 2 s, answers when it is done and waits no more. ExchangeCmdFloor = 400 * time.Millisecond - // exchangeInFlight bounds requests handled at once. The relay allows two - // open exchanges per room; this is the box's own ceiling. - exchangeInFlight = 8 + // exchangeInFlight bounds requests handled at once, as many as FTW Cloud + // sends a box at once. Past it a request is answered with silence. + exchangeInFlight = cloudwire.XInFlight - // exchangeReadLimit admits the largest request plus its id. - exchangeReadLimit = 1024 + // exchangeInbox is how many requests wait for a slot. + exchangeInbox = 2 * exchangeInFlight ) // exchangePrologue binds an exchange handshake to this box, and keeps it @@ -90,14 +99,25 @@ func exchangePrologue(boxStatic []byte) []byte { return append([]byte("ftw.exchange.v1:"), boxStatic...) } -var errExchangeRate = errors.New("appuplink: too many exchange handshakes this minute") +var ( + errExchangeRate = errors.New("appuplink: too many exchange handshakes this minute") + errExchangeStranger = errors.New("appuplink: FTW Cloud named a device that is no watch of this box") + errExchangeCode = errors.New("appuplink: a watch's hello carried a code; watches are added by key") +) // exchangeSession is one watch's short conversation with the box. type exchangeSession struct { transport *appwire.Transport handler *appproto.Handler - // appStatic is the watch's key as the handshake authenticated it. + // appStatic is the watch's key as the handshake authenticated it, and + // dev the device id FTW Cloud named it by. appStatic []byte + dev string + // proved is set by the first transport message that decrypts, which + // commit then stamps. Until then the session may be a replayed message + // 1: it has written nothing, and it goes first when room is needed. + proved bool + commit func() (appenroll.Grant, error) openedAt time.Time // run lets one request at a time through the handler. A command whose @@ -173,15 +193,14 @@ func (s *exchangeSession) close() { } // exchanges is the box's table of watch sessions. One per box: a watch's -// request may arrive on the old hour's socket or the new one, and finds its -// session either way. +// request may arrive on the uplink before a reconnect or after it, and finds +// its session either way. type exchanges struct { build HandlerBuilder enroll *appenroll.Identity static appwire.KeyPair log *slog.Logger - // now is the real clock. Session ages are durations, and the uplink's - // injectable clock answers which epoch it is. + // now is the clock session ages are measured on. now func() time.Time // settle is exchangeSettle; a test shortens it. settle time.Duration @@ -189,37 +208,46 @@ type exchanges struct { // which only a test that builds the table by hand gets. cmdFloor time.Duration - mu sync.Mutex - live []*exchangeSession - opens []time.Time + mu sync.Mutex + live []*exchangeSession + opens []time.Time + devOpens map[string][]time.Time } -// answer takes one request off the exchange socket and returns the reply, or -// nil for silence. A stranger, a removed key, a spent session, a malformed -// body and a handshake over the rate all get silence. -func (x *exchanges) answer(ctx context.Context, body []byte) []byte { +// answer takes one request for the watch FTW Cloud names dev and returns the +// reply, or nil for silence. A stranger, a removed key, a key that is not +// dev's, a spent session, a malformed body and a handshake over the rate all +// get silence. +func (x *exchanges) answer(ctx context.Context, dev string, body []byte) []byte { arrived := time.Now() x.prune() switch len(body) { case exchangeOpenBytes: - reply, err := x.open(ctx, body) + reply, err := x.open(ctx, dev, body) if err != nil { x.log.Debug("exchange handshake opened no session", "err", err) return nil } return reply case exchangeRequestBytes: - return x.route(ctx, body, arrived) + return x.route(ctx, dev, body, arrived) default: return nil } } +var errExchangeNotDev = errors.New("appuplink: the exchange handshake proved another device than FTW Cloud named") + // open runs the responder half of a handshake for a watch. The watch's hello // rides in message 1 and its hello_ok in message 2, so a session costs one // round trip. -func (x *exchanges) open(ctx context.Context, message1 []byte) ([]byte, error) { - if !x.allowOpen() { +func (x *exchanges) open(ctx context.Context, dev string, message1 []byte) ([]byte, error) { + // A device that is no watch of this box costs nothing: no token, no + // Diffie-Hellman. Its opens are counted apart from every other watch's. + if !x.enroll.HasWatch(dev) { + return nil, errExchangeStranger + } + if !x.allowOpen(dev) { return nil, errExchangeRate } handshake, err := appwire.NewResponder(appwire.HandshakeOptions{ @@ -245,15 +273,26 @@ func (x *exchanges) open(ctx context.Context, message1 []byte) ([]byte, error) { return nil, err } - // Checked before anything is written: answering a stranger would confirm - // that a box is on this handle and hand them a fresh ephemeral key. - grant, err := x.enroll.AuthoriseWatch(handshake.RemoteStatic(), hello.Code) + // FTW Cloud names the device by the key its own handshake proved. One + // that differs from the key this handshake proves is answered with + // silence, so a service cannot spend one watch's requests on another. + if cloudwire.DeviceID(handshake.RemoteStatic()) != dev { + return nil, errExchangeNotDev + } + // A watch is added by its key, so a hello with a code is no watch of + // this build's. + if len(hello.Code) > 0 { + return nil, errExchangeCode + } + // Checked before anything is written, and written only once the + // session proves itself (route). + grant, commit, err := x.enroll.CheckWatch(handshake.RemoteStatic()) if err != nil { return nil, err } now := x.now() - sess := &exchangeSession{appStatic: handshake.RemoteStatic(), openedAt: now, lastAt: now} + sess := &exchangeSession{appStatic: handshake.RemoteStatic(), dev: dev, openedAt: now, lastAt: now, commit: commit} handler, err := x.build(sess, apiauth.Caller{ Subject: subjectOf(grant.DeviceID), Kind: apiauth.KindApp, @@ -288,18 +327,24 @@ func (x *exchanges) open(ctx context.Context, message1 []byte) ([]byte, error) { return reply, nil } -// route finds the session a transport message belongs to by letting each -// one's AEAD try, and runs the exchange in it. -func (x *exchanges) route(ctx context.Context, message []byte, arrived time.Time) []byte { +// route finds the session a transport message belongs to by letting each of +// dev's sessions' AEAD try, and runs the exchange in it. +func (x *exchanges) route(ctx context.Context, dev string, message []byte, arrived time.Time) []byte { x.mu.Lock() live := slices.Clone(x.live) x.mu.Unlock() for _, sess := range live { + if sess.dev != dev { + continue + } frame, err := sess.transport.Decrypt(message) if err != nil { continue } + if !x.prove(sess) { + return nil + } reply := x.exchange(ctx, sess, frame) if reply != nil && x.cmdFloor > 0 && isCmd(frame) { holdUntil(ctx, arrived.Add(x.cmdFloor)) @@ -318,7 +363,7 @@ func isCmd(frame []byte) bool { return err == nil && f.Envelope.T == appproto.MsgCmd } -// holdUntil waits for a moment on the real clock, or for the socket to go. +// holdUntil waits for a moment on the real clock, or for the box to stop. func holdUntil(ctx context.Context, at time.Time) { wait := time.Until(at) if wait <= 0 { @@ -380,31 +425,104 @@ func (x *exchanges) exchange(ctx context.Context, sess *exchangeSession, frame [ return reply } -// allowOpen counts a handshake against the minute's budget. -func (x *exchanges) allowOpen() bool { +// prove marks a session whose first transport message decrypted, and stamps +// the watch then. False when the watch was removed meanwhile; the session +// goes. +func (x *exchanges) prove(sess *exchangeSession) bool { + sess.mu.Lock() + proved := sess.proved + sess.proved = true + sess.mu.Unlock() + if proved || sess.commit == nil { + return true + } + if _, err := sess.commit(); err != nil { + x.drop(sess) + return false + } + return true +} + +// allowOpen counts a handshake against the minute's budget and the watch's +// own share of it. +func (x *exchanges) allowOpen(dev string) bool { x.mu.Lock() defer x.mu.Unlock() now := x.now() - x.opens = slices.DeleteFunc(x.opens, func(at time.Time) bool { return now.Sub(at) >= time.Minute }) - if len(x.opens) >= exchangeOpensPerMinute { + recent := func(at time.Time) bool { return now.Sub(at) >= time.Minute } + x.opens = slices.DeleteFunc(x.opens, recent) + if x.devOpens == nil { + x.devOpens = map[string][]time.Time{} + } + mine := slices.DeleteFunc(x.devOpens[dev], recent) + if len(x.opens) >= exchangeOpensPerMinute || len(mine) >= exchangeOpensPerWatch { + x.devOpens[dev] = mine return false } x.opens = append(x.opens, now) + x.devOpens[dev] = append(mine, now) + for d, at := range x.devOpens { + if len(slices.DeleteFunc(at, recent)) == 0 { + delete(x.devOpens, d) + } + } return true } -// admit keeps a new session, ending the oldest when the table is full. The -// newcomer is the watch somebody is holding now. +// admit keeps a new session. A watch holds at most two, and the table eight. +// Past either, the oldest session goes that has not proved itself, then the +// oldest; never the newcomer, which is the watch somebody holds now. A +// replayed message 1 opens a session that never proves itself, so replays +// end only one another, never a working session or another watch's. func (x *exchanges) admit(sess *exchangeSession) { x.mu.Lock() defer x.mu.Unlock() x.live = append(x.live, sess) + mine := func(s *exchangeSession) bool { return s.dev == sess.dev } + for count(x.live, mine) > MaxExchangeSessionsPerWatch { + x.evictLocked(sess, mine) + } for len(x.live) > MaxExchangeSessions { - x.live[0].close() - x.live = x.live[1:] + x.evictLocked(sess, func(*exchangeSession) bool { return true }) } } +// evictLocked ends the oldest session among those that match, an unproved +// one before a proved one, and never keep. The caller holds x.mu. +func (x *exchanges) evictLocked(keep *exchangeSession, match func(*exchangeSession) bool) { + victim := -1 + for i, s := range x.live { + if s == keep || !match(s) { + continue + } + s.mu.Lock() + proved := s.proved + s.mu.Unlock() + if !proved { + victim = i + break + } + if victim < 0 { + victim = i + } + } + if victim < 0 { + return + } + x.live[victim].close() + x.live = slices.Delete(x.live, victim, victim+1) +} + +func count(live []*exchangeSession, match func(*exchangeSession) bool) int { + n := 0 + for _, s := range live { + if match(s) { + n++ + } + } + return n +} + // prune ends every session that has run its course. func (x *exchanges) prune() { now := x.now() @@ -432,7 +550,7 @@ func (x *exchanges) drop(sess *exchangeSession) { } // dropByAppKey ends every session a removed key holds. Its next message 1 -// meets AuthoriseWatch and gets silence. +// meets HasWatch and gets silence. func (x *exchanges) dropByAppKey(pub []byte) int { x.mu.Lock() defer x.mu.Unlock() @@ -455,103 +573,118 @@ func (x *exchanges) count() int { } // -------------------------------------------------------------------------- -// The socket +// The carrier // -------------------------------------------------------------------------- -// keepExchange holds the exchange socket for one relay connection's epoch and -// handle, and reconnects it with its own backoff until the connection ends. A -// relay that refuses the role, or a socket that drops, costs the watch its -// door and nothing else. -func (u *Uplink) keepExchange(ctx context.Context, epoch int64, handle string) { - attempt := 0 - for ctx.Err() == nil { - target := fmt.Sprintf("%s/r/%d/%s/exchange", u.opts.Endpoint, epoch, handle) - dialCtx, cancel := context.WithTimeout(ctx, dialTimeout) - conn, response, err := u.dialer.DialContext(dialCtx, target, nil) - cancel() - if response != nil && response.Body != nil { - _ = response.Body.Close() - } - if err == nil { - started := time.Now() - err = u.serveExchange(ctx, conn) - if time.Since(started) >= backoffCap { - attempt = 0 - } - } - if ctx.Err() != nil { - return - } - u.log.Debug("exchange socket dropped; retrying", "err", err) +// XSender sends one x up the box's link to FTW Cloud. cloudlink.Link is one. +// SendX never blocks; an error means the answer did not leave, and the watch +// asks again. XAllowed says whether an answer may leave now: the link keeps +// x to a share of the tunnel's budget, and the phones' last share for +// themselves. +type XSender interface { + SendX(m cloudwire.X) error + XAllowed() bool +} - ceiling := min(backoffBase<= time.Minute }) + if len(e.taken) >= exchangeRequestsPerMinute { + return false + } + e.taken = append(e.taken, now) + return true +} + +// reply sends the answer under the request's dev and id. Silence is an +// empty d, so the service answers the watch at once rather than at its +// deadline. +func (e *Exchange) reply(m cloudwire.X, d []byte) { + e.mu.Lock() + s := e.sender + e.mu.Unlock() + if s == nil { + return + } + if err := s.SendX(cloudwire.X{Dev: m.Dev, ID: m.ID, D: d}); err != nil { + e.log.Debug("an exchange answer did not leave", "err", err) } } diff --git a/go/internal/appuplink/exchange_test.go b/go/internal/appuplink/exchange_test.go index f4dd4fd23..13c638644 100644 --- a/go/internal/appuplink/exchange_test.go +++ b/go/internal/appuplink/exchange_test.go @@ -9,11 +9,12 @@ import ( "testing" "time" - "github.com/gorilla/websocket" "github.com/srcfl/ftw/go/internal/apiauth" "github.com/srcfl/ftw/go/internal/appenroll" "github.com/srcfl/ftw/go/internal/appproto" + "github.com/srcfl/ftw/go/protocol" "github.com/srcfl/ftw/go/protocol/appwire" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) // handlerWith builds every session's handler over one box with a charger: @@ -60,6 +61,12 @@ func newXRig(t *testing.T) *xrig { if err != nil { t.Fatal(err) } + return newXRigOn(t, enroll) +} + +// newXRigOn is the rig on a box that exists already. +func newXRigOn(t *testing.T, enroll *appenroll.Identity) *xrig { + t.Helper() code, _, err := enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) if err != nil { t.Fatal(err) @@ -95,19 +102,25 @@ func (r *xrig) advance(d time.Duration) { r.now = r.now.Add(d) } -// addWatch pairs a watch with a code its phone minted and opens its first +// addWatch adds a watch by its key, as its phone does, and opens its first // session. func (r *xrig) addWatch(t *testing.T) (*watchClient, appproto.ExchangeHelloOK) { t.Helper() - code, _, err := r.enroll.MintWatchCode(r.phone.DeviceID, appenroll.WatchTTL) + w := r.newWatch(t) + return w, r.open(t, w, nil) +} + +// newWatch adds a watch by its key and opens nothing. +func (r *xrig) newWatch(t *testing.T) *watchClient { + t.Helper() + w, err := newWatchClient() if err != nil { t.Fatal(err) } - w, err := newWatchClient() - if err != nil { + if _, err := r.enroll.AddWatch(r.phone.DeviceID, w.static.Public()); err != nil { t.Fatal(err) } - return w, r.open(t, w, code) + return w } // open runs X1 for a watch and fails unless the box answered. @@ -120,7 +133,7 @@ func (r *xrig) open(t *testing.T, w *watchClient, code []byte) appproto.Exchange if len(message1) != exchangeOpenBytes { t.Fatalf("message 1 is %d bytes, want %d", len(message1), exchangeOpenBytes) } - message2 := r.x.answer(context.Background(), message1) + message2 := r.x.answer(context.Background(), w.dev(), message1) if message2 == nil { t.Fatal("the box did not answer the watch's message 1") } @@ -141,7 +154,7 @@ func (r *xrig) ask(t *testing.T, w *watchClient, msgType string, body any) []app if len(request) != exchangeRequestBytes { t.Fatalf("a %s request is %d bytes, want %d", msgType, len(request), exchangeRequestBytes) } - reply := r.x.answer(context.Background(), request) + reply := r.x.answer(context.Background(), w.dev(), request) if reply == nil { return nil } @@ -244,7 +257,7 @@ func TestTheExchangeTakesCmdAndNowAndNothingElse(t *testing.T) { if err != nil { t.Fatal(err) } - frames, err := w.answer(r.x.answer(context.Background(), request)) + frames, err := w.answer(r.x.answer(context.Background(), w.dev(), request)) if err != nil { t.Fatalf("%s: %v", c.t, err) } @@ -261,14 +274,15 @@ func TestTheExchangeTakesCmdAndNowAndNothingElse(t *testing.T) { CmdID: "watch-mode", Op: appproto.OpSetMode, Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 10 * 60 * 1000, Expect: appproto.Expect{Rev: 1}, }) - res := decodeAs[appproto.CmdResult](t, frames[0]) - if res.State != appproto.CmdRejected || res.Error == nil || res.Error.Code != appproto.ErrScopeDenied { - t.Fatalf("site.mode.set from a watch = %+v", res) + // The exchange takes the charger's hold alone (ADR 0009, decision B): + // the mode is refused before the command runs. + if e := decodeAs[appproto.ErrorBody](t, frames[0]); frames[0].Envelope.T != appproto.MsgError || e.Code != protocol.ErrOpNotGranted { + t.Fatalf("site.mode.set from a watch answered %v %+v", types(frames), e) } } // Shape: every request is one length and every answer another, whatever the -// op and the outcome, so the relay cannot tell a read from a command or a +// op and the outcome, so FTW Cloud cannot tell a read from a command or a // refusal from a success. func TestEveryExchangeHasOneLength(t *testing.T) { r := newXRig(t) @@ -279,7 +293,7 @@ func TestEveryExchangeHasOneLength(t *testing.T) { }{ {"applied", appproto.MsgCmd, chargeNow("shape-1")}, {"asked again", appproto.MsgCmd, chargeNow("shape-1")}, - {"scope denied", appproto.MsgCmd, appproto.Cmd{CmdID: "shape-2", Op: appproto.OpSetMode, + {"not granted", appproto.MsgCmd, appproto.Cmd{CmdID: "shape-2", Op: appproto.OpSetMode, Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 600_000, Expect: appproto.Expect{Rev: 1}}}, {"expired", appproto.MsgCmd, appproto.Cmd{CmdID: "shape-3", Op: appproto.OpLoadpointHold, Args: map[string]any{"id": "lp1"}, NotValidAfterMs: 1, Expect: appproto.Expect{Rev: 1}}}, @@ -295,7 +309,7 @@ func TestEveryExchangeHasOneLength(t *testing.T) { if err != nil { t.Fatal(err) } - reply := r.x.answer(context.Background(), request) + reply := r.x.answer(context.Background(), w.dev(), request) if len(request) != exchangeRequestBytes || len(reply) != want { t.Fatalf("%s: request %d B, answer %d B; want %d and %d", c.name, len(request), len(reply), exchangeRequestBytes, want) @@ -338,24 +352,17 @@ func TestAnUnsettledResultIsAskedForAgain(t *testing.T) { // outcome. func TestReplayGetsNothingNew(t *testing.T) { r := newXRig(t) - code, _, err := r.enroll.MintWatchCode(r.phone.DeviceID, appenroll.WatchTTL) - if err != nil { - t.Fatal(err) - } - w, err := newWatchClient() - if err != nil { - t.Fatal(err) - } - message1, err := w.open(r.enroll.StaticKey().Public(), code) + w := r.newWatch(t) + message1, err := w.open(r.enroll.StaticKey().Public(), nil) if err != nil { t.Fatal(err) } - first := r.x.answer(context.Background(), message1) + first := r.x.answer(context.Background(), w.dev(), message1) if _, _, err := w.opened(first); err != nil { t.Fatal(err) } - again := r.x.answer(context.Background(), message1) + again := r.x.answer(context.Background(), w.dev(), message1) if again == nil || bytes.Equal(first[:appwire.DHBytes], again[:appwire.DHBytes]) { t.Fatal("a replayed message 1 drew the same box ephemeral key") } @@ -377,11 +384,11 @@ func TestReplayGetsNothingNew(t *testing.T) { if err != nil { t.Fatal(err) } - firstAnswer, err := w.answer(r.x.answer(context.Background(), request)) + firstAnswer, err := w.answer(r.x.answer(context.Background(), w.dev(), request)) if err != nil { t.Fatal(err) } - if reply := r.x.answer(context.Background(), request); reply != nil { + if reply := r.x.answer(context.Background(), w.dev(), request); reply != nil { t.Fatal("a recorded command was answered in its own session") } @@ -476,7 +483,7 @@ func TestOnlyAPairedWatchIsAnswered(t *testing.T) { if err != nil { t.Fatal(err) } - if r.x.answer(context.Background(), message1) != nil { + if r.x.answer(context.Background(), stranger.dev(), message1) != nil { t.Fatal("a stranger was answered") } @@ -499,7 +506,7 @@ func TestOnlyAPairedWatchIsAnswered(t *testing.T) { if err != nil { t.Fatal(err) } - if r.x.answer(context.Background(), streamOpen) != nil { + if r.x.answer(context.Background(), w.dev(), streamOpen) != nil { t.Fatal("a stream handshake opened an exchange session") } @@ -519,7 +526,7 @@ func TestOnlyAPairedWatchIsAnswered(t *testing.T) { if err != nil { t.Fatal(err) } - if r.x.answer(context.Background(), message1) != nil { + if r.x.answer(context.Background(), phone.dev(), message1) != nil { t.Fatal("a phone opened an exchange session") } } @@ -530,7 +537,7 @@ func TestRemovalEndsAWatchsSessions(t *testing.T) { for _, removePhone := range []bool{false, true} { r := newXRig(t) w, _ := r.addWatch(t) - target := deviceIDOf(w.static.Public()) + target := w.dev() if removePhone { target = r.phone.DeviceID } @@ -553,7 +560,7 @@ func TestRemovalEndsAWatchsSessions(t *testing.T) { if err != nil { t.Fatal(err) } - if r.x.answer(context.Background(), message1) != nil { + if r.x.answer(context.Background(), w.dev(), message1) != nil { t.Fatal("a removed watch's message 1 was answered") } } @@ -591,151 +598,281 @@ func TestASessionIsShort(t *testing.T) { } } -// At most sixty handshakes a minute; the rest are dropped unanswered. +// At most twelve handshakes a minute from one watch and sixty in all; the +// rest are dropped unanswered. A device FTW Cloud names that is no watch of +// this box spends nothing, not even a Diffie-Hellman, and one watch's opens +// never spend another's. func TestHandshakesAreRationed(t *testing.T) { r := newXRig(t) junk := make([]byte, exchangeOpenBytes) - for i := 0; i < exchangeOpensPerMinute; i++ { + for i := 0; i < 2*exchangeOpensPerMinute; i++ { junk[0] = byte(i) - r.x.answer(context.Background(), junk) + r.x.answer(context.Background(), "nobody00", junk) } - code, _, err := r.enroll.MintWatchCode(r.phone.DeviceID, appenroll.WatchTTL) + + w := r.newWatch(t) + message1, err := w.open(r.enroll.StaticKey().Public(), nil) if err != nil { t.Fatal(err) } - w, err := newWatchClient() - if err != nil { - t.Fatal(err) + for i := 0; i < exchangeOpensPerWatch; i++ { + if r.x.answer(context.Background(), w.dev(), message1) == nil { + t.Fatalf("open %d of a watch's twelve was not answered after a stranger's junk", i+1) + } } - message1, err := w.open(r.enroll.StaticKey().Public(), code) + if r.x.answer(context.Background(), w.dev(), message1) != nil { + t.Fatal("a watch's thirteenth handshake in a minute was answered") + } + // Another watch has a share of its own. + r.addWatch(t) + + // Sixty in all: four more watches spend their twelve each. + for k := 0; k < 3; k++ { + other := r.newWatch(t) + m1, err := other.open(r.enroll.StaticKey().Public(), nil) + if err != nil { + t.Fatal(err) + } + for i := 0; i < exchangeOpensPerWatch; i++ { + r.x.answer(context.Background(), other.dev(), m1) + } + } + late := r.newWatch(t) + m1, err := late.open(r.enroll.StaticKey().Public(), nil) if err != nil { t.Fatal(err) } - if r.x.answer(context.Background(), message1) != nil { - t.Fatal("the sixty-first handshake in a minute was answered") + for i := 0; i < exchangeOpensPerWatch; i++ { + r.x.answer(context.Background(), late.dev(), m1) + } + if r.x.answer(context.Background(), late.dev(), m1) != nil { + t.Fatal("a handshake past sixty in a minute was answered") } r.advance(time.Minute) - if r.x.answer(context.Background(), message1) == nil { + if r.x.answer(context.Background(), w.dev(), message1) == nil { t.Fatal("a handshake a minute later was not answered") } } -// Isolation: eight watch sessions leave four phone sessions running, and a +// A replayed message 1 opens a session that never proves itself: it stamps +// no visit, and replays end only one another, never the watch's working +// session and never another watch's. +func TestReplaysEndNoWorkingSession(t *testing.T) { + r := newXRig(t) + other, _ := r.addWatch(t) + if r.ask(t, other, appproto.MsgNow, struct{}{}) == nil { + t.Fatal("the other watch was not answered") + } + + w := r.newWatch(t) + message1, err := w.open(r.enroll.StaticKey().Public(), nil) + if err != nil { + t.Fatal(err) + } + message2 := r.x.answer(context.Background(), w.dev(), message1) + if _, _, err := w.opened(message2); err != nil { + t.Fatal(err) + } + if r.ask(t, w, appproto.MsgNow, struct{}{}) == nil { + t.Fatal("the watch's own session was not answered") + } + seen := func(dev string) int64 { + for _, d := range r.enroll.Devices() { + if d.ID == dev { + return d.LastSeenMs + } + } + return -1 + } + stamped := seen(w.dev()) + + r.advance(time.Second) + for i := 0; i < MaxExchangeSessions; i++ { + if r.x.answer(context.Background(), w.dev(), message1) == nil { + t.Fatalf("replay %d drew no answer", i+1) + } + } + if got := seen(w.dev()); got != stamped { + t.Fatalf("replays stamped the watch's visit: %d, was %d", got, stamped) + } + if n := count(r.x.live, func(s *exchangeSession) bool { return s.dev == w.dev() }); n != MaxExchangeSessionsPerWatch { + t.Fatalf("the watch holds %d sessions, want %d", n, MaxExchangeSessionsPerWatch) + } + if r.ask(t, w, appproto.MsgNow, struct{}{}) == nil { + t.Fatal("replays ended the watch's working session") + } + if r.ask(t, other, appproto.MsgNow, struct{}{}) == nil { + t.Fatal("one watch's replays ended another watch's session") + } +} + +// Removed mid-session, a watch's next command is refused and reaches no +// charger, even before anything ends its session. +func TestARevokeMidSessionRefusesTheNextCommand(t *testing.T) { + r := newXRig(t) + w, _ := r.addWatch(t) + if _, err := r.enroll.Revoke(w.dev(), appenroll.AtTheBox); err != nil { + t.Fatal(err) + } + frames := r.ask(t, w, appproto.MsgCmd, chargeNow("after-revoke")) + if frames != nil { + for _, f := range frames { + if f.Envelope.T == appproto.MsgCmdResult && decodeAs[appproto.CmdResult](t, f).State == appproto.CmdApplied { + t.Fatal("a removed watch's command was applied") + } + } + } + if r.charger.count() != 0 { + t.Fatalf("a removed watch's command reached the charger %d times", r.charger.count()) + } +} + +// A box that restarted has forgotten the commands it answered (protocol-cloud +// .md, "Exchange"). Its hello_ok says so: the uptime starts over, so the watch +// can tell a retry would reach a box that would act again. +func TestARestartedBoxSaysSoInItsClock(t *testing.T) { + r := newXRig(t) + _, before := r.addWatch(t) + restarted := newXRigOn(t, r.enroll) + restarted.x.build = func(sn appproto.Sender, caller apiauth.Caller, grants appproto.GrantReader) (*appproto.Handler, error) { + return appproto.New(appproto.Config{ + Clock: appproto.SystemClock{StartedAt: time.Now(), Source: "ntp"}, Site: stubSite{}, Info: stubInfo{}, + Modes: stubModes{}, Plans: stubPlans{}, Loadpoints: restarted.charger, Codec: codec{}, Sender: sn, + Caller: caller, Grants: grants, Commands: appproto.NewCmdLog(), Exchange: true, + SrcGrid: "meter", SrcPV: "meter", SrcBattery: "meter", Logger: slog.New(slog.DiscardHandler), + }) + } + _, after := restarted.addWatch(t) + if after.Clock.UptimeMs >= before.Clock.UptimeMs { + t.Fatalf("a restarted box's hello_ok says uptime %d, the first said %d", after.Clock.UptimeMs, before.Clock.UptimeMs) + } +} + +// Isolation: eight watch sessions leave the phones' sessions running, and a // ninth watch ends the oldest watch's session, never a phone's. func TestWatchesNeverTakeAPhonesPlace(t *testing.T) { - // The relay lets four streams into a room. const phones = 4 - r := newXRig(t) - streams := relayTable(&slots{}, r.enroll) + tr := newTunRig(t) + r := newXRigOn(t, tr.enroll) for i := 0; i < phones; i++ { - code, _, err := r.enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) - if err != nil { - t.Fatal(err) - } - app, err := newAppClient() - if err != nil { - t.Fatal(err) - } - relayOpen(t, streams, r.enroll, app, code) + app := tr.pair(t) + dev := cloudwire.DeviceID(app.staticPublic()) + tr.open(t, app, dev, uint32(i+1), nil) + tr.hello(t, app, dev, uint32(i+1)) + } + if tr.tunnel.count() != phones { + t.Fatalf("%d phone sessions, want %d", tr.tunnel.count(), phones) } first, _ := r.addWatch(t) for i := 1; i < MaxExchangeSessions; i++ { r.addWatch(t) } - if streams.count() != phones || r.x.count() != MaxExchangeSessions { + if tr.tunnel.count() != phones || r.x.count() != MaxExchangeSessions { t.Fatalf("%d phone and %d watch sessions, want %d and %d", - streams.count(), r.x.count(), phones, MaxExchangeSessions) + tr.tunnel.count(), r.x.count(), phones, MaxExchangeSessions) } r.addWatch(t) - if streams.count() != phones { - t.Fatalf("a ninth watch left %d phone sessions", streams.count()) + if tr.tunnel.count() != phones { + t.Fatalf("a ninth watch left %d phone sessions", tr.tunnel.count()) } if r.ask(t, first, appproto.MsgNow, struct{}{}) != nil { t.Fatal("the oldest watch session outlived a ninth") } } -// -------------------------------------------------------------------------- -// The socket, against a relay -// -------------------------------------------------------------------------- +// A request FTW Cloud names under another device gets silence, whether it +// opens a session or rides one: a service cannot spend one watch's requests +// on another. +func TestAnotherDevicesNameGetsSilence(t *testing.T) { + r := newXRig(t) + w, _ := r.addWatch(t) + other, _ := r.addWatch(t) -// newExchangeRig is a box whose uplink also keeps the exchange socket. -func newExchangeRig(t *testing.T, charger *fakeCharger) *rig { - t.Helper() - relay := newFakeRelay(481234) - t.Cleanup(relay.close) - enroll, err := appenroll.LoadOrCreate(filepath.Join(t.TempDir(), "nova.key")) + message1, err := w.open(r.enroll.StaticKey().Public(), nil) if err != nil { t.Fatal(err) } - cmds := appproto.NewCmdLog() - now := time.UnixMilli(481234 * EpochMs) - uplink, err := New(Options{ - Endpoint: relay.url(), - Enroll: enroll, - Handler: handlerWith(false, charger, cmds), - ExchangeHandler: handlerWith(true, charger, cmds), - Logger: slog.New(slog.DiscardHandler), - Now: func() time.Time { return now }, - Random: func() float64 { return 0 }, - }) + if r.x.answer(context.Background(), other.dev(), message1) != nil { + t.Fatal("a handshake was answered under another watch's id") + } + + r.open(t, w, nil) + request, err := w.request(appproto.MsgNow, struct{}{}) if err != nil { t.Fatal(err) } - ctx, cancel := context.WithCancel(context.Background()) - t.Cleanup(cancel) - done := make(chan error, 1) - go func() { done <- uplink.runRelay(ctx) }() - return &rig{relay: relay, enroll: enroll, uplink: uplink, cancel: cancel, done: done} + if r.x.answer(context.Background(), other.dev(), request) != nil { + t.Fatal("a request was answered under another watch's id") + } + if frames := r.ask(t, w, appproto.MsgNow, struct{}{}); frames == nil { + t.Fatal("the watch's own session died of a request under another id") + } } -func waitForExchange(t *testing.T, relay *fakeRelay) { - t.Helper() +// -------------------------------------------------------------------------- +// The carrier, as FTW Cloud drives it +// -------------------------------------------------------------------------- + +// capturedX is the box's link to FTW Cloud, as the exchange sees it. +// closed says the link's share for x is spent. +type capturedX struct { + ch chan cloudwire.X + closed bool +} + +func (c *capturedX) XAllowed() bool { return !c.closed } + +func (c *capturedX) SendX(m cloudwire.X) error { select { - case <-relay.exchangeJoined: - case <-time.After(5 * time.Second): - t.Fatal("the box never opened its exchange socket") + case c.ch <- m: + return nil + default: + return errTunGone } } -// The box opens /r///exchange beside its uplink, under the -// same handle, and a watch charges through it. -func TestTheBoxOpensAnExchangeBesideItsUplink(t *testing.T) { - charger := &fakeCharger{} - r := newExchangeRig(t, charger) - // These sockets start beside one another. Exchange-ready alone does - // not mean the uplink's handler has recorded its join yet. - deadline := time.NewTimer(5 * time.Second) - defer deadline.Stop() - joined, exchangeReady := 0, false - for joined < 2 || !exchangeReady { - select { - case <-r.relay.joined: - joined++ - case <-r.relay.exchangeJoined: - exchangeReady = true - case <-deadline.C: - t.Fatal("the box did not open both its uplink and exchange sockets") - } +func (c *capturedX) next(t *testing.T) cloudwire.X { + t.Helper() + select { + case m := <-c.ch: + return m + case <-time.After(5 * time.Second): + t.Fatal("the box sent no x") + return cloudwire.X{} } +} - handle, err := Handle(r.enroll.RendezvousSecret(), 481234) +// A watch its phone added by its key charges through the uplink's exchange: +// each request comes as x {dev, id, d} and each answer goes back as x with +// the same dev and id. +func TestAWatchChargesThroughFTWCloud(t *testing.T) { + enroll, err := appenroll.LoadOrCreate(filepath.Join(t.TempDir(), "nova.key")) if err != nil { t.Fatal(err) } - joinedUnder := 0 - for _, h := range r.relay.seenHandles() { - if h == handle { - joinedUnder++ - } + charger := &fakeCharger{} + cmds := appproto.NewCmdLog() + uplink, err := New(Options{ + Enroll: enroll, + Handler: handlerWith(false, charger, cmds), + ExchangeHandler: handlerWith(true, charger, cmds), + Logger: slog.New(slog.DiscardHandler), + }) + if err != nil { + t.Fatal(err) } - if joinedUnder < 2 { - t.Fatalf("the box joined its handle %d times; the uplink and the exchange share it", joinedUnder) + if uplink.exchange.table.cmdFloor != ExchangeCmdFloor { + t.Fatalf("the uplink holds commands for %s, want %s", uplink.exchange.table.cmdFloor, ExchangeCmdFloor) } + out := &capturedX{ch: make(chan cloudwire.X, 16)} + uplink.Exchange().Attach(out) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + go func() { _ = uplink.Run(ctx) }() - code, _, err := r.enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) + code, _, err := enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) if err != nil { t.Fatal(err) } @@ -743,85 +880,82 @@ func TestTheBoxOpensAnExchangeBesideItsUplink(t *testing.T) { if err != nil { t.Fatal(err) } - phone, err := r.enroll.Authorise(phoneKey.Public(), code) + phone, err := enroll.Authorise(phoneKey.Public(), code) if err != nil { t.Fatal(err) } - watchCode, _, err := r.enroll.MintWatchCode(phone.DeviceID, appenroll.WatchTTL) + w, err := newWatchClient() if err != nil { t.Fatal(err) } - w, err := newWatchClient() - if err != nil { + if _, err := enroll.AddWatch(phone.DeviceID, w.static.Public()); err != nil { t.Fatal(err) } - message1, err := w.open(r.enroll.StaticKey().Public(), watchCode) + + message1, err := w.open(enroll.StaticKey().Public(), nil) if err != nil { t.Fatal(err) } - message2 := r.relay.post(1, message1, 5*time.Second) - if message2 == nil { - t.Fatal("the box did not answer message 1 on its exchange socket") + uplink.Exchange().Deliver(cloudwire.X{Dev: w.dev(), ID: 7, D: message1}) + got := out.next(t) + if got.Dev != w.dev() || got.ID != 7 || len(got.D) == 0 { + t.Fatalf("the handshake's answer is %+v", got) } - if _, _, err := w.opened(message2); err != nil { + if _, _, err := w.opened(got.D); err != nil { t.Fatal(err) } - request, err := w.request(appproto.MsgCmd, chargeNow("socket-1")) + + request, err := w.request(appproto.MsgCmd, chargeNow("cloud-1")) if err != nil { t.Fatal(err) } - frames, err := w.answer(r.relay.post(2, request, 5*time.Second)) + uplink.Exchange().Deliver(cloudwire.X{Dev: w.dev(), ID: 8, D: request}) + got = out.next(t) + if got.ID != 8 { + t.Fatalf("the command's answer came as %+v", got) + } + frames, err := w.answer(got.D) if err != nil { t.Fatal(err) } - if got := types(frames); len(got) != 2 || got[1] != appproto.MsgCmdResult { - t.Fatalf("the answer carries %v", got) + if kinds := types(frames); len(kinds) != 2 || kinds[0] != appproto.MsgCmdAck || kinds[1] != appproto.MsgCmdResult { + t.Fatalf("the answer carries %v", kinds) } if charger.count() != 1 { t.Fatalf("the charger saw %d holds", charger.count()) } - // Removing the watch through the uplink ends its exchange session too. - if dropped := r.uplink.DropSessionsByAppKey(w.static.Public()); dropped != 1 { - t.Fatalf("DropSessionsByAppKey dropped %d sessions, want the watch's one", dropped) - } -} - -// Losing the exchange socket changes nothing for the uplink, and the box -// opens it again. -func TestLosingTheExchangeLeavesTheUplink(t *testing.T) { - r := newExchangeRig(t, &fakeCharger{}) - waitForExchange(t, r.relay) - - code, _, err := r.enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) - if err != nil { - t.Fatal(err) - } - conn := r.dialApp(t, 481234) - app, err := newAppClient() - if err != nil { - t.Fatal(err) - } - message1, err := app.message1(r.enroll.StaticKey().Public(), code) + // Silence is an empty d under the request's own number, so the service + // answers the watch at once. + again, err := w.request(appproto.MsgNow, struct{}{}) if err != nil { t.Fatal(err) } - if err := conn.WriteMessage(websocket.BinaryMessage, message1); err != nil { - t.Fatal(err) - } - if err := app.readMessage2(waitForBinary(t, conn)); err != nil { - t.Fatal(err) + uplink.Exchange().Deliver(cloudwire.X{Dev: "someone", ID: 9, D: again}) + if got = out.next(t); got.ID != 9 || got.Dev != "someone" || len(got.D) != 0 { + t.Fatalf("a request under another id got %+v", got) } - r.relay.dropExchange() - waitForExchange(t, r.relay) + // Removing the watch through the uplink ends its exchange session too. + if dropped := uplink.DropSessionsByAppKey(w.static.Public()); dropped != 1 { + t.Fatalf("DropSessionsByAppKey dropped %d sessions, want the watch's one", dropped) + } +} - send(t, conn, app, appproto.MsgHello, 512, appproto.Hello{ - Proto: appproto.ProtoRange{Min: 0, Max: appproto.ProtoMax}, - App: appproto.AppInfo{Build: "test", UA: "pwa"}, - }) - if got := receive(t, conn, app); got.Envelope.T != appproto.MsgHelloOK { - t.Fatalf("after the exchange dropped, the phone's session answered %q", got.Envelope.T) +// A box with more requests than it answers at once answers the rest with +// silence at once, rather than late. +func TestABusyExchangeAnswersSilenceAtOnce(t *testing.T) { + r := newXRig(t) + e := newExchange(r.x, slog.New(slog.DiscardHandler)) + out := &capturedX{ch: make(chan cloudwire.X, 2*exchangeInbox)} + e.Attach(out) + // Nothing runs the inbox, so it fills, and the one past it is answered. + for i := 0; i < exchangeInbox; i++ { + e.Deliver(cloudwire.X{Dev: "w", ID: uint32(i), D: []byte{1}}) + } + e.Deliver(cloudwire.X{Dev: "w", ID: 99, D: []byte{1}}) + if got := out.next(t); got.ID != 99 || len(got.D) != 0 { + t.Fatalf("the request past the inbox got %+v", got) } } @@ -837,7 +971,7 @@ func (r *xrig) timed(t *testing.T, w *watchClient, msgType string, body any) ([] t.Fatal(err) } started := time.Now() - reply := r.x.answer(context.Background(), request) + reply := r.x.answer(context.Background(), w.dev(), request) took := time.Since(started) frames, err := w.answer(reply) if err != nil { @@ -883,10 +1017,49 @@ func TestACommandAnswersNoSoonerThanTheFloor(t *testing.T) { } } -// The box wires the floor; only a table built by hand in a test goes without. -func TestTheUplinkHoldsCommandsForTheFloor(t *testing.T) { - r := newExchangeRig(t, &fakeCharger{}) - if r.uplink.exchanges.cmdFloor != ExchangeCmdFloor { - t.Fatalf("the uplink holds commands for %s, want %s", r.uplink.exchanges.cmdFloor, ExchangeCmdFloor) +// With the link's share for x spent, a request runs nothing: no answer +// could tell the watch what happened, so the command does not happen. +func TestNoAnswerMeansNoCommand(t *testing.T) { + r := newXRig(t) + w, _ := r.addWatch(t) + e := newExchange(r.x, slog.New(slog.DiscardHandler)) + out := &capturedX{ch: make(chan cloudwire.X, 4), closed: true} + e.Attach(out) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + go e.Run(ctx) + + request, err := w.request(appproto.MsgCmd, chargeNow("budget-spent")) + if err != nil { + t.Fatal(err) + } + e.Deliver(cloudwire.X{Dev: w.dev(), ID: 1, D: request}) + select { + case got := <-out.ch: + t.Fatalf("the box answered %+v with its share spent", got) + case <-time.After(1500 * time.Millisecond): + } + if r.charger.count() != 0 { + t.Fatal("a command ran whose answer could not leave") + } +} + +// Past the minute's requests the box drops the rest unanswered and runs +// none of them. +func TestTheExchangeTakesAMinutesRequestsAndNoMore(t *testing.T) { + r := newXRig(t) + e := newExchange(r.x, slog.New(slog.DiscardHandler)) + e.Attach(&capturedX{ch: make(chan cloudwire.X, 1)}) + for i := 0; i < exchangeRequestsPerMinute; i++ { + if !e.take() { + t.Fatalf("request %d of the minute was refused", i+1) + } + } + if e.take() { + t.Fatal("a request past the minute's was taken") + } + r.advance(time.Minute) + if !e.take() { + t.Fatal("a request a minute later was refused") } } diff --git a/go/internal/appuplink/exchange_timing_test.go b/go/internal/appuplink/exchange_timing_test.go new file mode 100644 index 000000000..1e967783c --- /dev/null +++ b/go/internal/appuplink/exchange_timing_test.go @@ -0,0 +1,51 @@ +package appuplink + +import ( + "fmt" + "os" + "slices" + "strconv" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/appproto" +) + +// TestExchangeTimingAtTheBox times an applied and a refused command at the +// box, the gap decision F's floor hides from whoever carries the exchange. +// Skipped unless FTW_EXCHANGE_TIMING is set; it reports and asserts nothing. +func TestExchangeTimingAtTheBox(t *testing.T) { + if os.Getenv("FTW_EXCHANGE_TIMING") == "" { + t.Skip("set FTW_EXCHANGE_TIMING=1 to time answers at the box") + } + save, _ := strconv.Atoi(os.Getenv("FTW_EV_SAVE_MS")) + r := newXRig(t) + r.x.cmdFloor = ExchangeCmdFloor + r.charger.save = time.Duration(save) * time.Millisecond + var applied, refused []time.Duration + var w *watchClient + for i := 0; i < 40; i++ { + if i%2 == 0 { + // A second on the rig's clock per handshake keeps the run under + // the sixty a minute the box allows. + r.advance(time.Second) + w, _ = r.addWatch(t) + } + started := time.Now() + r.ask(t, w, appproto.MsgCmd, chargeNow(fmt.Sprintf("box-timing-%d", i))) + applied = append(applied, time.Since(started)) + started = time.Now() + r.ask(t, w, appproto.MsgCmd, appproto.Cmd{CmdID: fmt.Sprintf("box-refused-%d", i), Op: appproto.OpSetMode, + Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 600_000, Expect: appproto.Expect{Rev: 1}}) + refused = append(refused, time.Since(started)) + } + t.Logf("at the box, EV save %d ms: applied %s; refused %s", save, spread(applied), spread(refused)) +} + +// spread is p50 and p95 of a sample. +func spread(d []time.Duration) string { + s := slices.Clone(d) + slices.Sort(s) + at := func(p float64) time.Duration { return s[min(len(s)-1, int(p*float64(len(s))))] } + return fmt.Sprintf("p50 %s p95 %s (n=%d)", at(0.5).Round(10*time.Microsecond), at(0.95).Round(10*time.Microsecond), len(s)) +} diff --git a/go/internal/appuplink/realrelay_test.go b/go/internal/appuplink/realrelay_test.go deleted file mode 100644 index 080373f74..000000000 --- a/go/internal/appuplink/realrelay_test.go +++ /dev/null @@ -1,236 +0,0 @@ -package appuplink - -import ( - "bytes" - "context" - "fmt" - "io" - "log/slog" - "net/http" - "os" - "path/filepath" - "slices" - "strconv" - "strings" - "testing" - "time" - - "github.com/srcfl/ftw/go/internal/apiauth" - "github.com/srcfl/ftw/go/internal/appenroll" - "github.com/srcfl/ftw/go/internal/appproto" - "github.com/srcfl/ftw/go/protocol/appwire" -) - -// The exchange end to end: this box, srcfl/ftw-webapp's relay, and a watch in -// Go posting over HTTP (ADR 0009, milestone 5). Skipped unless a relay with -// the exchange route is running: -// -// PORT=8797 HOST=127.0.0.1 node relay/src/main.ts # in srcfl/ftw-webapp -// FTW_RELAY_URL=ws://127.0.0.1:8797 go test ./internal/appuplink -run RealRelay -v -// -// FTW_EV_SAVE_MS sets how long the charger's save takes, so the timing below -// can show what an applied command waits for. The relay allows 30 exchanges a -// minute per room, so the run stays under that. -func TestExchangeThroughARealRelay(t *testing.T) { - endpoint := os.Getenv("FTW_RELAY_URL") - if endpoint == "" { - t.Skip("set FTW_RELAY_URL to a relay with the exchange route") - } - base := "http" + strings.TrimPrefix(endpoint, "ws") - save, _ := strconv.Atoi(os.Getenv("FTW_EV_SAVE_MS")) - - charger := &fakeCharger{save: time.Duration(save) * time.Millisecond} - enroll, err := appenroll.LoadOrCreate(filepath.Join(t.TempDir(), "nova.key")) - if err != nil { - t.Fatal(err) - } - cmds := appproto.NewCmdLog() - uplink, err := New(Options{ - Endpoint: endpoint, - Enroll: enroll, - Handler: handlerWith(false, charger, cmds), - ExchangeHandler: handlerWith(true, charger, cmds), - Logger: slog.New(slog.DiscardHandler), - }) - if err != nil { - t.Fatal(err) - } - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - go func() { _ = uplink.runRelay(ctx) }() - - code, _, err := enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) - if err != nil { - t.Fatal(err) - } - phoneKey, err := appwire.GenerateKeyPair() - if err != nil { - t.Fatal(err) - } - phone, err := enroll.Authorise(phoneKey.Public(), code) - if err != nil { - t.Fatal(err) - } - watchCode, _, err := enroll.MintWatchCode(phone.DeviceID, appenroll.WatchTTL) - if err != nil { - t.Fatal(err) - } - w, err := newWatchClient() - if err != nil { - t.Fatal(err) - } - - epoch := CurrentEpoch(time.Now().UnixMilli()) - post := func(body []byte) (int, []byte, time.Duration) { - t.Helper() - handle, err := Handle(enroll.RendezvousSecret(), epoch) - if err != nil { - t.Fatal(err) - } - started := time.Now() - resp, err := http.Post(fmt.Sprintf("%s/x/%d/%s", base, epoch, handle), "application/octet-stream", bytes.NewReader(body)) - if err != nil { - t.Fatal(err) - } - defer resp.Body.Close() - answer, _ := io.ReadAll(resp.Body) - return resp.StatusCode, answer, time.Since(started) - } - - // X1. The box may still be joining, which the relay answers 404 (no box) - // or 501 (a box with no exchange socket yet). - message1, err := w.open(enroll.StaticKey().Public(), watchCode) - if err != nil { - t.Fatal(err) - } - var status int - var message2 []byte - for try := 0; try < 6; try++ { - status, message2, _ = post(message1) - if status == http.StatusConflict { - if n, err := strconv.ParseInt(string(message2), 10, 64); err == nil { - epoch = n - } - continue - } - if status != http.StatusNotFound && status != http.StatusNotImplemented { - break - } - time.Sleep(500 * time.Millisecond) - } - if status != http.StatusOK { - t.Fatalf("X1 through the relay answered %d", status) - } - if len(message2) != appwire.Message2Overhead+1024 { - t.Fatalf("message 2 is %d bytes", len(message2)) - } - hello, _, err := w.opened(message2) - if err != nil { - t.Fatal(err) - } - if hello.Role != apiauth.RoleWatch { - t.Fatalf("hello_ok = %+v", hello) - } - - exchange := func(msgType string, body any) ([]appwire.Frame, time.Duration) { - t.Helper() - request, err := w.request(msgType, body) - if err != nil { - t.Fatal(err) - } - status, answer, took := post(request) - if status != http.StatusOK || len(answer) != exchangeAnswerFrames*exchangeRequestBytes { - t.Fatalf("%s through the relay answered %d with %d bytes", msgType, status, len(answer)) - } - frames, err := w.answer(answer) - if err != nil { - t.Fatal(err) - } - return frames, took - } - reopen := func() { - t.Helper() - message1, err := w.open(enroll.StaticKey().Public(), nil) - if err != nil { - t.Fatal(err) - } - status, message2, _ := post(message1) - if status != http.StatusOK { - t.Fatalf("reopening answered %d", status) - } - if _, _, err := w.opened(message2); err != nil { - t.Fatal(err) - } - } - - // A charge, the same charge again, a refusal and a read: one session. - frames, _ := exchange(appproto.MsgCmd, chargeNow("relay-1")) - if got := types(frames); got[0] != appproto.MsgCmdAck || got[1] != appproto.MsgCmdResult { - t.Fatalf("the charge answered %v", got) - } - frames, _ = exchange(appproto.MsgCmd, chargeNow("relay-1")) - if got := types(frames); got[1] != appproto.MsgCmdResult || charger.count() != 1 { - t.Fatalf("asking again answered %v after %d holds", got, charger.count()) - } - frames, _ = exchange(appproto.MsgCmd, appproto.Cmd{CmdID: "relay-2", Op: appproto.OpSetMode, - Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 600_000, Expect: appproto.Expect{Rev: 1}}) - if res := decodeAs[appproto.CmdResult](t, frames[0]); res.Error == nil || res.Error.Code != appproto.ErrScopeDenied { - t.Fatalf("site.mode.set from a watch answered %+v", res) - } - frames, _ = exchange(appproto.MsgNow, struct{}{}) - if frames[0].Envelope.T != appproto.MsgNow { - t.Fatalf("now answered %v", types(frames)) - } - - // Timing (decision F): applied and refused, through the relay. - var applied, refused []time.Duration - for i := 0; i < 8; i++ { - if i%2 == 0 { - reopen() - } - _, took := exchange(appproto.MsgCmd, chargeNow(fmt.Sprintf("relay-timing-%d", i))) - applied = append(applied, took) - _, took = exchange(appproto.MsgCmd, appproto.Cmd{CmdID: fmt.Sprintf("relay-refused-%d", i), Op: appproto.OpSetMode, - Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 600_000, Expect: appproto.Expect{Rev: 1}}) - refused = append(refused, took) - } - t.Logf("through the relay, EV save %d ms: applied %s; refused %s", save, spread(applied), spread(refused)) -} - -// TestExchangeTimingAtTheBox times the same two answers without the relay. -// Skipped unless FTW_EXCHANGE_TIMING is set; it reports and asserts nothing. -func TestExchangeTimingAtTheBox(t *testing.T) { - if os.Getenv("FTW_EXCHANGE_TIMING") == "" { - t.Skip("set FTW_EXCHANGE_TIMING=1 to time answers at the box") - } - save, _ := strconv.Atoi(os.Getenv("FTW_EV_SAVE_MS")) - r := newXRig(t) - r.x.cmdFloor = ExchangeCmdFloor - r.charger.save = time.Duration(save) * time.Millisecond - var applied, refused []time.Duration - var w *watchClient - for i := 0; i < 40; i++ { - if i%2 == 0 { - // A second on the rig's clock per handshake keeps the run under - // the sixty a minute the box allows. - r.advance(time.Second) - w, _ = r.addWatch(t) - } - started := time.Now() - r.ask(t, w, appproto.MsgCmd, chargeNow(fmt.Sprintf("box-timing-%d", i))) - applied = append(applied, time.Since(started)) - started = time.Now() - r.ask(t, w, appproto.MsgCmd, appproto.Cmd{CmdID: fmt.Sprintf("box-refused-%d", i), Op: appproto.OpSetMode, - Args: map[string]any{"mode": "charge"}, NotValidAfterMs: 600_000, Expect: appproto.Expect{Rev: 1}}) - refused = append(refused, time.Since(started)) - } - t.Logf("at the box, EV save %d ms: applied %s; refused %s", save, spread(applied), spread(refused)) -} - -// spread is p50 and p95 of a sample. -func spread(d []time.Duration) string { - s := slices.Clone(d) - slices.Sort(s) - at := func(p float64) time.Duration { return s[min(len(s)-1, int(p*float64(len(s))))] } - return fmt.Sprintf("p50 %s p95 %s (n=%d)", at(0.5).Round(10*time.Microsecond), at(0.95).Round(10*time.Microsecond), len(s)) -} diff --git a/go/internal/appuplink/roster.go b/go/internal/appuplink/roster.go index bd6db82b9..d1a2302f1 100644 --- a/go/internal/appuplink/roster.go +++ b/go/internal/appuplink/roster.go @@ -4,7 +4,6 @@ import ( "slices" "time" - "github.com/srcfl/ftw/go/internal/apiauth" "github.com/srcfl/ftw/go/internal/appenroll" "github.com/srcfl/ftw/go/internal/appproto" "github.com/srcfl/ftw/go/internal/cloudlink" @@ -16,7 +15,8 @@ import ( // device with the scopes the box derives for it, a phone's role, or for a // watch what its role and its phone's role share. With Tunnel set, a phone // may open a session through FTW Cloud and the QR code on the screen is -// listed; a watch has no stream and reaches the box through its exchange. +// listed, and a watch may reach the box's exchange through it: the box +// answers a watch's key on the exchange alone, never on a stream. type CloudRoster struct { Enroll *appenroll.Identity Tunnel bool @@ -31,7 +31,7 @@ func (r CloudRoster) devices(entries []appenroll.RosterEntry) []cloudlink.Roster var out []cloudlink.RosterDevice for _, e := range entries { d := cloudlink.RosterDevice{ID: e.ID, Key: e.Key, Role: e.Role, Epoch: e.Epoch, - Scopes: appproto.GrantScopes(e.Role, e.ParentRole), Tun: r.Tunnel && e.Role != apiauth.RoleWatch} + Scopes: appproto.GrantScopes(e.Role, e.ParentRole), Tun: r.Tunnel} if a := e.Agent; a != nil { // An agent row carries its grant, at its approver's scopes now, // so FTW Cloud can hold to it as well (docs/agents.md). It has diff --git a/go/internal/appuplink/testdata/exchange-vectors.json b/go/internal/appuplink/testdata/exchange-vectors.json index 8dffe046c..ff06a3278 100644 --- a/go/internal/appuplink/testdata/exchange-vectors.json +++ b/go/internal/appuplink/testdata/exchange-vectors.json @@ -8,16 +8,16 @@ "watchStatic": "2122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f40", "watchStaticPublic": "5869aff450549732cbaaed5e5df9b30a6da31cb0e5742bad5ad4a1a768f1a67b", "watchEphemeral": "4142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f60", - "handshakeHash": "774830fbffaa569c1fa626f5c527ad12e1085cb996a2cd60d60028344675ca8b", + "handshakeHash": "d936deb10439e4db202de8480d5e83caf3b57fa132408499c2f95d3a1517f364", "steps": [ { "name": "x1.request", "from": "watch", "frames": [ - "01000000005aa261746568656c6c6f6162a463617070a26275616777617463686f73656275696c6467766563746f727364636f646550a1a2a3a4a5a6a7a8a9aaabacadaeafb06570726f746fa2636d617801636d696e00676c6f63616c6573f60000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + "010000000044a261746568656c6c6f6162a363617070a26275616777617463686f73656275696c6467766563746f72736570726f746fa2636d617801636d696e00676c6f63616c6573f6000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], - "body": "64b101b1d0be5a8704bd078f9895001fc03e8e9f9522f188dd128d9846d4846616dbb9de4a9e094239c6b6f322ce389abeff592a58ae1d32e9ce750b0388b9801ea66db7bd728aecae5f2e271dfa40f7c36852a5e3fbccf7a61a1f6de53d0f6bb0a0e1974d707403b77e19748b7c341b29697ead61a52340567a0abbdbee1e9307318322030306e6dbfe817ae2fbfeb54ca044286a855abc7335a3d67961e8d3f8665a14e25de60db22318fc40fdfddfc2d9896c4e6495995219e081124e3089707b6b66337280b6e4ab0182eb19e5e4ec57392f2a802c1dc2dbdc774e723ac0bf16f68b9f4a79a66388043beb5073a972e3160bebc192c02cc82339e72cf31b8662a6de21104774c0c22de6772b2fa19603b6f18185c2d8eafbaf5bd9dbf1546f2f02ddb2e24b3b6adcdbf2e1726202861e647c8943fa24c53ac7d81039d93f8399611761a3310b3794fc0a6afcfea03702be673a46002d5d6cc1c3f58d214092a1342020353b13daf7f1a4a73b367d2418cc0f25e4cfde316dd67cdb55f14136d754bb2277eee456653ac05da873b0e327210a70de55bbe3ae430137336f49a0f51858db8c8b6761575d56a0c9a6aa36357da6dad200d51b32eb4c0b85b129feb10d2d40ee3be0c0eeed918e34d5cc58ddcbfc7cdff9660d0e953a978ad7cba41c6a0746bb5d3c40b59ee4b482e035084d40a5817104a5fe80982283c0849406bdf36b32497ad46aa94a84f66f081f4b248fc29f50e289bf888d2a7ab8d74ac2821db0faa0c13f7ea5ed4e7eaca6b3c1444d05baa50aa71be40edc003db7418b78645680e26c92d8672be523350a3803a57ace446dd88e20bdbdc465fc36b4", - "relay": "8182838485868788898a8b8c8d8e8f9064b101b1d0be5a8704bd078f9895001fc03e8e9f9522f188dd128d9846d4846616dbb9de4a9e094239c6b6f322ce389abeff592a58ae1d32e9ce750b0388b9801ea66db7bd728aecae5f2e271dfa40f7c36852a5e3fbccf7a61a1f6de53d0f6bb0a0e1974d707403b77e19748b7c341b29697ead61a52340567a0abbdbee1e9307318322030306e6dbfe817ae2fbfeb54ca044286a855abc7335a3d67961e8d3f8665a14e25de60db22318fc40fdfddfc2d9896c4e6495995219e081124e3089707b6b66337280b6e4ab0182eb19e5e4ec57392f2a802c1dc2dbdc774e723ac0bf16f68b9f4a79a66388043beb5073a972e3160bebc192c02cc82339e72cf31b8662a6de21104774c0c22de6772b2fa19603b6f18185c2d8eafbaf5bd9dbf1546f2f02ddb2e24b3b6adcdbf2e1726202861e647c8943fa24c53ac7d81039d93f8399611761a3310b3794fc0a6afcfea03702be673a46002d5d6cc1c3f58d214092a1342020353b13daf7f1a4a73b367d2418cc0f25e4cfde316dd67cdb55f14136d754bb2277eee456653ac05da873b0e327210a70de55bbe3ae430137336f49a0f51858db8c8b6761575d56a0c9a6aa36357da6dad200d51b32eb4c0b85b129feb10d2d40ee3be0c0eeed918e34d5cc58ddcbfc7cdff9660d0e953a978ad7cba41c6a0746bb5d3c40b59ee4b482e035084d40a5817104a5fe80982283c0849406bdf36b32497ad46aa94a84f66f081f4b248fc29f50e289bf888d2a7ab8d74ac2821db0faa0c13f7ea5ed4e7eaca6b3c1444d05baa50aa71be40edc003db7418b78645680e26c92d8672be523350a3803a57ace446dd88e20bdbdc465fc36b4" + "body": "64b101b1d0be5a8704bd078f9895001fc03e8e9f9522f188dd128d9846d4846616dbb9de4a9e094239c6b6f322ce389abeff592a58ae1d32e9ce750b0388b9801ea66db7bd728aecae5f2e271dfa40f7c36852a5e3e5ccf7a61a1f6de53d0f6bb0a7e1974d707403b77e19748b7c341b29697ead61a52340567a0abbdbee1e9306229e29123c0527153b5cdd263e3e71e76b85e9a65453a972acd7b9db0285b2806739798b33e66ade4c7b9d2c988e29c2d9896c4e6495995219e081124e3089707b6b66337280b6e4ab0182eb19e5e4ec57392f2a802c1dc2dbdc774e723ac0bf16f68b9f4a79a66388043beb5073a972e3160bebc192c02cc82339e72cf31b8662a6de21104774c0c22de6772b2fa19603b6f18185c2d8eafbaf5bd9dbf1546f2f02ddb2e24b3b6adcdbf2e1726202861e647c8943fa24c53ac7d81039d93f8399611761a3310b3794fc0a6afcfea03702be673a46002d5d6cc1c3f58d214092a1342020353b13daf7f1a4a73b367d2418cc0f25e4cfde316dd67cdb55f14136d754bb2277eee456653ac05da873b0e327210a70de55bbe3ae430137336f49a0f51858db8c8b6761575d56a0c9a6aa36357da6dad200d51b32eb4c0b85b129feb10d2d40ee3be0c0eeed918e34d5cc58ddcbfc7cdff9660d0e953a978ad7cba41c6a0746bb5d3c40b59ee4b482e035084d40a5817104a5fe80982283c0849406bdf36b32497ad46aa94a84f66f081f4b248fc29f50e289bf888d2a7ab8d74ac2821db0faa0c13f7ea5ed4e7eaca6b3c1444d05baa50aa71be40edc003db7418b78645680e26c92d8672be523350a381e0e1c0b3c8a375ffb7ef031e983de01", + "uplink": "010100000286a36162590277a3616459026064b101b1d0be5a8704bd078f9895001fc03e8e9f9522f188dd128d9846d4846616dbb9de4a9e094239c6b6f322ce389abeff592a58ae1d32e9ce750b0388b9801ea66db7bd728aecae5f2e271dfa40f7c36852a5e3e5ccf7a61a1f6de53d0f6bb0a7e1974d707403b77e19748b7c341b29697ead61a52340567a0abbdbee1e9306229e29123c0527153b5cdd263e3e71e76b85e9a65453a972acd7b9db0285b2806739798b33e66ade4c7b9d2c988e29c2d9896c4e6495995219e081124e3089707b6b66337280b6e4ab0182eb19e5e4ec57392f2a802c1dc2dbdc774e723ac0bf16f68b9f4a79a66388043beb5073a972e3160bebc192c02cc82339e72cf31b8662a6de21104774c0c22de6772b2fa19603b6f18185c2d8eafbaf5bd9dbf1546f2f02ddb2e24b3b6adcdbf2e1726202861e647c8943fa24c53ac7d81039d93f8399611761a3310b3794fc0a6afcfea03702be673a46002d5d6cc1c3f58d214092a1342020353b13daf7f1a4a73b367d2418cc0f25e4cfde316dd67cdb55f14136d754bb2277eee456653ac05da873b0e327210a70de55bbe3ae430137336f49a0f51858db8c8b6761575d56a0c9a6aa36357da6dad200d51b32eb4c0b85b129feb10d2d40ee3be0c0eeed918e34d5cc58ddcbfc7cdff9660d0e953a978ad7cba41c6a0746bb5d3c40b59ee4b482e035084d40a5817104a5fe80982283c0849406bdf36b32497ad46aa94a84f66f081f4b248fc29f50e289bf888d2a7ab8d74ac2821db0faa0c13f7ea5ed4e7eaca6b3c1444d05baa50aa71be40edc003db7418b78645680e26c92d8672be523350a381e0e1c0b3c8a375ffb7ef031e983de0162696401636465766857476d7639464255617461786373696740" }, { "name": "x1.answer", @@ -25,8 +25,8 @@ "frames": [ "0101000000a9a261746868656c6c6f5f6f6b6162a66372657607646d6f64656466756c6c64726f6c6565776174636865636c6f636ba366736f75726365636e747068757074696d654d7319ea606a73796e63656441744d731b00000199c82cc0006570726f746f016673636f70657384726674772e64697370617463682e77726974656f6674772e6576656e74732e726561646d6674772e6c6976652e726561646d6674772e706c616e2e72656164000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], - "body": "244fe3b963e899dd295baffce248d3530f3a9a7479ba063002680ebfe7adad4991a408cd653ddf237f674b24d2df0548a107b8b35917e568c1d82f4463ba1da63832f4cd8f1db95e1d3a4ac19d27eb387fc479d455516c17e3e7a00136f9bf3b09fb666854531c0dd747cdc85109219b3996982844b365370e1616744294a597c19813bab2c15d2a3d9f59ef3a4e06ae1b5d18bdbe3356f12480871c20682670114ad3fb5b4b1e4cf6ea3ca85e172f39e934b73092cd80e280a496233c68179307c72d9e1ec6fe539d3bfa0a56dfbd1d10a02d54bc03bd2e3977bf7490b5fb4e1abb6ee28641ee44444c9beca86c673266971f98d2b1f0dbd123804c8a0fb9d3709c65348370e410c43f2042e7ffaeda2745fe81a11c6a961ac90ffb86f51c3e108ab8ea932620a2b7ef48377d4cc7b8478db79c2e2d329f4d22efa8a2bc098d6db8872b0271d2d8298a8bbff67dfd7a1bd63d284087fdb063f6b2bb32deee2740862e8ce3bd83bfd7dc2e898296e0b9caccad7027126f40c1987992d5a75d9824498d2e226740f892dd780c715f38b34034ccbdd98171b7868201ab8b4ebe0970d599f1962ed15d2ecc045667734ae1b1c7f210b6e0b44e198f2c44155a00d1befd484430cf97a974d540ce6bb20df298690ffb5606cc02c4b27f24e99dbe086df83abb7a240e873144e4091495be53a3b274956de126a10f84e5b18dd50666cc99459a23bb0ec16fb78110c577b4b46888fb15e8617d853876ea185a45b68375c8fff37cffc564ae0f6ee397ce4cba22011ac1c6bf87312a5a354b74c83e1cc4584adb51ae20695f69b2d6708e860a95d17b206afad31a7848e89a1c489cd849bb338d2e0601e1925b0265768d4e85233c185c107dad76328e9398c314d763ab2c1c400295d5f48ed728fcfa789bed80ad818b94dbad263632a9ec8f1828b56f0bd0345da0f218c81483a9c688eaaf26fb35085e5db204bc52c0f70d300795207b2f30d1975c6798c8e4ce6c8494886bc655753c9d5e851da7c22d631fd7da87b485f0506d906ebade0a9c9140bc29b5f92655da7f88c0fc3523d861320b9f9cf91cedecf0c06eafff0d9eeb9acf71f4cd00922b270dc85534daf03ac7e92e0e0a3da9781e2635677ecf28f1e6a7475c021d0dc141a9ab2b7a54375a3eff83093c8d07a00e5a20b105ac7977ee03b281c44c7e6e54ae3d7e946e4192c57ad65a55282e7472ac2a923a749ea03aa491bc4fffda6f7192a1a417b5b50a9f315b9753d72536c37ff047ca40b11a68a90d6468dec4e2e18134389fd876c0d550ba7f8cc15e7cc4fb02356c2e3e45e01e646fb357a81c9d47ea4454f982037864eea74762eb1fc3e3123e461e73dace602cffe66dcfc89ed0ae0545e5715992303a724003e175c86e7da8a9b77c6866d9afb50d505247c313b86aead1fbc0a7130a7e409cb9b8439f36155a8ca64ddb6430c952a4509d3d95fc84452ddcc502514d025891f424ead9ccd1ae8b04540f1c8f", - "relay": "8182838485868788898a8b8c8d8e8f90244fe3b963e899dd295baffce248d3530f3a9a7479ba063002680ebfe7adad4991a408cd653ddf237f674b24d2df0548a107b8b35917e568c1d82f4463ba1da63832f4cd8f1db95e1d3a4ac19d27eb387fc479d455516c17e3e7a00136f9bf3b09fb666854531c0dd747cdc85109219b3996982844b365370e1616744294a597c19813bab2c15d2a3d9f59ef3a4e06ae1b5d18bdbe3356f12480871c20682670114ad3fb5b4b1e4cf6ea3ca85e172f39e934b73092cd80e280a496233c68179307c72d9e1ec6fe539d3bfa0a56dfbd1d10a02d54bc03bd2e3977bf7490b5fb4e1abb6ee28641ee44444c9beca86c673266971f98d2b1f0dbd123804c8a0fb9d3709c65348370e410c43f2042e7ffaeda2745fe81a11c6a961ac90ffb86f51c3e108ab8ea932620a2b7ef48377d4cc7b8478db79c2e2d329f4d22efa8a2bc098d6db8872b0271d2d8298a8bbff67dfd7a1bd63d284087fdb063f6b2bb32deee2740862e8ce3bd83bfd7dc2e898296e0b9caccad7027126f40c1987992d5a75d9824498d2e226740f892dd780c715f38b34034ccbdd98171b7868201ab8b4ebe0970d599f1962ed15d2ecc045667734ae1b1c7f210b6e0b44e198f2c44155a00d1befd484430cf97a974d540ce6bb20df298690ffb5606cc02c4b27f24e99dbe086df83abb7a240e873144e4091495be53a3b274956de126a10f84e5b18dd50666cc99459a23bb0ec16fb78110c577b4b46888fb15e8617d853876ea185a45b68375c8fff37cffc564ae0f6ee397ce4cba22011ac1c6bf87312a5a354b74c83e1cc4584adb51ae20695f69b2d6708e860a95d17b206afad31a7848e89a1c489cd849bb338d2e0601e1925b0265768d4e85233c185c107dad76328e9398c314d763ab2c1c400295d5f48ed728fcfa789bed80ad818b94dbad263632a9ec8f1828b56f0bd0345da0f218c81483a9c688eaaf26fb35085e5db204bc52c0f70d300795207b2f30d1975c6798c8e4ce6c8494886bc655753c9d5e851da7c22d631fd7da87b485f0506d906ebade0a9c9140bc29b5f92655da7f88c0fc3523d861320b9f9cf91cedecf0c06eafff0d9eeb9acf71f4cd00922b270dc85534daf03ac7e92e0e0a3da9781e2635677ecf28f1e6a7475c021d0dc141a9ab2b7a54375a3eff83093c8d07a00e5a20b105ac7977ee03b281c44c7e6e54ae3d7e946e4192c57ad65a55282e7472ac2a923a749ea03aa491bc4fffda6f7192a1a417b5b50a9f315b9753d72536c37ff047ca40b11a68a90d6468dec4e2e18134389fd876c0d550ba7f8cc15e7cc4fb02356c2e3e45e01e646fb357a81c9d47ea4454f982037864eea74762eb1fc3e3123e461e73dace602cffe66dcfc89ed0ae0545e5715992303a724003e175c86e7da8a9b77c6866d9afb50d505247c313b86aead1fbc0a7130a7e409cb9b8439f36155a8ca64ddb6430c952a4509d3d95fc84452ddcc502514d025891f424ead9ccd1ae8b04540f1c8f" + "body": "244fe3b963e899dd295baffce248d3530f3a9a7479ba063002680ebfe7adad4991a408cd653ddf237f674b24d2df0548a107b8b35917e568c1d82f4463ba1da63832f4cd8f1db95e1d3a4ac19d27eb387fc479d455516c17e3e7a00136f9bf3b09fb666854531c0dd747cdc85109219b3996982844b365370e1616744294a597c19813bab2c15d2a3d9f59ef3a4e06ae1b5d18bdbe3356f12480871c20682670114ad3fb5b4b1e4cf6ea3ca85e172f39e934b73092cd80e280a496233c68179307c72d9e1ec6fe539d3bfa0a56dfbd1d10a02d54bc03bd2e3977bf7490b5fb4e1abb6ee28641ee44444c9beca86c673266971f98d2b1f0dbd123804c8a0fb9d3709c65348370e410c43f2042e7ffaeda2745fe81a11c6a961ac90ffb86f51c3e108ab8ea932620a2b7ef48377d4cc7b8478db79c2e2d329f4d22efa8a2bc098d6db8872b0271d2d8298a8bbff67dfd7a1bd63d284087fdb063f6b2bb32deee2740862e8ce3bd83bfd7dc2e898296e0b9caccad7027126f40c1987992d5a75d9824498d2e226740f892dd780c715f38b34034ccbdd98171b7868201ab8b4ebe0970d599f1962ed15d2ecc045667734ae1b1c7f210b6e0b44e198f2c44155a00d1befd484430cf97a974d540ce6bb20df298690ffb5606cc02c4b27f24e99dbe086df83abb7a240e873144e4091495be53a3b274956de126a10f84e5b18dd50666cc99459a23bb0ec16fb78110c577b4b46888fb15e8617d853876ea185a45b68375c8fff37cffc564ae0f6ee397ce4cba22011ac1c6bf87312a5a354b74c83e1cc4584adb51ae20695f69b2d6708e860a95d17b206afad31a7848e89a1c489cd849bb338d2e0601e1925b0265768d4e85233c185c107dad76328e9398c314d763ab2c1c400295d5f48ed728fcfa789bed80ad818b94dbad263632a9ec8f1828b56f0bd0345da0f218c81483a9c688eaaf26fb35085e5db204bc52c0f70d300795207b2f30d1975c6798c8e4ce6c8494886bc655753c9d5e851da7c22d631fd7da87b485f0506d906ebade0a9c9140bc29b5f92655da7f88c0fc3523d861320b9f9cf91cedecf0c06eafff0d9eeb9acf71f4cd00922b270dc85534daf03ac7e92e0e0a3da9781e2635677ecf28f1e6a7475c021d0dc141a9ab2b7a54375a3eff83093c8d07a00e5a20b105ac7977ee03b281c44c7e6e54ae3d7e946e4192c57ad65a55282e7472ac2a923a749ea03aa491bc4fffda6f7192a1a417b5b50a9f315b9753d72536c37ff047ca40b11a68a90d6468dec4e2e18134389fd876c0d550ba7f8cc15e7cc4fb02356c2e3e45e01e646fb357a81c9d47ea4454f982037864eea74762eb1fc3e3123e461e73dace602cffe66dcfc89ed0ae0545e5715992303a724003e175c86e7da8a9b77c6866d9afb50d505247c313b86aead1fbc0a7130a7e409cb9b8439f36155a8ca64ddb6430c952a4509d3d95fc84452ddcc502514d2a53684fed13eac565ac5c65f9642ed9", + "uplink": "010100000456a36162590447a36164590430244fe3b963e899dd295baffce248d3530f3a9a7479ba063002680ebfe7adad4991a408cd653ddf237f674b24d2df0548a107b8b35917e568c1d82f4463ba1da63832f4cd8f1db95e1d3a4ac19d27eb387fc479d455516c17e3e7a00136f9bf3b09fb666854531c0dd747cdc85109219b3996982844b365370e1616744294a597c19813bab2c15d2a3d9f59ef3a4e06ae1b5d18bdbe3356f12480871c20682670114ad3fb5b4b1e4cf6ea3ca85e172f39e934b73092cd80e280a496233c68179307c72d9e1ec6fe539d3bfa0a56dfbd1d10a02d54bc03bd2e3977bf7490b5fb4e1abb6ee28641ee44444c9beca86c673266971f98d2b1f0dbd123804c8a0fb9d3709c65348370e410c43f2042e7ffaeda2745fe81a11c6a961ac90ffb86f51c3e108ab8ea932620a2b7ef48377d4cc7b8478db79c2e2d329f4d22efa8a2bc098d6db8872b0271d2d8298a8bbff67dfd7a1bd63d284087fdb063f6b2bb32deee2740862e8ce3bd83bfd7dc2e898296e0b9caccad7027126f40c1987992d5a75d9824498d2e226740f892dd780c715f38b34034ccbdd98171b7868201ab8b4ebe0970d599f1962ed15d2ecc045667734ae1b1c7f210b6e0b44e198f2c44155a00d1befd484430cf97a974d540ce6bb20df298690ffb5606cc02c4b27f24e99dbe086df83abb7a240e873144e4091495be53a3b274956de126a10f84e5b18dd50666cc99459a23bb0ec16fb78110c577b4b46888fb15e8617d853876ea185a45b68375c8fff37cffc564ae0f6ee397ce4cba22011ac1c6bf87312a5a354b74c83e1cc4584adb51ae20695f69b2d6708e860a95d17b206afad31a7848e89a1c489cd849bb338d2e0601e1925b0265768d4e85233c185c107dad76328e9398c314d763ab2c1c400295d5f48ed728fcfa789bed80ad818b94dbad263632a9ec8f1828b56f0bd0345da0f218c81483a9c688eaaf26fb35085e5db204bc52c0f70d300795207b2f30d1975c6798c8e4ce6c8494886bc655753c9d5e851da7c22d631fd7da87b485f0506d906ebade0a9c9140bc29b5f92655da7f88c0fc3523d861320b9f9cf91cedecf0c06eafff0d9eeb9acf71f4cd00922b270dc85534daf03ac7e92e0e0a3da9781e2635677ecf28f1e6a7475c021d0dc141a9ab2b7a54375a3eff83093c8d07a00e5a20b105ac7977ee03b281c44c7e6e54ae3d7e946e4192c57ad65a55282e7472ac2a923a749ea03aa491bc4fffda6f7192a1a417b5b50a9f315b9753d72536c37ff047ca40b11a68a90d6468dec4e2e18134389fd876c0d550ba7f8cc15e7cc4fb02356c2e3e45e01e646fb357a81c9d47ea4454f982037864eea74762eb1fc3e3123e461e73dace602cffe66dcfc89ed0ae0545e5715992303a724003e175c86e7da8a9b77c6866d9afb50d505247c313b86aead1fbc0a7130a7e409cb9b8439f36155a8ca64ddb6430c952a4509d3d95fc84452ddcc502514d2a53684fed13eac565ac5c65f9642ed962696401636465766857476d7639464255617461786373696740" }, { "name": "cmd.request", @@ -35,7 +35,7 @@ "0100000000a0a2617463636d646162a5626f706e6c6f6164706f696e742e686f6c646461726773a4626964636c703166686f6c645f730067706f7765725f77191cc06a70686173655f6d6f646562337065636d644964782430313932663261302d376331652d373030302d383030302d30313233343536373839616266657870656374a2637265760766677561726473f66f6e6f7456616c696441667465724d731a00015f9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], "body": "00000000000000002f05f2f827669013b6401855f2c3530841cb799d4eefd8e17b1e37645aae08390436a01cd65d95d5d99ee068e6c4b29e17fae5691ed76e590e137c5c5bc511aae997370da80239ab098e976b3f52268757596ceec30e2c53b8c06a0a045bbc1cc1eae5b96b6bb5ebef1de9e33ef289d080e76709f70ad415fbe8f4384c854af091a421343dba5b1517dac9b4b6bcb36666b0788dbe76f941ed69c63ecffda45567b4bf8c2added7b80ad2fa111d1a2108b3bfe4c496a80fa9cb0a7b5ea783b98a6aafea982ff73c09b8f9aacfdf1cac0344b7d49023b16fccd2f06f79f8adfbe50394962711a16cb9f6ca18220d279eeb34eb8492d0e81bbec5277eeba7e3457f6616aa78c7099441245cf1fcf2006842a3ed9e4da83c28a867a470f8ff28c3444f2c443b222690cd1396f3005474511729c8db56d280789df8518ef6caecaa3dbab9b8cc24ba8bf6051d4dcef0da189694b2a5f7e537dcc3f10782b2b8cf39e2dc42be4910aed75a46e56a5d629480b331dce2443c072275dbfeda40e03c76d2afb6467af46e8b1f8055b612f9745612fd7dea0c3f64366c72e0e032a9fc2ed5f997ba12e88483a9a9ff539c3ad075a6227efe77009de6d0306298c059d394c2258a9efc165f5ac5be04e3e88a124a82454ab5581c9313ab193e2e44ea7f99220a9bb909d8fe6c0a274e0db5b7058ee7042492bf35ea0d12853283138d9ca4b9639dfd1674528f79ba59485e77617a3", - "relay": "8182838485868788898a8b8c8d8e8f9000000000000000002f05f2f827669013b6401855f2c3530841cb799d4eefd8e17b1e37645aae08390436a01cd65d95d5d99ee068e6c4b29e17fae5691ed76e590e137c5c5bc511aae997370da80239ab098e976b3f52268757596ceec30e2c53b8c06a0a045bbc1cc1eae5b96b6bb5ebef1de9e33ef289d080e76709f70ad415fbe8f4384c854af091a421343dba5b1517dac9b4b6bcb36666b0788dbe76f941ed69c63ecffda45567b4bf8c2added7b80ad2fa111d1a2108b3bfe4c496a80fa9cb0a7b5ea783b98a6aafea982ff73c09b8f9aacfdf1cac0344b7d49023b16fccd2f06f79f8adfbe50394962711a16cb9f6ca18220d279eeb34eb8492d0e81bbec5277eeba7e3457f6616aa78c7099441245cf1fcf2006842a3ed9e4da83c28a867a470f8ff28c3444f2c443b222690cd1396f3005474511729c8db56d280789df8518ef6caecaa3dbab9b8cc24ba8bf6051d4dcef0da189694b2a5f7e537dcc3f10782b2b8cf39e2dc42be4910aed75a46e56a5d629480b331dce2443c072275dbfeda40e03c76d2afb6467af46e8b1f8055b612f9745612fd7dea0c3f64366c72e0e032a9fc2ed5f997ba12e88483a9a9ff539c3ad075a6227efe77009de6d0306298c059d394c2258a9efc165f5ac5be04e3e88a124a82454ab5581c9313ab193e2e44ea7f99220a9bb909d8fe6c0a274e0db5b7058ee7042492bf35ea0d12853283138d9ca4b9639dfd1674528f79ba59485e77617a3" + "uplink": "01010000023ea3616259022fa3616459021800000000000000002f05f2f827669013b6401855f2c3530841cb799d4eefd8e17b1e37645aae08390436a01cd65d95d5d99ee068e6c4b29e17fae5691ed76e590e137c5c5bc511aae997370da80239ab098e976b3f52268757596ceec30e2c53b8c06a0a045bbc1cc1eae5b96b6bb5ebef1de9e33ef289d080e76709f70ad415fbe8f4384c854af091a421343dba5b1517dac9b4b6bcb36666b0788dbe76f941ed69c63ecffda45567b4bf8c2added7b80ad2fa111d1a2108b3bfe4c496a80fa9cb0a7b5ea783b98a6aafea982ff73c09b8f9aacfdf1cac0344b7d49023b16fccd2f06f79f8adfbe50394962711a16cb9f6ca18220d279eeb34eb8492d0e81bbec5277eeba7e3457f6616aa78c7099441245cf1fcf2006842a3ed9e4da83c28a867a470f8ff28c3444f2c443b222690cd1396f3005474511729c8db56d280789df8518ef6caecaa3dbab9b8cc24ba8bf6051d4dcef0da189694b2a5f7e537dcc3f10782b2b8cf39e2dc42be4910aed75a46e56a5d629480b331dce2443c072275dbfeda40e03c76d2afb6467af46e8b1f8055b612f9745612fd7dea0c3f64366c72e0e032a9fc2ed5f997ba12e88483a9a9ff539c3ad075a6227efe77009de6d0306298c059d394c2258a9efc165f5ac5be04e3e88a124a82454ab5581c9313ab193e2e44ea7f99220a9bb909d8fe6c0a274e0db5b7058ee7042492bf35ea0d12853283138d9ca4b9639dfd1674528f79ba59485e77617a362696402636465766857476d7639464255617461786373696740" }, { "name": "cmd.answer", @@ -45,7 +45,7 @@ "010000000073a261746a636d642e726573756c746162a365636d644964782430313932663261302d376331652d373030302d383030302d303132333435363738396162657374617465676170706c696564686f62736572766564a36373726364636f72656576616c7565f96f3068757074696d654d7319ee4800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], "body": "0000000000000000838f22d621c1e66d73bbdc82f73caa54360eef54cda0234b1fbfab2b827995c8ba42e1f96d434086fcc169a948ced5c831a611c8ffa39b090fd153aca92d25ef3ef58ac0e2ab6c8a22a5deca2b417f379cc7c5aa24d97ba847bdde7f7bf2934c508d678e53e484c92330f1ff050d413c4ab057e2b63a3b9dad88c2674e30771bb2a4702bb4b22aa37e21d8c795216571bdb5965d735e287ca457c4e7718d703e37472df910ca24b1bf28ec623862938dcf0e18bcafcdcad040ccbfbbc673d4ad0eb41adb10e9562581c83b98faa33dd2ee024d2445a899771f256e3a5f32c8486c22dda854c7343366eae435b3b13e774ef777a73cb074e7008d81ac888ce91655ea6a301372a91bc7a1838394cc7068a134f5055fc173d1366e176e2920415a74db10eda3fdfa004e24e2791d92694577013fc2cd24ba4157ad4ad338ff86c27ec265765d5eeb82b7e784d321c26d572b673623ec07faef9c97e18560ddff9f841b01a4b8bbe4599d5efb45b76020d7b76b2173df6881f8d37718f48caecb7a7e0099ad624d1798190194aaaca22f490c2c644f7c7d7e8d15704e80882b4012264889a429b9d71bded1d1f7878d5fa2dccdd261ca5059455156769e85e8f430ed12dc85d0aa8161066434ea18307eb766c460fece36993ca4a0e55f79d94eb74b1bc3759f75726b82082f661aa977f71e97d85fa7c82ec5a30029e50a8e76d23757f68dcf53585fa336e5b309621d2e0000000000000001bcec1b5bc5c32c1ccbeb75970da3607c892818005fa238dd8d499f2872d82be3c629c65e672f33090d8961a0dc2360e37b9648b11f888428ebd32b6a8cbc75154eadbfe23814cf8210a2154c13225781cafaa1d5fd7cd5dd0b851c6d1c927886b40f34b301ce45b8d387063506cfcc1e8c2e0ea6459e6300cff9095eff3ba6101256a7c1aee64ae8bf4be263ad6e989606f171d7f21e2f5bcdf2c27ec0d5ead1d37e3f669286691d9b5190a3825dbd51a8917df93893844350bbb048d73b005e9a6d869eb01155bd26ad6d82b2b88f50e45d9fc3ab35f1498a2bff32799b5d59d293fa7b88629e4b8bbeec640cf123595962da041278709aa32dd55a75088fb11e584fac66b82e1776444b67f049485c9101ef81868b3ffd2f872644c87f66d324e5b38e5fe96f000081f139b73e57dd42e58e7456105c2dbce6b8660d3c93ce64e35262867586bedc8405f698855c3f993ffbbddb34b320794e512da21ff829d3ae8ca35c42b7381397b8d4b92d6f2b73df93c5194ba4b31d2754784fc42629d9659bc403a80d9243d6ece4869b2c1d2f39e7da18ea77429ecd29e1acb41aaaca7516fd7908f04ec447e02c928d062ddf25b8dae802d9e1d4db2a8d38f6a8f74c1029209f132a0bcdbeb61cda0ff632c031c62669561805d75a317f8b10d690e5bdee169b0f225d746b9616ad245ebee0fc8a386de1c7aabf9d4c13bd51882f48d1808a995d69f9c88b988434a74712", - "relay": "8182838485868788898a8b8c8d8e8f900000000000000000838f22d621c1e66d73bbdc82f73caa54360eef54cda0234b1fbfab2b827995c8ba42e1f96d434086fcc169a948ced5c831a611c8ffa39b090fd153aca92d25ef3ef58ac0e2ab6c8a22a5deca2b417f379cc7c5aa24d97ba847bdde7f7bf2934c508d678e53e484c92330f1ff050d413c4ab057e2b63a3b9dad88c2674e30771bb2a4702bb4b22aa37e21d8c795216571bdb5965d735e287ca457c4e7718d703e37472df910ca24b1bf28ec623862938dcf0e18bcafcdcad040ccbfbbc673d4ad0eb41adb10e9562581c83b98faa33dd2ee024d2445a899771f256e3a5f32c8486c22dda854c7343366eae435b3b13e774ef777a73cb074e7008d81ac888ce91655ea6a301372a91bc7a1838394cc7068a134f5055fc173d1366e176e2920415a74db10eda3fdfa004e24e2791d92694577013fc2cd24ba4157ad4ad338ff86c27ec265765d5eeb82b7e784d321c26d572b673623ec07faef9c97e18560ddff9f841b01a4b8bbe4599d5efb45b76020d7b76b2173df6881f8d37718f48caecb7a7e0099ad624d1798190194aaaca22f490c2c644f7c7d7e8d15704e80882b4012264889a429b9d71bded1d1f7878d5fa2dccdd261ca5059455156769e85e8f430ed12dc85d0aa8161066434ea18307eb766c460fece36993ca4a0e55f79d94eb74b1bc3759f75726b82082f661aa977f71e97d85fa7c82ec5a30029e50a8e76d23757f68dcf53585fa336e5b309621d2e0000000000000001bcec1b5bc5c32c1ccbeb75970da3607c892818005fa238dd8d499f2872d82be3c629c65e672f33090d8961a0dc2360e37b9648b11f888428ebd32b6a8cbc75154eadbfe23814cf8210a2154c13225781cafaa1d5fd7cd5dd0b851c6d1c927886b40f34b301ce45b8d387063506cfcc1e8c2e0ea6459e6300cff9095eff3ba6101256a7c1aee64ae8bf4be263ad6e989606f171d7f21e2f5bcdf2c27ec0d5ead1d37e3f669286691d9b5190a3825dbd51a8917df93893844350bbb048d73b005e9a6d869eb01155bd26ad6d82b2b88f50e45d9fc3ab35f1498a2bff32799b5d59d293fa7b88629e4b8bbeec640cf123595962da041278709aa32dd55a75088fb11e584fac66b82e1776444b67f049485c9101ef81868b3ffd2f872644c87f66d324e5b38e5fe96f000081f139b73e57dd42e58e7456105c2dbce6b8660d3c93ce64e35262867586bedc8405f698855c3f993ffbbddb34b320794e512da21ff829d3ae8ca35c42b7381397b8d4b92d6f2b73df93c5194ba4b31d2754784fc42629d9659bc403a80d9243d6ece4869b2c1d2f39e7da18ea77429ecd29e1acb41aaaca7516fd7908f04ec447e02c928d062ddf25b8dae802d9e1d4db2a8d38f6a8f74c1029209f132a0bcdbeb61cda0ff632c031c62669561805d75a317f8b10d690e5bdee169b0f225d746b9616ad245ebee0fc8a386de1c7aabf9d4c13bd51882f48d1808a995d69f9c88b988434a74712" + "uplink": "010100000456a36162590447a361645904300000000000000000838f22d621c1e66d73bbdc82f73caa54360eef54cda0234b1fbfab2b827995c8ba42e1f96d434086fcc169a948ced5c831a611c8ffa39b090fd153aca92d25ef3ef58ac0e2ab6c8a22a5deca2b417f379cc7c5aa24d97ba847bdde7f7bf2934c508d678e53e484c92330f1ff050d413c4ab057e2b63a3b9dad88c2674e30771bb2a4702bb4b22aa37e21d8c795216571bdb5965d735e287ca457c4e7718d703e37472df910ca24b1bf28ec623862938dcf0e18bcafcdcad040ccbfbbc673d4ad0eb41adb10e9562581c83b98faa33dd2ee024d2445a899771f256e3a5f32c8486c22dda854c7343366eae435b3b13e774ef777a73cb074e7008d81ac888ce91655ea6a301372a91bc7a1838394cc7068a134f5055fc173d1366e176e2920415a74db10eda3fdfa004e24e2791d92694577013fc2cd24ba4157ad4ad338ff86c27ec265765d5eeb82b7e784d321c26d572b673623ec07faef9c97e18560ddff9f841b01a4b8bbe4599d5efb45b76020d7b76b2173df6881f8d37718f48caecb7a7e0099ad624d1798190194aaaca22f490c2c644f7c7d7e8d15704e80882b4012264889a429b9d71bded1d1f7878d5fa2dccdd261ca5059455156769e85e8f430ed12dc85d0aa8161066434ea18307eb766c460fece36993ca4a0e55f79d94eb74b1bc3759f75726b82082f661aa977f71e97d85fa7c82ec5a30029e50a8e76d23757f68dcf53585fa336e5b309621d2e0000000000000001bcec1b5bc5c32c1ccbeb75970da3607c892818005fa238dd8d499f2872d82be3c629c65e672f33090d8961a0dc2360e37b9648b11f888428ebd32b6a8cbc75154eadbfe23814cf8210a2154c13225781cafaa1d5fd7cd5dd0b851c6d1c927886b40f34b301ce45b8d387063506cfcc1e8c2e0ea6459e6300cff9095eff3ba6101256a7c1aee64ae8bf4be263ad6e989606f171d7f21e2f5bcdf2c27ec0d5ead1d37e3f669286691d9b5190a3825dbd51a8917df93893844350bbb048d73b005e9a6d869eb01155bd26ad6d82b2b88f50e45d9fc3ab35f1498a2bff32799b5d59d293fa7b88629e4b8bbeec640cf123595962da041278709aa32dd55a75088fb11e584fac66b82e1776444b67f049485c9101ef81868b3ffd2f872644c87f66d324e5b38e5fe96f000081f139b73e57dd42e58e7456105c2dbce6b8660d3c93ce64e35262867586bedc8405f698855c3f993ffbbddb34b320794e512da21ff829d3ae8ca35c42b7381397b8d4b92d6f2b73df93c5194ba4b31d2754784fc42629d9659bc403a80d9243d6ece4869b2c1d2f39e7da18ea77429ecd29e1acb41aaaca7516fd7908f04ec447e02c928d062ddf25b8dae802d9e1d4db2a8d38f6a8f74c1029209f132a0bcdbeb61cda0ff632c031c62669561805d75a317f8b10d690e5bdee169b0f225d746b9616ad245ebee0fc8a386de1c7aabf9d4c13bd51882f48d1808a995d69f9c88b988434a7471262696402636465766857476d7639464255617461786373696740" }, { "name": "now.request", @@ -54,7 +54,7 @@ "01000000000aa26174636e6f776162a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], "body": "0000000000000001eb023948c7de3836d2d2b54f2c07a885cbf5571e6f49c475b2b8975c6f2a04f5820a0e2cfacb4c98391c88b5a96539d611340e69050bfe0276bb5f3d634457c8ea82f347837fd85aa7b64ec223a8c818c0207b1eb79173cf86cb3133e939f19d6188e9d318dbc6b63826379fc08f27e9aa3636abca17bba3f11c260b281884f51d45d1290e97048cdc7683d1ff29da30d8e20804bab193c80f9a67d15cf637d228964d13ea89e471a00c085aa8d2437bdb38755a51fecbd26624e3130161b38a85223ae40bf8125dd06fab7bd341529572aa364b778a5c976c281e32214252fd288463ba2ee566f07b24b8a49039c0485f51244f1d1f18e950a077f55a0ddeca83c9dbfdaf894d2f7f2dde4c9a89efc46a4021b9139d854dd9a36782a810a92840d962d1e711e1e1044e24b4bebed02649922254ba96bb4f99d89fd161f11e0a8554109dc03b747f11a8a92149958d9c8be127de0f4e255a3244d0633db10df472500d710d4db65f86f928efa73dea0d23a133682e5fca385ad37924de2cfc772c04474e3b4c0b7d230fec5e25019f3e341379315e0a4a30c78c40af81ae932f76746f2bdf7a39b1aa2ad098679c3626e6b940d67c9d6c35cffc3c6070c7a33ee5d231336a231b9098e1440c30714fcdadf7031c128c46f881eb16eaa83668a2ba8d8021faf6db9522919de2918a05cdca883c1c3170c77c605e030a89835535378d68207e24b6b1565e9ef3196d5415", - "relay": "8182838485868788898a8b8c8d8e8f900000000000000001eb023948c7de3836d2d2b54f2c07a885cbf5571e6f49c475b2b8975c6f2a04f5820a0e2cfacb4c98391c88b5a96539d611340e69050bfe0276bb5f3d634457c8ea82f347837fd85aa7b64ec223a8c818c0207b1eb79173cf86cb3133e939f19d6188e9d318dbc6b63826379fc08f27e9aa3636abca17bba3f11c260b281884f51d45d1290e97048cdc7683d1ff29da30d8e20804bab193c80f9a67d15cf637d228964d13ea89e471a00c085aa8d2437bdb38755a51fecbd26624e3130161b38a85223ae40bf8125dd06fab7bd341529572aa364b778a5c976c281e32214252fd288463ba2ee566f07b24b8a49039c0485f51244f1d1f18e950a077f55a0ddeca83c9dbfdaf894d2f7f2dde4c9a89efc46a4021b9139d854dd9a36782a810a92840d962d1e711e1e1044e24b4bebed02649922254ba96bb4f99d89fd161f11e0a8554109dc03b747f11a8a92149958d9c8be127de0f4e255a3244d0633db10df472500d710d4db65f86f928efa73dea0d23a133682e5fca385ad37924de2cfc772c04474e3b4c0b7d230fec5e25019f3e341379315e0a4a30c78c40af81ae932f76746f2bdf7a39b1aa2ad098679c3626e6b940d67c9d6c35cffc3c6070c7a33ee5d231336a231b9098e1440c30714fcdadf7031c128c46f881eb16eaa83668a2ba8d8021faf6db9522919de2918a05cdca883c1c3170c77c605e030a89835535378d68207e24b6b1565e9ef3196d5415" + "uplink": "01010000023ea3616259022fa361645902180000000000000001eb023948c7de3836d2d2b54f2c07a885cbf5571e6f49c475b2b8975c6f2a04f5820a0e2cfacb4c98391c88b5a96539d611340e69050bfe0276bb5f3d634457c8ea82f347837fd85aa7b64ec223a8c818c0207b1eb79173cf86cb3133e939f19d6188e9d318dbc6b63826379fc08f27e9aa3636abca17bba3f11c260b281884f51d45d1290e97048cdc7683d1ff29da30d8e20804bab193c80f9a67d15cf637d228964d13ea89e471a00c085aa8d2437bdb38755a51fecbd26624e3130161b38a85223ae40bf8125dd06fab7bd341529572aa364b778a5c976c281e32214252fd288463ba2ee566f07b24b8a49039c0485f51244f1d1f18e950a077f55a0ddeca83c9dbfdaf894d2f7f2dde4c9a89efc46a4021b9139d854dd9a36782a810a92840d962d1e711e1e1044e24b4bebed02649922254ba96bb4f99d89fd161f11e0a8554109dc03b747f11a8a92149958d9c8be127de0f4e255a3244d0633db10df472500d710d4db65f86f928efa73dea0d23a133682e5fca385ad37924de2cfc772c04474e3b4c0b7d230fec5e25019f3e341379315e0a4a30c78c40af81ae932f76746f2bdf7a39b1aa2ad098679c3626e6b940d67c9d6c35cffc3c6070c7a33ee5d231336a231b9098e1440c30714fcdadf7031c128c46f881eb16eaa83668a2ba8d8021faf6db9522919de2918a05cdca883c1c3170c77c605e030a89835535378d68207e24b6b1565e9ef3196d541562696403636465766857476d7639464255617461786373696740" }, { "name": "now.answer", @@ -64,7 +64,7 @@ "01000000001ca26174647469636b6162a2637365710068757074696d654d7319f23000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ], "body": "0000000000000002eb18033457557e46323af76b55c1075142e8a82f1df52f9317ff6bda3dca6c75ff401c1b178f915736e4afc88fc70fbf6f26f0f6288d3ef9675299cf5852ea6fe489fece2e55a278c996f4371fa056c290d95d3f72a5ed925e85f957a8461f1670ce1ca74a8d17e770ad950adfa84011f7a25e23bcaa0fcb088bdfc37014476d2ba18e7b06b751992ceb59b692ee1b1170f09466936bda632cfac87bc422bd6e0faf9bb41c40241a4d6cb52c78eec556d32d08fddb0262dee192aadedd0647963ae88c344838d393200865724ff5cf704ff238bee8beb2c2476f49166765eccc354d85cccf28d2b1ee66fdd5863f2f85762706fb335245b3f9b447734b1f52e40a76448c33e03c189979c7f8a9849da73526bcbb0801a9024531000f51424202ec778ee70316909d6bb9df79187ce1201434b90808cb01a4c8f9ca94a7a8a4fde75dc931fb429c69ffa09c52f554bd84ad09559fa4a650e8267b2b99cdd464d30ad6a9b1bce6d678ce4f964045bd35fb66e245d9ec3c4f68bbae38d9d96a077bc6b9f0e1555bf89d0e60b929258083316d45d5e61867c11a7b8a6c9ff2ccc4ef0faac3d946d78157b8757eca309e20f96bfcddd8c424380599357885bb9873c65972953bf46c3ef48277ac25ed7d8d6689a076ebd812678a30644eb29b134b3dab441d104e8e08b13226bc4c57fb8a633fc7306339707eaa1bb2c7740c19a5e236c70c8d8c51aa8cbe3f6eea6597a29b000000000000000390a50f3904d67213ca2b7434b382230d68f7562fbfe5e283b2548559239aeea7e2b91e7d63bfe8e628b7b53d134a83e321386dd0c4034c8d77cd4a675cb28477d5b27ff62135c8b8e1964b002f083ede920a3f23a8244eb98f1869ce01916fb848529b614d3f6b9f74d46462486cc79b9410369dd91c0b3add128f848f94f4effbb0fd035da1e780c595c8375aa26f6305fccdd25241b1b05a44b56044db443ce2e5100b19ab29e020a1b3c4a18463aaeabd9c9f42ae682bca5b4df1e739fc61ce9ad49a51b039e9abf2fb4f74c68ffce90325e51568f44bf7fd634abef35376f416334cc64547371b0298a8a1dbebf4d8291265e3260a4e94f84c8779df418c4c9dda8d6e5d0ffabbe4506e4732723f8a463b56fca8aeca1b4bf20cf831e88c309729d49bdbfd290090ea55caced3deb8805994e3e90440b6d79066ec00a337ff9e02cde9f7520faaf27824de79f1177c301ff7bfc41bdfbcfaa9f93ccfd2e2298f5023fe5176209c785cf6a397bb5d2531c50f84bcfe1cba89633df4a6da8658b76a10f3f5864c2f4875260468754d801f48d24b693a8178fe4c1acc4f13240db5393d6c34a9aa0b2730c48c248fa6f0dc440997f2d37ce4fdfd234350f8f58e3219572f92bcd828ae6d3c27b90644669d2268fe865d818e9dd811b9542a9e9ae553070bad06f63b2a8a54f3905e8207f2aafe899659ce966cc4d0ffb47ebd94af6202855672789b1373931964d8d4", - "relay": "8182838485868788898a8b8c8d8e8f900000000000000002eb18033457557e46323af76b55c1075142e8a82f1df52f9317ff6bda3dca6c75ff401c1b178f915736e4afc88fc70fbf6f26f0f6288d3ef9675299cf5852ea6fe489fece2e55a278c996f4371fa056c290d95d3f72a5ed925e85f957a8461f1670ce1ca74a8d17e770ad950adfa84011f7a25e23bcaa0fcb088bdfc37014476d2ba18e7b06b751992ceb59b692ee1b1170f09466936bda632cfac87bc422bd6e0faf9bb41c40241a4d6cb52c78eec556d32d08fddb0262dee192aadedd0647963ae88c344838d393200865724ff5cf704ff238bee8beb2c2476f49166765eccc354d85cccf28d2b1ee66fdd5863f2f85762706fb335245b3f9b447734b1f52e40a76448c33e03c189979c7f8a9849da73526bcbb0801a9024531000f51424202ec778ee70316909d6bb9df79187ce1201434b90808cb01a4c8f9ca94a7a8a4fde75dc931fb429c69ffa09c52f554bd84ad09559fa4a650e8267b2b99cdd464d30ad6a9b1bce6d678ce4f964045bd35fb66e245d9ec3c4f68bbae38d9d96a077bc6b9f0e1555bf89d0e60b929258083316d45d5e61867c11a7b8a6c9ff2ccc4ef0faac3d946d78157b8757eca309e20f96bfcddd8c424380599357885bb9873c65972953bf46c3ef48277ac25ed7d8d6689a076ebd812678a30644eb29b134b3dab441d104e8e08b13226bc4c57fb8a633fc7306339707eaa1bb2c7740c19a5e236c70c8d8c51aa8cbe3f6eea6597a29b000000000000000390a50f3904d67213ca2b7434b382230d68f7562fbfe5e283b2548559239aeea7e2b91e7d63bfe8e628b7b53d134a83e321386dd0c4034c8d77cd4a675cb28477d5b27ff62135c8b8e1964b002f083ede920a3f23a8244eb98f1869ce01916fb848529b614d3f6b9f74d46462486cc79b9410369dd91c0b3add128f848f94f4effbb0fd035da1e780c595c8375aa26f6305fccdd25241b1b05a44b56044db443ce2e5100b19ab29e020a1b3c4a18463aaeabd9c9f42ae682bca5b4df1e739fc61ce9ad49a51b039e9abf2fb4f74c68ffce90325e51568f44bf7fd634abef35376f416334cc64547371b0298a8a1dbebf4d8291265e3260a4e94f84c8779df418c4c9dda8d6e5d0ffabbe4506e4732723f8a463b56fca8aeca1b4bf20cf831e88c309729d49bdbfd290090ea55caced3deb8805994e3e90440b6d79066ec00a337ff9e02cde9f7520faaf27824de79f1177c301ff7bfc41bdfbcfaa9f93ccfd2e2298f5023fe5176209c785cf6a397bb5d2531c50f84bcfe1cba89633df4a6da8658b76a10f3f5864c2f4875260468754d801f48d24b693a8178fe4c1acc4f13240db5393d6c34a9aa0b2730c48c248fa6f0dc440997f2d37ce4fdfd234350f8f58e3219572f92bcd828ae6d3c27b90644669d2268fe865d818e9dd811b9542a9e9ae553070bad06f63b2a8a54f3905e8207f2aafe899659ce966cc4d0ffb47ebd94af6202855672789b1373931964d8d4" + "uplink": "010100000456a36162590447a361645904300000000000000002eb18033457557e46323af76b55c1075142e8a82f1df52f9317ff6bda3dca6c75ff401c1b178f915736e4afc88fc70fbf6f26f0f6288d3ef9675299cf5852ea6fe489fece2e55a278c996f4371fa056c290d95d3f72a5ed925e85f957a8461f1670ce1ca74a8d17e770ad950adfa84011f7a25e23bcaa0fcb088bdfc37014476d2ba18e7b06b751992ceb59b692ee1b1170f09466936bda632cfac87bc422bd6e0faf9bb41c40241a4d6cb52c78eec556d32d08fddb0262dee192aadedd0647963ae88c344838d393200865724ff5cf704ff238bee8beb2c2476f49166765eccc354d85cccf28d2b1ee66fdd5863f2f85762706fb335245b3f9b447734b1f52e40a76448c33e03c189979c7f8a9849da73526bcbb0801a9024531000f51424202ec778ee70316909d6bb9df79187ce1201434b90808cb01a4c8f9ca94a7a8a4fde75dc931fb429c69ffa09c52f554bd84ad09559fa4a650e8267b2b99cdd464d30ad6a9b1bce6d678ce4f964045bd35fb66e245d9ec3c4f68bbae38d9d96a077bc6b9f0e1555bf89d0e60b929258083316d45d5e61867c11a7b8a6c9ff2ccc4ef0faac3d946d78157b8757eca309e20f96bfcddd8c424380599357885bb9873c65972953bf46c3ef48277ac25ed7d8d6689a076ebd812678a30644eb29b134b3dab441d104e8e08b13226bc4c57fb8a633fc7306339707eaa1bb2c7740c19a5e236c70c8d8c51aa8cbe3f6eea6597a29b000000000000000390a50f3904d67213ca2b7434b382230d68f7562fbfe5e283b2548559239aeea7e2b91e7d63bfe8e628b7b53d134a83e321386dd0c4034c8d77cd4a675cb28477d5b27ff62135c8b8e1964b002f083ede920a3f23a8244eb98f1869ce01916fb848529b614d3f6b9f74d46462486cc79b9410369dd91c0b3add128f848f94f4effbb0fd035da1e780c595c8375aa26f6305fccdd25241b1b05a44b56044db443ce2e5100b19ab29e020a1b3c4a18463aaeabd9c9f42ae682bca5b4df1e739fc61ce9ad49a51b039e9abf2fb4f74c68ffce90325e51568f44bf7fd634abef35376f416334cc64547371b0298a8a1dbebf4d8291265e3260a4e94f84c8779df418c4c9dda8d6e5d0ffabbe4506e4732723f8a463b56fca8aeca1b4bf20cf831e88c309729d49bdbfd290090ea55caced3deb8805994e3e90440b6d79066ec00a337ff9e02cde9f7520faaf27824de79f1177c301ff7bfc41bdfbcfaa9f93ccfd2e2298f5023fe5176209c785cf6a397bb5d2531c50f84bcfe1cba89633df4a6da8658b76a10f3f5864c2f4875260468754d801f48d24b693a8178fe4c1acc4f13240db5393d6c34a9aa0b2730c48c248fa6f0dc440997f2d37ce4fdfd234350f8f58e3219572f92bcd828ae6d3c27b90644669d2268fe865d818e9dd811b9542a9e9ae553070bad06f63b2a8a54f3905e8207f2aafe899659ce966cc4d0ffb47ebd94af6202855672789b1373931964d8d462696403636465766857476d7639464255617461786373696740" } ] } diff --git a/go/internal/appuplink/vectors_test.go b/go/internal/appuplink/vectors_test.go index 9905f4070..d93bc53ca 100644 --- a/go/internal/appuplink/vectors_test.go +++ b/go/internal/appuplink/vectors_test.go @@ -10,15 +10,17 @@ import ( "github.com/srcfl/ftw/go/internal/appproto" "github.com/srcfl/ftw/go/protocol/appwire" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) // The exchange's golden vectors (ADR 0009, "Vectors"). // // testdata/exchange-vectors.json holds the bytes of every step of one watch -// session, from fixed keys: X1 with a pairing code, a command answered by its -// ack and result, and a read answered by now and a tick. The Swift client in -// srcfl/ftw-app replays the same file, so Go and Swift agree on every byte or -// one of them fails. Regenerate with +// session, from fixed keys: X1 for a watch its phone added by its key, a +// command answered by its ack and result, and a read answered by now and a +// tick, each also as the x that carries it on the uplink to FTW Cloud. The +// Swift client in srcfl/ftw-app replays the same file, so Go and Swift agree +// on every byte or one of them fails. Regenerate with // // go test ./internal/appuplink -run TestExchangeVectors -update-exchange-vectors @@ -40,16 +42,16 @@ type exchangeVectors struct { Steps []exchangeVectorStep `json:"steps"` } -// exchangeVectorStep is one message across the relay. Frames are the -// plaintext lane frames inside it; Body is what follows the relay's id, the -// bytes the watch posts or the answer it receives; Relay is id ‖ Body as it -// crosses the box's exchange socket. +// exchangeVectorStep is one message of the exchange. Frames are the +// plaintext lane frames inside it; Body is the exchange message itself, the d +// of the watch's x {d} in its one-shot request or of the answer it receives; +// Uplink is the frame x {dev, id, d} as it crosses the box's uplink. type exchangeVectorStep struct { Name string `json:"name"` From string `json:"from"` Frames []string `json:"frames"` Body string `json:"body"` - Relay string `json:"relay"` + Uplink string `json:"uplink"` } // fixed is n bytes counting up from first: a key nobody would use, which is @@ -88,7 +90,7 @@ func buildExchangeVectors(t *testing.T) exchangeVectors { } boxStatic, boxEphemeral := keyPair(0x01), keyPair(0x61) watchStatic, watchEphemeral := keyPair(0x21), keyPair(0x41) - id := fixed(0x81, ExchangeIDBytes) + dev := cloudwire.DeviceID(watchStatic.Public()) prologueBytes := exchangePrologue(boxStatic.Public()) watch, err := appwire.NewInitiator(appwire.InitiatorOptions{ @@ -117,19 +119,31 @@ func buildExchangeVectors(t *testing.T) exchangeVectors { WatchEphemeral: hex.EncodeToString(fixed(0x41, appwire.DHBytes)), } step := func(name, from string, frames [][]byte, body []byte) { - s := exchangeVectorStep{Name: name, From: from, Body: hex.EncodeToString(body), - Relay: hex.EncodeToString(append(append([]byte(nil), id...), body...))} + // A request and its answer share the service's number. + x, err := cloudwire.Marshal(cloudwire.X{Dev: dev, ID: uint32(len(v.Steps)/2 + 1), D: body}) + if err != nil { + t.Fatal(err) + } + raw, err := cloudwire.Envelope{T: cloudwire.MsgX, B: x}.Encode() + if err != nil { + t.Fatal(err) + } + uplink, err := cloudwire.EncodeFrame(raw) + if err != nil { + t.Fatal(err) + } + s := exchangeVectorStep{Name: name, From: from, Body: hex.EncodeToString(body), Uplink: hex.EncodeToString(uplink)} for _, f := range frames { s.Frames = append(s.Frames, hex.EncodeToString(f)) } v.Steps = append(v.Steps, s) } - // X1: hello with the pairing code in message 1, hello_ok in message 2. + // X1: hello in message 1, hello_ok in message 2. The phone added the + // watch by its key, so the hello carries no code. hello := frameOf(t, appwire.LaneControl, exchangeFrameBytes, appproto.MsgHello, appproto.Hello{ Proto: appproto.ProtoRange{Min: appproto.ProtoMin, Max: appproto.ProtoMax}, App: appproto.AppInfo{Build: "vectors", UA: "watchos"}, - Code: fixed(0xa1, 16), }) message1, err := watch.WriteMessage(hello) if err != nil { @@ -262,7 +276,7 @@ func TestExchangeVectors(t *testing.T) { "; if the change is meant, regenerate it and update srcfl/ftw-app's copy in the same pair of PRs") } - // The sizes the relay checks and the docs promise. + // The sizes the docs promise, each inside one x on the uplink. var v exchangeVectors if err := json.Unmarshal(want, &v); err != nil { t.Fatal(err) @@ -276,6 +290,9 @@ func TestExchangeVectors(t *testing.T) { if n := len(s.Body) / 2; n != sizes[s.Name] { t.Fatalf("%s is %d bytes, want %d", s.Name, n, sizes[s.Name]) } + if n := len(s.Uplink) / 2; n > cloudwire.MaxXFrame { + t.Fatalf("%s's x is %d bytes, over %d", s.Name, n, cloudwire.MaxXFrame) + } } } diff --git a/go/internal/appuplink/watch_test.go b/go/internal/appuplink/watch_test.go index 7303ad0f2..d9a0c1f5b 100644 --- a/go/internal/appuplink/watch_test.go +++ b/go/internal/appuplink/watch_test.go @@ -4,57 +4,35 @@ import ( "testing" "time" - "github.com/gorilla/websocket" - "github.com/srcfl/ftw/go/internal/apiauth" - "github.com/srcfl/ftw/go/internal/appenroll" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) // A watch reaches the box through the exchange and never through the stream, -// so it can never take a phone's place. Its key on the stream -// gets the silence a stranger gets. +// so it can never take a phone's place. The roster gives it tun, because its +// exchange rides FTW Cloud, but its key in the tunnel gets the silence a +// stranger gets. func TestAWatchKeyGetsNoStream(t *testing.T) { - r := newRig(t, 481234) - - code, _, err := r.enroll.MintPairingCode(apiauth.RoleOwner, appenroll.PairingTTL) - if err != nil { - t.Fatalf("MintPairingCode: %v", err) - } - phone, err := newAppClient() - if err != nil { - t.Fatalf("newAppClient: %v", err) - } - grant, err := r.enroll.Authorise(phone.staticPublic(), code) - if err != nil { - t.Fatalf("pairing the phone: %v", err) - } - watchCode, _, err := r.enroll.MintWatchCode(grant.DeviceID, appenroll.WatchTTL) - if err != nil { - t.Fatalf("MintWatchCode: %v", err) + r := newTunRig(t) + phone := r.pair(t) + grant, ok := r.enroll.GrantFor(cloudwire.DeviceID(phone.staticPublic())) + if !ok { + t.Fatal("the phone did not pair") } watch, err := newAppClient() if err != nil { - t.Fatalf("newAppClient: %v", err) + t.Fatal(err) } - if _, err := r.enroll.AuthoriseWatch(watch.staticPublic(), watchCode); err != nil { + if _, err := r.enroll.AddWatch(grant.DeviceID, watch.staticPublic()); err != nil { t.Fatalf("adding the watch: %v", err) } - conn := r.dialApp(t, 481234) - message1, err := watch.message1(r.enroll.StaticKey().Public(), nil) + m1, err := watch.message1(r.enroll.StaticKey().Public(), nil) if err != nil { - t.Fatalf("message1: %v", err) - } - if err := conn.WriteMessage(websocket.BinaryMessage, message1); err != nil { - t.Fatalf("sending message 1: %v", err) + t.Fatal(err) } - _ = conn.SetReadDeadline(time.Now().Add(750 * time.Millisecond)) - for { - kind, _, err := conn.ReadMessage() - if err != nil { - return // the deadline, which is the pass - } - if kind == websocket.BinaryMessage { - t.Fatal("the box opened a stream for a watch") - } + r.tunnel.Deliver(cloudwire.Tun{Dev: cloudwire.DeviceID(watch.staticPublic()), Sid: 1, D: m1}) + r.silent(t, 750*time.Millisecond) + if r.tunnel.count() != 0 { + t.Fatal("the box opened a stream for a watch") } } diff --git a/go/internal/appuplink/watchclient_test.go b/go/internal/appuplink/watchclient_test.go index 0d08289b0..038af9c5a 100644 --- a/go/internal/appuplink/watchclient_test.go +++ b/go/internal/appuplink/watchclient_test.go @@ -9,6 +9,7 @@ import ( "github.com/srcfl/ftw/go/internal/appproto" "github.com/srcfl/ftw/go/internal/loadpoint" "github.com/srcfl/ftw/go/protocol/appwire" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) // watchClient is a watch, in Go: the initiator side of the exchange, built on @@ -20,6 +21,9 @@ type watchClient struct { transport *appwire.Transport } +// dev is the device id FTW Cloud names this watch by. +func (w *watchClient) dev() string { return cloudwire.DeviceID(w.static.Public()) } + func newWatchClient() (*watchClient, error) { static, err := appwire.GenerateKeyPair() if err != nil { diff --git a/go/internal/backup/archive.go b/go/internal/backup/archive.go index d71b2d463..d51752584 100644 --- a/go/internal/backup/archive.go +++ b/go/internal/backup/archive.go @@ -105,7 +105,7 @@ type RestoreResult struct { // Inspect performs a full verification and returns stable archive metadata. func Inspect(ctx context.Context, archivePath string) (Manifest, Info, error) { - manifest, err := Verify(archivePath) + manifest, err := verifyInWorkspaceContext(ctx, archivePath, "") if err != nil { return Manifest{}, Info{}, err } @@ -155,6 +155,8 @@ func Create(ctx context.Context, opts CreateOptions) (Info, error) { if err := os.MkdirAll(outputDir, 0o700); err != nil { return Info{}, fmt.Errorf("backup: create output dir: %w", err) } + release := state.HoldBackupScratch(outputDir) + defer release() if opts.Progress != nil { opts.Progress(state.BackupProgress{Phase: "waiting_for_maintenance"}) } @@ -658,6 +660,8 @@ func verifyInWorkspaceProgressContext(ctx context.Context, archivePath, workspac if allowFallback { workspace = filepath.Dir(archivePath) } + release := state.HoldBackupScratch(workspace) + defer release() tmpDir, err := os.MkdirTemp(workspace, ".ftw-backup-verify-") if err != nil && allowFallback { tmpDir, err = os.MkdirTemp("", ".ftw-backup-verify-") diff --git a/go/internal/backup/archive_cancel_test.go b/go/internal/backup/archive_cancel_test.go index 7a791256f..4138d9e7e 100644 --- a/go/internal/backup/archive_cancel_test.go +++ b/go/internal/backup/archive_cancel_test.go @@ -3,6 +3,7 @@ package backup import ( "context" "errors" + "os" "path/filepath" "sync" "testing" @@ -11,6 +12,21 @@ import ( "github.com/srcfl/ftw/go/internal/state" ) +func TestInspectCanceledContextDoesNotParseArchive(t *testing.T) { + archive := filepath.Join(t.TempDir(), "interrupted.ftwbak") + // A canceled inspection must stop before reading even an incomplete + // archive. Otherwise its full verification runs without the caller. + if err := os.WriteFile(archive, []byte("incomplete archive"), 0o600); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, _, err := Inspect(ctx, archive) + if !errors.Is(err, context.Canceled) { + t.Fatalf("canceled inspection parsed the archive: %v", err) + } +} + func TestBackupMutexWaitCanBeCancelled(t *testing.T) { root := t.TempDir() path := filepath.Join(root, "state.db") diff --git a/go/internal/backup/scratch.go b/go/internal/backup/scratch.go new file mode 100644 index 000000000..255af5b10 --- /dev/null +++ b/go/internal/backup/scratch.go @@ -0,0 +1,25 @@ +package backup + +import ( + "io/fs" + "strings" + + "github.com/srcfl/ftw/go/internal/state" +) + +// RemoveStaleScratch removes what stopped backups left in outputDir and in +// /dev/shm. Core calls it at start, before it accepts a backup request. +func RemoveStaleScratch(outputDir string) { + state.RemoveStaleBackupScratchInShm() + state.RemoveStaleBackupScratch(outputDir, isScratch) +} + +// isScratch matches the stage and verification folders and the unpublished +// archive that Create and verification keep beside the backups. +func isScratch(entry fs.DirEntry) bool { + name := entry.Name() + if entry.IsDir() { + return state.IsTempName(name, ".ftw-backup-stage-") || state.IsTempName(name, ".ftw-backup-verify-") + } + return entry.Type().IsRegular() && strings.HasPrefix(name, ".ftw-full-backup-") && strings.HasSuffix(name, ".ftwbak.tmp") +} diff --git a/go/internal/backup/scratch_test.go b/go/internal/backup/scratch_test.go new file mode 100644 index 000000000..fe6c2ab43 --- /dev/null +++ b/go/internal/backup/scratch_test.go @@ -0,0 +1,96 @@ +//go:build unix + +package backup + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/srcfl/ftw/go/internal/state" +) + +func TestStaleScratchIsRemovedAndArchivesKept(t *testing.T) { + out := t.TempDir() + writeTestFile(t, filepath.Join(out, ".ftw-backup-stage-123", "database.gz.raw.tmp"), strings.Repeat("r", 3000)) + writeTestFile(t, filepath.Join(out, ".ftw-backup-verify-456", "database.gz"), strings.Repeat("v", 1000)) + writeTestFile(t, filepath.Join(out, ".ftw-full-backup-20261013T103000.000Z.ftwbak.tmp"), strings.Repeat("a", 2000)) + keep := []string{ + "ftw-full-backup-20261010T020000.000Z.ftwbak", + "ftw-full-backup-20261010T020000.000Z.ftwbak.verified.json", + "ftw-full-backup-20261010T020000.000Z.ftwbak.verified.json.tmp", + ".ftw-backup-stage-x/database.gz", + ".ftw-pre-restore-20261010T020000Z/state.db", + "notes.txt", + } + for _, name := range keep { + writeTestFile(t, filepath.Join(out, name), "keep") + } + + removed, bytes := state.RemoveStaleBackupScratch(out, isScratch) + if removed != 3 || bytes != 6000 { + t.Fatalf("removed %d entries, %d bytes; want 3, 6000", removed, bytes) + } + for _, name := range []string{".ftw-backup-stage-123", ".ftw-backup-verify-456", ".ftw-full-backup-20261013T103000.000Z.ftwbak.tmp"} { + if _, err := os.Stat(filepath.Join(out, name)); !os.IsNotExist(err) { + t.Fatalf("%s left behind: %v", name, err) + } + } + for _, name := range keep { + if _, err := os.Stat(filepath.Join(out, name)); err != nil { + t.Fatalf("removed %s: %v", name, err) + } + } +} + +func TestRunningBackupKeepsItsScratch(t *testing.T) { + root := t.TempDir() + dataDir := filepath.Join(root, "data") + if err := os.MkdirAll(dataDir, 0o700); err != nil { + t.Fatal(err) + } + statePath := filepath.Join(dataDir, "state.db") + st, err := state.Open(statePath) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + out := filepath.Join(root, "backups") + seen := map[string]bool{} + // A Core starting beside this backup sweeps at every step of it. + sweep := func(state.BackupProgress) { + if removed, _ := state.RemoveStaleBackupScratch(out, isScratch); removed != 0 { + t.Errorf("removed %d entries of a running backup", removed) + } + entries, _ := os.ReadDir(out) + for _, entry := range entries { + if isScratch(entry) { + seen[strings.TrimRight(entry.Name(), "0123456789")] = true + } + } + } + info, err := Create(context.Background(), CreateOptions{ + State: st, StatePath: statePath, DataDir: dataDir, OutputDir: out, Progress: sweep, + }) + if err != nil { + t.Fatal(err) + } + tmp := false + for name := range seen { + tmp = tmp || strings.HasSuffix(name, ".ftwbak.tmp") + } + if !seen[".ftw-backup-stage-"] || !tmp { + t.Fatalf("no sweep ran while the stage folder and archive temp existed; saw %v", seen) + } + + // Verifying a finished backup, as POST /api/backups/{id}/verify does. + clear(seen) + if _, err := verifyInWorkspaceProgressContext(context.Background(), info.Path, "", sweep); err != nil { + t.Fatal(err) + } + if !seen[".ftw-backup-verify-"] { + t.Fatalf("no sweep ran while the verification folder existed; saw %v", seen) + } +} diff --git a/go/internal/cloudlink/choice.go b/go/internal/cloudlink/choice.go index 776a265a7..4137df789 100644 --- a/go/internal/cloudlink/choice.go +++ b/go/internal/cloudlink/choice.go @@ -2,11 +2,17 @@ package cloudlink import ( "errors" + "slices" + "github.com/srcfl/ftw/go/protocol" "github.com/srcfl/ftw/go/protocol/cloudwire" ) -var ErrCloudNeedsStatistics = errors.New("cloudlink: statistics are part of FTW Cloud; turn FTW Cloud off to stop contributing") +var ErrCloudNeedsStatistics = errors.New("cloudlink: statistics are part of FTW Cloud; turn FTW Cloud off, then statistics, to stop contributing") + +// ErrCloudTermsFirst is statistics asked for while Cloud runs under older +// terms, which did not include them. +var ErrCloudTermsFirst = errors.New("cloudlink: accept the current FTW Cloud terms first; they include statistics") // includeStatistics records the exact scope the owner read. Service approval // still gates every report. The caller holds mu. @@ -34,36 +40,6 @@ func (l *Link) enableLinkChoice() error { return nil } -// startStatisticsOnly reuses the signed uplink, without publishing any phone, -// pairing code, Noise key, live reading or grant for remote access. -func (l *Link) startStatisticsOnly(req StatsRequest) error { - if err := l.linkLocked(); err != nil { - return err - } - if l.link.Ending { - return ErrLinkEnding - } - id, err := randomBytes(cloudwire.GrantIDBytes) - if err != nil { - return err - } - key, err := randomBytes(cloudwire.GrantKeyBytes) - if err != nil { - return err - } - g := grantRecord{ID: id, GrantKey: key, KeyFp: l.link.KeyFp, Policy: req.Policy, By: req.By, Controller: cloudwire.ControllerSourceful, Scopes: []string{}, Purposes: []string{protocol.CloudPurposeStats}} - if l.clockSynced() { - g.AtMs = l.opts.Now().UnixMilli() - } - if err := l.includeStatistics(&g, true); err != nil { - return err - } - if err := l.enableLinkChoice(); err != nil { - return err - } - return l.storeLocked(g, "local statistics accepted") -} - // End stops all Cloud functions, purges the home and remembers the choice so // a paired phone or a restart cannot silently turn the link back on. func (l *Link) End(by string) error { @@ -103,17 +79,126 @@ func (l *Link) End(by string) error { return nil } -// StatisticsChoice is the saved answer for both project reports. The daily -// report must not turn itself on before the owner answers the new question. -func StatisticsChoice(kv KV) bool { +// SharesStatistics is the one statistics answer, for both anonymous daily +// reports: the fleet report and the energy report (Fredrik, 24 October 2026). +// With FTW Cloud they are part of Cloud, with no separate off switch. A Cloud +// grant under older terms sends nothing until the owner accepts the current +// ones. Without Cloud they are on unless the owner turned them off. +// +// This form is for a box whose Cloud link is not running, so no stored grant +// is honoured: the local answer alone. Link.SharesStatistics counts the grant +// the running link honours, which it does only with the key it was made with. +func SharesStatistics(kv KV) bool { if kv == nil { return false } - if disabled, _ := kv.LoadConfig(disabledKey); disabled == "true" { + share, _ := kv.LoadConfig(statsShareKey) + return sharesStatistics(nil, share) +} + +// SaidYesToStatistics reports an active yes, for a box whose Cloud link is +// not running: a local yes. A box that shares only by default has not given +// one, so its reports reach FTW Cloud only through the relay, which hides its +// address; nothing goes straight to FTW Cloud from it. +func SaidYesToStatistics(kv KV) bool { + if kv == nil { return false } + share, _ := kv.LoadConfig(statsShareKey) + return saidYesToStatistics(nil, share) +} + +// SharesStatistics is the one answer with the grant this link honours. +func (l *Link) SharesStatistics() bool { + g, share := l.statisticsAnswer() + return sharesStatistics(g, share) +} + +// SaidYesToStatistics is an active yes with the grant this link honours: +// Cloud under the current terms, or a local yes. +func (l *Link) SaidYesToStatistics() bool { + g, share := l.statisticsAnswer() + return saidYesToStatistics(g, share) +} + +// statisticsAnswer is the grant this link honours, nil once Cloud is turned +// off, and the stored local answer. +func (l *Link) statisticsAnswer() (*grantRecord, string) { + l.mu.Lock() + g := l.grant + if l.disabled { + g = nil + } + l.mu.Unlock() + share, _ := l.opts.KV.LoadConfig(statsShareKey) + return g, share +} + +func sharesStatistics(g *grantRecord, share string) bool { + if g != nil && !g.Revoked && !g.statisticsOnly() { + return g.Policy == cloudwire.CloudTermsPolicy + } + return share == "on" || share == "default" +} + +func saidYesToStatistics(g *grantRecord, share string) bool { + if g != nil && !g.Revoked && !g.statisticsOnly() { + return g.Policy == cloudwire.CloudTermsPolicy + } + return share == "on" +} + +// cloudUnderOlderTerms is Cloud on under terms older than the current ones, +// which did not include statistics: nothing goes until the owner accepts. +func cloudUnderOlderTerms(g *grantRecord) bool { + return g != nil && !g.Revoked && !g.statisticsOnly() && g.Policy != cloudwire.CloudTermsPolicy +} + +// ReconcileStatistics makes the grant follow the one answer once it is off: +// the stats purpose and statsId go, and a statistics-only link ends, as +// turning statistics off does. Core calls it after MigrateStatistics, so an +// earlier fleet switch set to off also stops the hourly upload it never +// covered. +func (l *Link) ReconcileStatistics(by string) error { + g, share := l.statisticsAnswer() + if share != "off" || g == nil || g.Revoked || cloudRequiresStatistics(g) || + (len(g.StatsID) == 0 && !slices.Contains(g.Purposes, protocol.CloudPurposeStats)) { + return nil + } + return l.StopStats(by) +} + +// cloudRequiresStatistics reports whether the grant is Cloud under the +// current terms, where statistics have no separate off switch. +func cloudRequiresStatistics(g *grantRecord) bool { + return g != nil && !g.Revoked && g.Policy == cloudwire.CloudTermsPolicy +} + +// MigrateStatistics writes the one answer once, from what an earlier release +// stored. A local No, a fleet report switched off, or FTW Cloud turned off +// (whose screen said that stops statistics) stays off. A local yes stays yes. +// Every other box is on by default: a box without Cloud shares unless its +// owner turns it off. Until this has run, SharesStatistics reads a box +// without Cloud as off. +func MigrateStatistics(kv KV, fleetReportOff bool) error { + if v, ok := kv.LoadConfig(statsShareKey); ok && (v == "on" || v == "off" || v == "default") { + return nil + } g, err := loadJSON[grantRecord](kv, grantKey) - return err == nil && g != nil && !g.Revoked && g.StatsPolicy == cloudwire.StatsPolicy && len(g.StatsID) == 16 + if err != nil { + return err + } + decision, _ := kv.LoadConfig(statsDecisionKey) + disabled, _ := kv.LoadConfig(disabledKey) + answeredYes := g != nil && !g.Revoked && g.StatsPolicy == cloudwire.StatsPolicy && len(g.StatsID) == 16 + share := "default" + switch { + case fleetReportOff || disabled == "true" || (decision != "" && !answeredYes): + share = "off" + case answeredYes && g.statisticsOnly(): + share = "on" + } + return kv.SaveConfig(statsShareKey, share) } // An unset clock cannot place a choice in UTC. Start its upload window only diff --git a/go/internal/cloudlink/choice_test.go b/go/internal/cloudlink/choice_test.go index be516aa8b..1fae0a3b3 100644 --- a/go/internal/cloudlink/choice_test.go +++ b/go/internal/cloudlink/choice_test.go @@ -24,44 +24,34 @@ func TestCloudTermsIncludeStatisticsAndCannotOptOutSeparately(t *testing.T) { if err := h.link.StopStats("owner"); !errors.Is(err, ErrCloudNeedsStatistics) { t.Fatal("Cloud could opt out independently", err) } - if !h.link.Status().CloudTermsAccepted || !h.link.Status().StatisticsActive { + if s := h.link.Status(); !s.CloudTermsAccepted || !s.StatisticsActive || !s.Statistics || !s.StatisticsRequired { t.Fatal("new terms not active") } } -func TestLocalStatisticsPublishesNoRemoteAccess(t *testing.T) { +// Without Cloud, statistics are an answer on the box. Turning them on or off +// links nothing to Cloud and publishes no remote access. +func TestLocalStatisticsLinkNothingToCloud(t *testing.T) { svc := newFakeService(t) h := newHarness(t, svc) + if s := h.link.Status(); s.StatisticsDecisionRequired || s.Statistics || s.StatisticsRequired { + t.Fatal("a box asked, or shared before its answer was recorded", s) + } if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy, By: "local owner"}); err != nil { t.Fatal(err) } - r := h.nextRoster(t) - if len(r.Devices) != 0 || len(r.Pending) != 0 || len(r.Noise) != 0 || len(r.WebPush) != 0 || len(r.Grant.Scopes) != 0 || len(r.Grant.Purposes) != 1 || r.Grant.Purposes[0] != protocol.CloudPurposeStats { - t.Fatal("statistics-only exposed Cloud access", r) + s := h.link.Status() + if !s.Statistics || s.StatisticsRequired || s.Link != "off" || s.StatisticsOnly || !SharesStatistics(h.kv) { + t.Fatal("local statistics are not a local answer", s) } - if h.link.TakesTun() || h.link.Home() != "" || h.link.SessionInfo() != nil || h.link.current().copyOn() { + if h.link.Home() != "" || h.link.SessionInfo() != nil || h.link.current().copyOn() || h.link.current().g != nil { t.Fatal("local statistics enabled Cloud functions") } - if err := h.link.SendTun(cloudwire.Tun{}, false); !errors.Is(err, ErrNoUplink) { - t.Fatal("statistics-only queued a tunnel", err) - } - if !h.link.Status().StatisticsOnly || h.link.Status().State != "statistics" { - t.Fatal("local statistics labelled Cloud") - } -} - -func TestDecliningLocalStatisticsRecordsTheAnswerWithoutCreatingALink(t *testing.T) { - svc := newFakeService(t) - h := newHarness(t, svc) - if !h.link.Status().StatisticsDecisionRequired { - t.Fatal("an unanswered choice looked accepted") - } if err := h.link.StopStats("local owner"); err != nil { t.Fatal(err) } - s := h.link.Status() - if s.StatisticsDecisionRequired || s.Link != "off" || s.StatisticsActive { - t.Fatal("decline did not stay local", s) + if s := h.link.Status(); s.Statistics || s.Link != "off" || s.StatisticsActive || SharesStatistics(h.kv) { + t.Fatal("off did not stay off and local", s) } } @@ -83,31 +73,70 @@ func TestEndingCloudPreventsAutomaticRelinkFromPairedPhones(t *testing.T) { } } -func TestDailyReportRequiresCurrentAnswerAndHonoursDurableOff(t *testing.T) { - kv := &memKV{} - if StatisticsChoice(kv) { - t.Fatal("unanswered box may send") - } - g := grantRecord{StatsPolicy: "ftw-project-stats-2026-10-v1", StatsID: make([]byte, 16)} - if err := saveJSON(kv, grantKey, g); err != nil { - t.Fatal(err) - } - if StatisticsChoice(kv) { - t.Fatal("old choice may send new report") - } - g.StatsPolicy = cloudwire.StatsPolicy - if err := saveJSON(kv, grantKey, g); err != nil { - t.Fatal(err) - } - if !StatisticsChoice(kv) { - t.Fatal("current answer may not send") +// Every earlier No stays no; every other box shares, since a box without +// Cloud shares until its owner turns it off. The answer is written once. +func TestMigrationKeepsEveryEarlierNo(t *testing.T) { + yes := grantRecord{StatsPolicy: cloudwire.StatsPolicy, StatsID: make([]byte, 16), Purposes: []string{protocol.CloudPurposeStats}} + for name, c := range map[string]struct { + kv map[string]string + grant *grantRecord + fleetOff bool + want bool + }{ + "never asked": {want: true}, + "local yes": {grant: &yes, want: true}, + "local no": {kv: map[string]string{statsDecisionKey: cloudwire.StatsPolicy}, want: false}, + "no to an older policy": {kv: map[string]string{statsDecisionKey: "ftw-project-stats-2026-10-v1"}, want: false}, + "Cloud turned off": {kv: map[string]string{disabledKey: "true"}, want: false}, + "fleet report off": {fleetOff: true, want: false}, + "yes, fleet report off": {grant: &yes, fleetOff: true, want: false}, + "already answered: on": {kv: map[string]string{statsShareKey: "on", disabledKey: "true"}, want: true}, + "already on by default": {kv: map[string]string{statsShareKey: "default"}, want: true}, + "already answered: off": {kv: map[string]string{statsShareKey: "off"}, want: false}, + } { + kv := &memKV{m: map[string]string{}} + for k, v := range c.kv { + kv.m[k] = v + } + if c.grant != nil { + if err := saveJSON(kv, grantKey, *c.grant); err != nil { + t.Fatal(err) + } + } + if SharesStatistics(kv) && c.kv[statsShareKey] == "" { + t.Fatalf("%s: shared before the answer was written", name) + } + if err := MigrateStatistics(kv, c.fleetOff); err != nil { + t.Fatal(err) + } + if got := SharesStatistics(kv); got != c.want { + t.Fatalf("%s: shares %v, want %v", name, got, c.want) + } + // Only a yes that was given counts as one. + if said := SaidYesToStatistics(kv); said != (c.want && (c.grant != nil || c.kv[statsShareKey] == "on")) { + t.Fatalf("%s: said yes %v", name, said) + } + // Written once: a later start reads the answer, not the old flags. + if err := MigrateStatistics(kv, !c.fleetOff); err != nil || SharesStatistics(kv) != c.want { + t.Fatalf("%s: a second start changed the answer", name) + } } - // This flag is saved first, even if writing a later purge record fails. - if err := kv.SaveConfig(disabledKey, "true"); err != nil { +} + +// Without a running link no stored grant counts, so the box's own answer +// alone decides. The link's answer, with the grant it honours, is tested in +// link_test.go. +func TestWithoutALinkOnlyTheLocalAnswerCounts(t *testing.T) { + kv := &memKV{m: map[string]string{statsShareKey: "off"}} + cloud := grantRecord{Policy: cloudwire.CloudTermsPolicy, Purposes: []string{protocol.CloudPurposeServe, protocol.CloudPurposeStats}} + if err := saveJSON(kv, grantKey, cloud); err != nil { t.Fatal(err) } - if StatisticsChoice(kv) { - t.Fatal("durable Cloud off did not stop daily report") + for share, want := range map[string][2]bool{"off": {false, false}, "default": {true, false}, "on": {true, true}} { + kv.m[statsShareKey] = share + if got := [2]bool{SharesStatistics(kv), SaidYesToStatistics(kv)}; got != want { + t.Fatalf("%s: shares, said yes = %v, want %v", share, got, want) + } } } @@ -166,7 +195,7 @@ func TestWholeCloudOffStopsImmediatelyWhenLaterStorageFails(t *testing.T) { if err := h.link.End("owner"); err == nil { t.Fatal("storage failure was not returned") } - if h.link.TakesTun() || h.link.Home() != "" || h.link.SessionInfo() != nil || h.link.current().copyOn() || StatisticsChoice(h.kv) { + if h.link.TakesTun() || h.link.Home() != "" || h.link.SessionInfo() != nil || h.link.current().copyOn() || SharesStatistics(h.kv) { t.Fatal("a failed follow-up write left Cloud sharing active") } if disabled, _ := h.kv.LoadConfig(disabledKey); disabled != "true" { diff --git a/go/internal/cloudlink/conn.go b/go/internal/cloudlink/conn.go index c31f29d82..d581371cc 100644 --- a/go/internal/cloudlink/conn.go +++ b/go/internal/cloudlink/conn.go @@ -300,6 +300,7 @@ func (l *Link) serve(ctx context.Context, conn uplinkConn, snap grantSnapshot) e // so revokes made since then still arrive. l.bumpRoster(ok.RosterSeq) } + w.have = ok.Have tunOut, tunBulk := make(chan []byte, tunQueue), make(chan []byte, tunBulkQueue) l.mu.Lock() @@ -412,10 +413,11 @@ func parseDownlink(kind int, msg []byte) (cloudwire.Envelope, error) { } // read is the downlink. After the handshake the service may send watch, -// resync, purged and tun; challenge and hello_ok a second time are refused +// resync, purged, tun and x; challenge and hello_ok a second time are refused // like any stranger. watch and resync go to the sender, which owns the // cadence. A purge receipt is taken here, before the service hangs up, so the -// box never misses one it was sent. A tun goes to the Tunnel unread. +// box never misses one it was sent. A tun goes to the Tunnel and an x to the +// Exchange, unread. func (l *Link) read(conn uplinkConn, snap grantSnapshot, events chan<- cloudwire.Envelope) error { _ = conn.SetReadDeadline(time.Now().Add(readTimeout)) conn.SetPingHandler(func(data string) error { @@ -441,6 +443,14 @@ func (l *Link) read(conn uplinkConn, snap grantSnapshot, events chan<- cloudwire if l.TakesTun() { l.opts.Tunnel.Deliver(t) } + case cloudwire.MsgX: + var x cloudwire.X + if len(env.Sig) != 0 || cloudwire.Unmarshal(env.B, &x) != nil || len(x.D) > cloudwire.XDataMax || x.Dev == "" { + return fmt.Errorf("%w: an x the box cannot read", errService) + } + if l.opts.Exchange != nil && l.opts.Tunnel != nil { + l.opts.Exchange.Deliver(x) + } case cloudwire.MsgWatch, cloudwire.MsgResync: events <- env default: @@ -454,6 +464,8 @@ type writer struct { conn uplinkConn link *Link mu sync.Mutex + // have is the closed tiles hello_ok said the service holds. + have map[string]string } // send seals body as type t, frames it and writes it. The byte budget is @@ -539,6 +551,64 @@ func (l *Link) SendTun(t cloudwire.Tun, bulk bool) error { } } +// SendX queues a watch's exchange answer for the uplink. It is small and goes +// with the tun lane 0 frames, first and never held, under the tun budget and +// a share of its own (xAllowedLocked). It never blocks. nil means the message +// is queued. +func (l *Link) SendX(x cloudwire.X) error { + frame, err := xFrame(x) + l.mu.Lock() + defer l.mu.Unlock() + switch { + case l.tunOut == nil: + l.tunDropped++ + return ErrNoUplink + case err != nil || !l.xAllowedLocked(): + l.tunDropped++ + return ErrTunDropped + } + select { + case l.tunOut <- frame: + l.xBudget.add(l.uptimeMs(), int64(len(frame))) + return nil + default: + l.tunDropped++ + return ErrTunDropped + } +} + +// XAllowed reports whether a watch's exchange answer may leave now. The +// exchange asks before it runs a request, so a command never acts when its +// answer could not go. +func (l *Link) XAllowed() bool { + l.mu.Lock() + defer l.mu.Unlock() + return l.tunOut != nil && l.xAllowedLocked() +} + +// xAllowedLocked keeps x to xBudgetBytes over a rolling 24 hours, and stops +// it once the tunnel has spent xStopsAt of its own budget, so the phones keep +// the rest for their handshakes, commands and ends. The caller holds l.mu. +func (l *Link) xAllowedLocked() bool { + up := l.uptimeMs() + return l.xBudget.total(up) < xBudgetBytes && l.tunBudget.total(up) < int64(xStopsAt*tunBudgetBytes) +} + +func xFrame(x cloudwire.X) ([]byte, error) { + if len(x.D) > cloudwire.XDataMax { + return nil, ErrTunDropped + } + body, err := cloudwire.Marshal(x) + if err != nil { + return nil, err + } + raw, err := cloudwire.Envelope{T: cloudwire.MsgX, B: body}.Encode() + if err != nil { + return nil, err + } + return cloudwire.EncodeFrame(raw) +} + func tunFrame(t cloudwire.Tun) ([]byte, error) { if len(t.D) > cloudwire.TunDataMax { return nil, ErrTunDropped diff --git a/go/internal/cloudlink/grant.go b/go/internal/cloudlink/grant.go index 625ad0c8b..5a26160c8 100644 --- a/go/internal/cloudlink/grant.go +++ b/go/internal/cloudlink/grant.go @@ -28,6 +28,10 @@ const ( purgedKey = "cloud/purged_v1" disabledKey = "cloud/disabled_v1" statsDecisionKey = "cloud/statistics_decision_v1" + // statsShareKey is the one statistics answer of a box without FTW Cloud: + // "on" (the owner said yes), "default" (on, never answered) or "off" + // (MigrateStatistics, SharesStatistics). + statsShareKey = "statistics/share_v1" ) // linkRecord is the box's link to FTW Cloud: which cloud.key it holds. The diff --git a/go/internal/cloudlink/link.go b/go/internal/cloudlink/link.go index c0b19a624..aad8f1c4b 100644 --- a/go/internal/cloudlink/link.go +++ b/go/internal/cloudlink/link.go @@ -6,11 +6,12 @@ // // FTW Cloud never chooses an action. This package holds that line in its // build: it reaches the box only through the read-only ports below and the -// Tunnel port, and may import nothing from the box (CI checks it with go list -// -deps). What the service may send back is a closed set of six messages. -// Five ask the box for nothing but to send sooner, send again or forget a -// key. The sixth, tun, carries a phone's Noise session, which this package -// hands to the Tunnel unread. +// Tunnel and Exchange ports, and may import nothing from the box (CI checks +// it with go list -deps). What the service may send back is a closed set of +// seven messages. Five ask the box for nothing but to send sooner, send again +// or forget a key. tun carries a phone's Noise session, which this package +// hands to the Tunnel unread, and x a watch's exchange with the box, which it +// hands to the Exchange unread. // // The box links when it first shows a pairing code, has a phone that may // open a tunnel, or an owner grants the readable copy. A link record or a @@ -210,6 +211,15 @@ type Tunnel interface { Drop() } +// Exchange answers the watches' requests that FTW Cloud carries (ADR 0009). +// The box's exchange table implements it; the link hands it each x from the +// service unread. +type Exchange interface { + // Deliver takes one x from the service. It must not block: the link + // reads the service on the same goroutine. + Deliver(x cloudwire.X) +} + // WebPushSource renders and encrypts the notices the service may send when the box is away. type WebPushSource interface { WebPushRows() ([]cloudwire.WebPushRow, error) @@ -234,6 +244,9 @@ type Options struct { // Tunnel serves the phones' sessions. Nil takes no tun: the box does not // list tun in hello.caps, and the service sends it none. Tunnel Tunnel + // Exchange answers the watches. Nil, or no Tunnel, takes no x: the box + // does not list x in hello.caps, and the service sends it none. + Exchange Exchange // Arrivals hears each finished driver poll (telemetry.Store.Arrivals). // While a phone watches, live goes when a poll lands rather than on the // next check. Nil leaves live on its checks alone. @@ -296,9 +309,12 @@ type Link struct { // tunOut and tunBulk carry framed tun messages, lane 0 and bulk, to // the connection's tun writer. Nil while no uplink is up. tunBudget counts what left, apart from // the readable copy's budget, and tunDropped what never did. - tunOut chan []byte - tunBulk chan []byte - tunBudget budget + tunOut chan []byte + tunBulk chan []byte + tunBudget budget + // xBudget counts the watches' exchange answers apart, as well as in + // tunBudget once they leave. + xBudget budget tunDropped int64 sent map[string]sentFrame srcRev uint64 @@ -756,26 +772,36 @@ type StatsRequest struct { By string `json:"-"` } -// Stats adds the stats purpose to a grant already on. The first time, it -// draws the statsId that names this box's rows at the service; later calls -// keep it. It sends no statistics itself. +// Stats turns statistics on. For a box without FTW Cloud that is the one +// answer on the box, and nothing links to Cloud. Under a grant already on it +// also adds the stats purpose; the first time, it draws the statsId that +// names this box's rows at the service, and later calls keep it. It sends no +// statistics itself. func (l *Link) Stats(req StatsRequest) error { - if len(l.opts.ServiceKeys) == 0 { - return ErrNoService - } if strings.TrimSpace(req.Policy) == "" { return fmt.Errorf("%w: name the policy the owner accepted", ErrGrantRequest) } l.mu.Lock() defer l.mu.Unlock() switch { - case l.grant == nil: + case l.grant == nil || l.grant.Revoked || l.disabled: + // The box's own answer, which needs no grant: also while a Cloud + // that was turned off waits for its deletion receipt. if req.Policy != cloudwire.StatsPolicy { return ErrNotGranted } - return l.startStatisticsOnly(req) - case l.grant.Revoked: - return ErrPurgePending + if err := l.opts.KV.SaveConfig(statsShareKey, "on"); err != nil { + return err + } + l.log.Info("audit: statistics on", "by", req.By) + return nil + case cloudUnderOlderTerms(l.grant): + return ErrCloudTermsFirst + case len(l.opts.ServiceKeys) == 0: + return ErrNoService + } + if err := l.opts.KV.SaveConfig(statsShareKey, "on"); err != nil { + return err } g := *l.grant if len(g.StatsID) == 0 || g.StatsPolicy != req.Policy { @@ -796,16 +822,21 @@ func (l *Link) Stats(req StatsRequest) error { return l.storeLocked(g, "statistics on") } -// StopStats removes the stats purpose and forgets the statsId. Turning it on -// again draws a new one. +// StopStats turns statistics off for a box without FTW Cloud, removes the +// stats purpose and forgets the statsId. Turning it on again draws a new one. +// With Cloud under the current terms statistics are part of Cloud. func (l *Link) StopStats(by string) error { l.mu.Lock() - if l.grant != nil && !l.grant.Revoked && l.grant.Policy == cloudwire.CloudTermsPolicy { + if cloudRequiresStatistics(l.grant) { l.mu.Unlock() return ErrCloudNeedsStatistics } only := l.grant != nil && l.grant.statisticsOnly() l.mu.Unlock() + if err := l.opts.KV.SaveConfig(statsShareKey, "off"); err != nil { + return err + } + l.log.Info("audit: statistics off", "by", by) if only { if err := l.End(by); err != nil { return err @@ -921,6 +952,13 @@ type Status struct { CloudTermsAccepted bool `json:"cloudTermsAccepted"` StatisticsOnly bool `json:"statisticsOnly"` StatisticsDecisionRequired bool `json:"statisticsDecisionRequired"` + // Statistics is the one answer for both anonymous daily reports, and + // StatisticsRequired says it is part of FTW Cloud, with no off switch. + Statistics bool `json:"statistics"` + StatisticsRequired bool `json:"statisticsRequired"` + // StatisticsNeedTerms is Cloud on under older terms: statistics wait + // until the owner accepts the current ones. + StatisticsNeedTerms bool `json:"statisticsNeedTerms"` // State is the readable copy's: off, on or revoking. State string `json:"state"` StatisticsPolicy string `json:"statisticsPolicy"` @@ -979,17 +1017,23 @@ func (l *Link) Status() Status { TunDropped: l.tunDropped, } s.CloudPolicy = cloudwire.CloudTermsPolicy - decision, _ := l.opts.KV.LoadConfig(statsDecisionKey) - s.StatisticsDecisionRequired = decision != cloudwire.StatsPolicy - if l.grant != nil && !l.grant.Revoked && l.grant.StatsPolicy == cloudwire.StatsPolicy && len(l.grant.StatsID) == 16 { - s.StatisticsDecisionRequired = false + // Nobody is asked any more: statistics are part of Cloud, and on for a + // box without it until its owner turns them off. The field stays false + // for clients that still read it. + share, _ := l.opts.KV.LoadConfig(statsShareKey) + g := l.grant + if l.disabled { + g = nil } + s.Statistics = sharesStatistics(g, share) + s.StatisticsRequired = cloudRequiresStatistics(g) + s.StatisticsNeedTerms = cloudUnderOlderTerms(g) s.StatisticsOnly = l.grant.statisticsOnly() s.CloudTermsAccepted = l.grant != nil && !l.grant.Revoked && l.grant.Policy == cloudwire.CloudTermsPolicy s.StatisticsPolicy = cloudwire.StatsPolicy s.StatisticsReady = l.connected && l.serviceStatsPolicy == cloudwire.StatsPolicy if g := l.grant; g != nil { - s.StatisticsActive = !g.Revoked && g.StatsAtMs > 0 && g.StatsPolicy == cloudwire.StatsPolicy && len(g.StatsID) == 16 && slices.Contains(g.Purposes, protocol.CloudPurposeStats) + s.StatisticsActive = s.Statistics && !g.Revoked && g.StatsAtMs > 0 && g.StatsPolicy == cloudwire.StatsPolicy && len(g.StatsID) == 16 && slices.Contains(g.Purposes, protocol.CloudPurposeStats) } s.BudgetReached = s.BytesLast24h >= budgetBytes if !l.lastSend.IsZero() { @@ -1037,6 +1081,11 @@ func (l *Link) TakesTun() bool { return l.Available() && l.opts.Tunnel != nil && !l.disabled && !l.grant.statisticsOnly() } +// TakesX reports whether the link carries the watches' exchange as x, which +// it does only beside the tunnel: a watch's roster entry opens its door +// through tun. +func (l *Link) TakesX() bool { return l.TakesTun() && l.opts.Exchange != nil } + // SessionInfo is what a phone needs, over its Noise session with the box, to // reach this home through FTW Cloud: the home id, the key that signs the // readable copy, the boot that anchors its times and names for the opaque @@ -1196,14 +1245,18 @@ func (snap grantSnapshot) allowed(class string) bool { func (l *Link) caps(snap grantSnapshot) []string { var out []string _, cars := l.opts.Site.(CarSite) - for _, c := range []string{cloudwire.MsgLive, cloudwire.MsgLP, cloudwire.MsgCar, cloudwire.MsgPrices, cloudwire.MsgHB, cloudwire.MsgEvent} { - if snap.allowed(c) && (c != cloudwire.MsgCar || cars) { + _, history := l.opts.Site.(HistorySite) + for _, c := range []string{cloudwire.MsgLive, cloudwire.MsgLP, cloudwire.MsgCar, cloudwire.MsgPrices, cloudwire.MsgHB, cloudwire.MsgEvent, cloudwire.MsgTiles} { + if snap.allowed(c) && (c != cloudwire.MsgCar || cars) && (c != cloudwire.MsgTiles || history) { out = append(out, c) } } out = append(out, cloudwire.MsgRoster, cloudwire.MsgNotice) if l.opts.Tunnel != nil { out = append(out, cloudwire.CapTun) + if l.opts.Exchange != nil { + out = append(out, cloudwire.CapX) + } } return append(out, cloudwire.CapShard) } diff --git a/go/internal/cloudlink/link_test.go b/go/internal/cloudlink/link_test.go index bcc5f51f8..d0ed40611 100644 --- a/go/internal/cloudlink/link_test.go +++ b/go/internal/cloudlink/link_test.go @@ -217,6 +217,8 @@ type fakeService struct { // dials again. kick int statsPolicy string + // have is the closed tiles hello_ok names as held. + have map[string]string // usedEarly says, per QUIC connection, whether it took early data. usedEarly []bool } @@ -326,8 +328,14 @@ func (s *fakeService) serve(conn uplinkConn, ekm []byte) { if s.wrongKey { signer, _ = ecdsa.GenerateKey(elliptic.P256(), rand.Reader) } + have := map[string]string{} + s.mu.Lock() + for id, etag := range s.have { + have[id] = etag + } + s.mu.Unlock() s.write(conn, signer, home, cloudwire.MsgHelloOK, cloudwire.HelloOK{ - BoxNonce: hello.BoxNonce, EKM: okEKM, Have: map[string]string{}, RosterSeq: s.rosterSeqOK, StatsPolicy: s.statsPolicy}) + BoxNonce: hello.BoxNonce, EKM: okEKM, Have: have, RosterSeq: s.rosterSeqOK, StatsPolicy: s.statsPolicy}) if s.afterHello != nil { go s.afterHello(conn, home) } @@ -360,10 +368,10 @@ func (s *fakeService) serve(conn uplinkConn, ekm []byte) { case err != nil: s.t.Errorf("box frame: %v", err) return - case env.T == cloudwire.MsgTun: - // d is Noise ciphertext; the box signs no tun. + case env.T == cloudwire.MsgTun, env.T == cloudwire.MsgX: + // d is Noise ciphertext; the box signs no tun and no x. if len(env.Sig) != 0 { - s.t.Errorf("the box signed a tun") + s.t.Errorf("the box signed a %s", env.T) } case cloudwire.Verify(pub, home, env) != nil: s.t.Errorf("box envelope %q does not verify", env.T) @@ -1139,43 +1147,35 @@ func (h *harness) nextRoster(t *testing.T) cloudwire.Roster { return r } -// Statistics are a second consent under a grant already on. The box draws a -// statsId once, keeps it while statistics stay on, tells the service through -// the roster and sends no statistics yet. -func TestStatisticsAreASecondConsentUnderTheGrant(t *testing.T) { +// Under Cloud, statistics come with the current terms. Under older terms they +// wait: asking for them is refused rather than added beside the old terms. +func TestStatisticsComeWithTheCurrentCloudTerms(t *testing.T) { svc := newFakeService(t) h := newHarness(t, svc) - if err := h.link.Stats(StatsRequest{Policy: "stats-2026-10"}); !errors.Is(err, ErrNotGranted) { - t.Fatalf("statistics with FTW Cloud off: %v", err) - } - h.grant(t) - first := h.nextRoster(t) - svc.waitFor(cloudwire.MsgLive, 5*time.Second) if err := h.link.Stats(StatsRequest{}); !errors.Is(err, ErrGrantRequest) { t.Fatalf("statistics without a policy: %v", err) } - - if err := h.link.Stats(StatsRequest{Policy: "stats-2026-10", By: "app:aaaa1111"}); err != nil { - t.Fatal(err) - } - r := h.nextRoster(t) - if strings.Join(r.Grant.Purposes, ",") != "serve,stats" || r.RosterSeq <= first.RosterSeq { - t.Fatalf("roster after statistics on: purposes %v, seq %d after %d", r.Grant.Purposes, r.RosterSeq, first.RosterSeq) + h.grant(t) + first := h.nextRoster(t) + if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy, By: "app:aaaa1111"}); !errors.Is(err, ErrCloudTermsFirst) { + t.Fatalf("statistics under older terms: %v", err) } - if st := h.link.Status(); strings.Join(st.Purposes, ",") != "serve,stats" { - t.Fatalf("status purposes %v", st.Purposes) + if g := h.record(t); len(g.StatsID) != 0 || strings.Join(g.Purposes, ",") != "serve" { + t.Fatalf("older terms gained statistics: %+v", g) } - g := h.record(t) - if len(g.StatsID) != statsIDBytes || g.StatsPolicy != "stats-2026-10" || g.Policy != "2026-10" { - t.Fatalf("grant record %+v", g) + if st := h.link.Status(); !st.StatisticsNeedTerms || st.Statistics || h.link.SharesStatistics() { + t.Fatalf("status under older terms: %+v", st) } - // A new policy needs a new id and consent start. - if err := h.link.Stats(StatsRequest{Policy: "stats-2026-11"}); err != nil { + if err := h.link.Grant(GrantRequest{Scopes: []string{protocol.ScopeLiveRead}, Policy: cloudwire.CloudTermsPolicy, By: "owner"}); err != nil { t.Fatal(err) } - if again := h.record(t); bytes.Equal(again.StatsID, g.StatsID) || again.StatsPolicy != "stats-2026-11" { - t.Fatalf("statsId %x became %x", g.StatsID, again.StatsID) + r := h.nextRoster(t) + if strings.Join(r.Grant.Purposes, ",") != "serve,stats" || r.RosterSeq <= first.RosterSeq || len(r.Grant.StatsID) != statsIDBytes { + t.Fatalf("roster under the current terms: %+v", r.Grant) + } + if st := h.link.Status(); !st.Statistics || !st.StatisticsRequired || st.StatisticsNeedTerms || !h.link.SaidYesToStatistics() { + t.Fatalf("status under the current terms: %+v", st) } // A restart reads it back. @@ -1184,74 +1184,83 @@ func TestStatisticsAreASecondConsentUnderTheGrant(t *testing.T) { if err != nil { t.Fatal(err) } - if st := reopened.Status(); strings.Join(st.Purposes, ",") != "serve,stats" { + if st := reopened.Status(); strings.Join(st.Purposes, ",") != "serve,stats" || !st.Statistics { t.Fatalf("after a restart: %+v", st) } - for _, env := range svc.received() { if env.T == "stats" { - t.Fatal("the box sent statistics") + t.Fatal("the box sent statistics before the service approved them") } } } -// Turning statistics off forgets the statsId; turning them on again draws a -// new one, because the service refuses an id it has deleted. -func TestStoppingStatisticsForgetsTheID(t *testing.T) { +// While a Cloud that was turned off waits for its deletion receipt, the box's +// own answer can still change. It touches nothing in the grant being deleted. +func TestTheLocalAnswerChangesWhileTheCopyIsBeingDeleted(t *testing.T) { svc := newFakeService(t) h := newHarness(t, svc) - if err := h.link.StopStats("lan"); err != nil { - t.Fatalf("statistics off with FTW Cloud off: %v", err) - } h.grant(t) - h.nextRoster(t) - if err := h.link.Stats(StatsRequest{Policy: "p"}); err != nil { - t.Fatal(err) - } - h.nextRoster(t) - old := h.record(t).StatsID - - if err := h.link.StopStats("lan"); err != nil { + if err := h.link.Revoke("test"); err != nil { t.Fatal(err) } - r := h.nextRoster(t) - if strings.Join(r.Grant.Purposes, ",") != "serve" { - t.Fatalf("roster after statistics off: %v", r.Grant.Purposes) - } - if g := h.record(t); len(g.StatsID) != 0 || g.StatsPolicy != "" || strings.Join(g.Purposes, ",") != "serve" { - t.Fatalf("grant record after statistics off: %+v", g) + if err := h.link.Stats(StatsRequest{Policy: "p"}); !errors.Is(err, ErrNotGranted) { + t.Fatalf("an unknown policy: %v", err) } - if err := h.link.StopStats("lan"); err != nil { - t.Fatalf("statistics off twice: %v", err) + if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy}); err != nil { + t.Fatalf("statistics while revoking: %v", err) } - - if err := h.link.Stats(StatsRequest{Policy: "p"}); err != nil { - t.Fatal(err) + if g := h.record(t); !g.Revoked || len(g.StatsID) != 0 || !h.link.SharesStatistics() { + t.Fatalf("grant while revoking: %+v", g) } - if renewed := h.record(t).StatsID; len(renewed) != statsIDBytes || bytes.Equal(renewed, old) { - t.Fatalf("statsId %x after %x", renewed, old) + if err := h.link.StopStats("lan"); err != nil || h.link.SharesStatistics() { + t.Fatalf("off while revoking: %v", err) } } -func TestNoStatisticsWhileTheCopyIsBeingDeleted(t *testing.T) { +// A stored grant counts only when the link honours it. With a cloud.key that +// is not the one the grant was made with, the box's own answer stands, and a +// box that only shares by default has said no yes. +func TestOnlyAGrantTheLinkHonoursCounts(t *testing.T) { svc := newFakeService(t) h := newHarness(t, svc) - h.grant(t) - if err := h.link.Revoke("test"); err != nil { + if err := h.link.Grant(GrantRequest{Scopes: []string{protocol.ScopeLiveRead}, Policy: cloudwire.CloudTermsPolicy, By: "owner"}); err != nil { t.Fatal(err) } - if err := h.link.Stats(StatsRequest{Policy: "p"}); !errors.Is(err, ErrPurgePending) { - t.Fatalf("statistics while revoking: %v", err) + h.nextRoster(t) + if err := h.kv.SaveConfig(statsShareKey, "default"); err != nil { + t.Fatal(err) + } + if !h.link.SaidYesToStatistics() { + t.Fatal("the current terms are not a yes") + } + if err := os.Remove(h.keyPath); err != nil { + t.Fatal(err) + } + if _, err := createKey(h.keyPath); err != nil { + t.Fatal(err) + } + reopened, err := New(Options{Site: h.site, Roster: fakeRoster{}, KV: h.kv, KeyPath: h.keyPath, ServiceKeys: h.link.opts.ServiceKeys}) + if err != nil { + t.Fatal(err) + } + if reopened.Status().State != "off" || reopened.SaidYesToStatistics() || !reopened.SharesStatistics() { + t.Fatalf("a grant made with another key counted: %+v", reopened.Status()) } } -func TestABuildWithoutAServiceRefusesStatistics(t *testing.T) { - link, err := New(Options{Site: &fakeSite{}, Roster: fakeRoster{}, KV: &memKV{}, +// Without Cloud the statistics answer is the box's own, so a build with no +// service still records it. It still names the policy the owner read. +func TestABuildWithoutAServiceStillRecordsTheLocalAnswer(t *testing.T) { + kv := &memKV{} + link, err := New(Options{Site: &fakeSite{}, Roster: fakeRoster{}, KV: kv, KeyPath: filepath.Join(t.TempDir(), "cloud.key"), ServiceKeys: []*ecdsa.PublicKey{}}) if err != nil { t.Fatal(err) } - if err := link.Stats(StatsRequest{Policy: "p"}); !errors.Is(err, ErrNoService) { + if err := link.Stats(StatsRequest{Policy: "p"}); !errors.Is(err, ErrNotGranted) { + t.Fatalf("an unknown policy: %v", err) + } + if err := link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy}); err != nil || !SharesStatistics(kv) { t.Fatalf("statistics: %v", err) } } diff --git a/go/internal/cloudlink/sender.go b/go/internal/cloudlink/sender.go index b377eaf7a..03aa4725e 100644 --- a/go/internal/cloudlink/sender.go +++ b/go/internal/cloudlink/sender.go @@ -34,6 +34,12 @@ const ( // holding a session open all day at 1 Hz sends about 50 MB; this is five // of them, and stops a runaway before it eats a household's uplink. tunBudgetBytes = 250_000_000 + // xBudgetBytes is the watches' share of it: their exchange answers, a + // tenth. An answer is about a kilobyte, so this is some twenty thousand + // a day. xStopsAt is the share of the tunnel's budget past which x + // stops, so the phones keep the last of it. + xBudgetBytes = tunBudgetBytes / 10 + xStopsAt = 0.9 ) // sentFrame is the last frame of a class that left the house, exactly. @@ -101,6 +107,9 @@ type cadence struct { // arrived says a driver finished a poll since live was last composed, // while a phone watches. arrived bool + + // tiles is what this connection sent of the tiles class. + tiles tileState } // liveCheckEvery is how often the source table is looked at, so a table @@ -115,7 +124,7 @@ func (w *writer) run(ctx context.Context, events <-chan cloudwire.Envelope) erro connHome := l.current().home // A roster goes at every connect, which also renews its ttlS: the // service counts that from when it received the last one. - st := &cadence{full: true} + st := &cadence{full: true, tiles: newTileState(w.have)} var purgeSince time.Time // slot fires when an arrival waits for the watched live's period. var slot <-chan time.Time @@ -208,6 +217,9 @@ func (w *writer) downlink(snap grantSnapshot, st *cadence, env cloudwire.Envelop if slices.Contains(b.What, cloudwire.ResyncRoster) { st.rosterAt = time.Time{} } + if slices.Contains(b.What, cloudwire.ResyncTiles) { + st.tiles.resync(b.Tiles) + } } return nil } @@ -385,7 +397,8 @@ func (w *writer) tick(snap grantSnapshot, st *cadence, now time.Time) error { st.anyAt = now } } - return nil + // Tiles go last: a reading never waits for the ledger. + return w.tiles(snap, st, now) } // takeRoster says whether a roster is due, and builds it from the link as it diff --git a/go/internal/cloudlink/stats.go b/go/internal/cloudlink/stats.go index 7edd28e12..88aabd9c2 100644 --- a/go/internal/cloudlink/stats.go +++ b/go/internal/cloudlink/stats.go @@ -28,6 +28,13 @@ func (w *writer) statistics(snap grantSnapshot, st *cadence, now time.Time) erro !slices.Contains(g.Purposes, protocol.CloudPurposeStats) || !w.link.clockSynced() { return nil } + // The one answer covers this upload too, and only a given yes sends an + // hour: Cloud under the current terms or a local yes. Off, a box that + // shares only by default, or Cloud under older terms sends none, + // whatever the grant still carries. + if share, _ := w.link.opts.KV.LoadConfig(statsShareKey); !saidYesToStatistics(g, share) { + return nil + } hour := now.UTC().Truncate(time.Hour).Add(-time.Hour).UnixMilli() if hour < g.StatsAtMs || st.statsHour == hour || now.Sub(st.statsCheckedAt) < time.Minute { return nil diff --git a/go/internal/cloudlink/stats_test.go b/go/internal/cloudlink/stats_test.go index 80b784e74..9bf5ddffb 100644 --- a/go/internal/cloudlink/stats_test.go +++ b/go/internal/cloudlink/stats_test.go @@ -3,9 +3,12 @@ package cloudlink import ( "context" "crypto/ecdsa" - "github.com/srcfl/ftw/go/protocol/cloudwire" + "errors" "testing" "time" + + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/cloudwire" ) type statsSite struct{ *fakeSite } @@ -13,25 +16,54 @@ type statsSite struct{ *fakeSite } func (s statsSite) Statistics(_ context.Context, now time.Time) (cloudwire.Stats, bool) { return cloudwire.Stats{HourMS: now.UTC().Truncate(time.Hour).Add(-time.Hour).UnixMilli(), Metrics: map[string]int64{"pv_generation_wh": 120}}, true } -func TestStatisticsSendsSignedClosedHourOnlyWithCurrentConsent(t *testing.T) { + +func statsHarness(t *testing.T, approved bool) (*fakeService, *harness) { svc := newFakeService(t) - svc.statsPolicy = cloudwire.StatsPolicy + if approved { + svc.statsPolicy = cloudwire.StatsPolicy + } h := newHarnessWith(t, svc, []*ecdsa.PublicKey{&svc.key.PublicKey}, nil, func(o *Options) { o.Site = statsSite{&fakeSite{}} o.Now = func() time.Time { return time.Now().Add(-2 * time.Hour) } }) + return svc, h +} + +func sentStats(svc *fakeService) bool { + for _, e := range svc.received() { + if e.T == cloudwire.MsgStats { + return true + } + } + return false +} + +// The hourly upload goes only under the current Cloud terms. Asking for it +// under older terms is refused, and a grant from before this rule that still +// carries the stats purpose under older terms sends nothing either. +func TestStatisticsSendsSignedClosedHourOnlyWithCurrentConsent(t *testing.T) { + svc, h := statsHarness(t, true) h.grant(t) h.nextRoster(t) - if err := h.link.Stats(StatsRequest{Policy: "old-policy"}); err != nil { + if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy}); !errors.Is(err, ErrCloudTermsFirst) { + t.Fatalf("statistics under older terms: %v", err) + } + // As an earlier release could leave it: older terms with the stats purpose. + g := h.record(t) + g.Purposes, g.StatsID, g.StatsPolicy, g.StatsAtMs = []string{protocol.CloudPurposeServe, protocol.CloudPurposeStats}, make([]byte, statsIDBytes), cloudwire.StatsPolicy, 1 + h.link.mu.Lock() + err := h.link.storeLocked(g, "test") + h.link.mu.Unlock() + if err != nil { t.Fatal(err) } h.nextRoster(t) - for _, e := range svc.received() { - if e.T == cloudwire.MsgStats { - t.Fatal("old consent sent statistics") - } + time.Sleep(1500 * time.Millisecond) + if sentStats(svc) { + t.Fatal("older terms sent an hour") } - if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy}); err != nil { + + if err := h.link.Grant(GrantRequest{Scopes: []string{protocol.ScopeLiveRead}, Policy: cloudwire.CloudTermsPolicy, By: "owner"}); err != nil { t.Fatal(err) } r := h.nextRoster(t) @@ -46,33 +78,82 @@ func TestStatisticsSendsSignedClosedHourOnlyWithCurrentConsent(t *testing.T) { if !h.link.Status().StatisticsActive { t.Fatal("current consent inactive") } - if err := h.link.StopStats("box"); err != nil { + if err := h.link.StopStats("box"); !errors.Is(err, ErrCloudNeedsStatistics) { + t.Fatalf("Cloud stopped statistics alone: %v", err) + } + if err := h.link.End("box"); err != nil { t.Fatal(err) } - r = h.nextRoster(t) - if len(r.Grant.StatsID) != 0 || h.link.Status().StatisticsActive { - t.Fatal("withdrawn consent remained active") + if h.link.Status().StatisticsActive { + t.Fatal("ended Cloud kept statistics active") } } -func TestStatisticsWaitsForTheServicesApprovedPolicy(t *testing.T) { - svc := newFakeService(t) - h := newHarnessWith(t, svc, []*ecdsa.PublicKey{&svc.key.PublicKey}, nil, func(o *Options) { - o.Site = statsSite{&fakeSite{}} - o.Now = func() time.Time { return time.Now().Add(-2 * time.Hour) } - }) +// A statistics-only link from a local yes follows the one answer: off sends +// no hour, and ReconcileStatistics ends the link as turning statistics off +// does. +func TestALocalNoStopsTheHourlyUpload(t *testing.T) { + svc, h := statsHarness(t, true) h.grant(t) h.nextRoster(t) - if err := h.link.Stats(StatsRequest{Policy: cloudwire.StatsPolicy}); err != nil { + g := h.record(t) + g.Policy, g.Scopes, g.Purposes = cloudwire.StatsPolicy, []string{}, []string{protocol.CloudPurposeStats} + g.StatsID, g.StatsPolicy, g.StatsAtMs = make([]byte, statsIDBytes), cloudwire.StatsPolicy, 1 + if err := h.kv.SaveConfig(statsShareKey, "off"); err != nil { + t.Fatal(err) + } + h.link.mu.Lock() + err := h.link.storeLocked(g, "test") + h.link.mu.Unlock() + if err != nil { + t.Fatal(err) + } + h.nextRoster(t) + time.Sleep(1500 * time.Millisecond) + if sentStats(svc) { + t.Fatal("an answer of off sent an hour") + } + if st := h.link.Status(); st.Statistics || st.StatisticsActive { + t.Fatalf("status says statistics run: %+v", st) + } + + // Sharing only by default is no given yes, so no hour either. + if err := h.kv.SaveConfig(statsShareKey, "default"); err != nil { + t.Fatal(err) + } + time.Sleep(1500 * time.Millisecond) + if sentStats(svc) { + t.Fatal("an answer that was never given sent an hour") + } + + // The same link with the answer on does send, so the answer is the gate. + if err := h.kv.SaveConfig(statsShareKey, "on"); err != nil { + t.Fatal(err) + } + svc.waitFor(cloudwire.MsgStats, 3*time.Second) + + if err := h.kv.SaveConfig(statsShareKey, "off"); err != nil { + t.Fatal(err) + } + if err := h.link.ReconcileStatistics("test"); err != nil { + t.Fatal(err) + } + if rec := h.record(t); !rec.Revoked || h.link.Status().Link != "ending" { + t.Fatalf("the statistics-only link did not end: %+v %+v", rec, h.link.Status()) + } +} + +func TestStatisticsWaitsForTheServicesApprovedPolicy(t *testing.T) { + svc, h := statsHarness(t, false) + if err := h.link.Grant(GrantRequest{Scopes: []string{protocol.ScopeLiveRead}, Policy: cloudwire.CloudTermsPolicy, By: "owner"}); err != nil { t.Fatal(err) } h.nextRoster(t) if h.link.Status().StatisticsReady { t.Fatal("unapproved service declared ready") } - for _, e := range svc.received() { - if e.T == cloudwire.MsgStats { - t.Fatal("statistics sent without service approval") - } + time.Sleep(time.Second) + if sentStats(svc) { + t.Fatal("statistics sent without service approval") } } diff --git a/go/internal/cloudlink/tiles.go b/go/internal/cloudlink/tiles.go new file mode 100644 index 000000000..c47d13d86 --- /dev/null +++ b/go/internal/cloudlink/tiles.go @@ -0,0 +1,239 @@ +package cloudlink + +import ( + "context" + "slices" + "time" + + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// HistorySite is a Site that keeps history. HistoryWindow answers stored +// series as the app protocol's history provider does: every series that +// overlaps [fromMs, toMs), at the stored width stepMs, without missing +// buckets. A Site without it sends no tiles. +type HistorySite interface { + HistoryWindow(ctx context.Context, names []string, stepMs, fromMs, toMs int64) (map[string][]protocol.HistorySample, error) +} + +// tilesRecheckEvery is how often the closed tiles of the window are packed +// again, so a gap the ledger repaired reaches the service. A test shortens +// it. +var tilesRecheckEvery = time.Hour + +const ( + // tilesRetryAfter holds tiles back after a read failed or the budget + // refused one. + tilesRetryAfter = time.Minute + // tilesReadTimeout bounds one read of the ledger. + tilesReadTimeout = 5 * time.Second + // tilesNamedMax is how many tiles a resync may name. + tilesNamedMax = cloudwire.TilesClosedMax + 1 +) + +// tileState is one connection's view of the tiles it has sent. +type tileState struct { + // have is each closed tile the service holds, by id, as hello_ok named + // them and as sent since. + have map[string]string + // trail is the start of the trailing tile and through the end of the + // last of its buckets sent on this connection; through == trail means + // none of it has gone. appends counts appends since it last went whole. + trail, through int64 + appends int + // all asks for every tile of the window, at connect and after a resync + // that names none; force sends them whatever have says. named are the + // tiles a resync named. + all, force bool + named []string + // passing says a pass over the closed tiles of the window is under way, + // and passAt is the start of the next one it looks at, newest first. + passing bool + passAt int64 + recheckAt time.Time + retryAt time.Time +} + +func newTileState(have map[string]string) tileState { + held := make(map[string]string, len(have)) + for id, etag := range have { + held[id] = etag + } + return tileState{have: held, all: true} +} + +// resync takes the service's resync of tiles: the tiles it names, or every +// one of the window when it names none. A pass under way starts again. +func (ts *tileState) resync(named []string) { + ts.force, ts.passing = true, false + if len(named) == 0 || len(named) > tilesNamedMax { + ts.all, ts.named = true, nil + return + } + ts.named = slices.Clone(named) +} + +// tiles sends what the tiles class owes: the trailing tile as its buckets +// close, a tile that closed, and at connect or a resync the tiles of the +// last 48 hours the service lacks (docs/protocol-cloud.md, tiles). It reads +// the ledger at most once a tick, and only when a bucket closed or a pass is +// under way, so a reading never waits long behind it. +func (w *writer) tiles(snap grantSnapshot, st *cadence, now time.Time) error { + l := w.link + hs, ok := l.opts.Site.(HistorySite) + // Tiles are dated by wall time, so a box without NTP sends none. + if !ok || !l.clockSynced() { + return nil + } + if !snap.allowed(cloudwire.MsgTiles) { + // Without history in the grant the service drops what it held, so + // when history comes back every tile goes again, as at connect. + st.tiles = newTileState(nil) + return nil + } + ts := &st.tiles + if now.Before(ts.retryAt) { + return nil + } + res := cloudwire.TileRes + step, span := protocol.ResolutionStepMs[res], protocol.ResolutionTileSpanMs[res] + wall := l.opts.Now().UnixMilli() + closedThrough := (wall - cloudwire.TileSettleMs) / step * step + open := protocol.TileStartFor(res, closedThrough) + oldest := wall - cloudwire.TileKeepMs + + // A pass over the closed tiles: at connect, after a resync, and once an + // hour. The trailing tile goes whole again with every tile. + if !ts.passing && (ts.all || len(ts.named) > 0 || !now.Before(ts.recheckAt)) { + ts.passing, ts.passAt = true, open-span + if ts.all || slices.Contains(ts.named, protocol.TileIDFor(res, 1, open)) { + ts.trail, ts.through = open, open + } + } + if ts.trail != open { + // The trailing tile closed since its last piece: it goes closed. + if prev := ts.trail; prev != 0 && prev < open && prev+span > oldest { + sent, err := w.closedTile(hs, ts, prev, closedThrough, false) + if sent { + ts.trail, ts.through, ts.appends = open, open, 0 + } + return err + } + ts.trail, ts.through, ts.appends = open, open, 0 + } + if closedThrough > ts.through { + whole := ts.through == open || ts.appends >= cloudwire.TileAppendsMax + from := ts.through + if whole { + from = open + } + c, err := w.pack(hs, ts, open, from, closedThrough) + if err != nil { + return nil + } + piece := cloudwire.Piece(c, int((from-open)/step), int((closedThrough-open)/step)) + if whole { + piece = cloudwire.Piece(c, 0, int(protocol.PointsPerTile(res))) + } + sent, err := w.sendTile(ts, &piece) + if sent { + ts.through = closedThrough + if whole { + ts.appends = 0 + } else { + ts.appends++ + } + } + return err + } + if !ts.passing { + return nil + } + // One closed tile of the window a tick, newest first, unless the service + // holds it at this etag; or each one a resync named. + for ; ts.passAt+span > oldest; ts.passAt -= span { + if !ts.all && len(ts.named) > 0 && !slices.Contains(ts.named, protocol.TileIDFor(res, 1, ts.passAt)) { + continue + } + sent, err := w.closedTile(hs, ts, ts.passAt, closedThrough, ts.force) + if sent { + ts.passAt -= span + } + return err + } + ts.passing, ts.all, ts.force, ts.named = false, false, false, nil + ts.recheckAt = now.Add(tilesRecheckEvery) + return nil +} + +// closedTile packs the tile at start whole and sends it unless the service +// holds it at that etag and force is false. sent is false when nothing could +// go and the caller should try again later; a tile that needs no sending +// counts as sent. +func (w *writer) closedTile(hs HistorySite, ts *tileState, start, closedThrough int64, force bool) (bool, error) { + span := protocol.ResolutionTileSpanMs[cloudwire.TileRes] + if start+span > closedThrough { + return true, nil + } + c, err := w.pack(hs, ts, start, start, start+span) + if err != nil { + return false, nil + } + if !force && ts.have[c.TileID] == c.Etag { + return true, nil + } + body := cloudwire.Tiles{TileID: c.TileID, Etag: c.Etag, Res: c.Res, StartMs: c.StartMs, StepMs: c.StepMs, + Series: c.Series, Data: c.Data} + sent, err := w.sendTile(ts, &body) + if sent { + ts.have[c.TileID] = c.Etag + } + return sent, err +} + +// PackTile reads [from, to) of the tile at start through hs and packs the +// tile as the app protocol does, so a closed tile is the same bytes as the +// box serves over Noise. Buckets from to on are left out: they have not +// closed. +func PackTile(ctx context.Context, hs HistorySite, start, from, to int64) (protocol.HistChunk, error) { + res := cloudwire.TileRes + windows, err := hs.HistoryWindow(ctx, cloudwire.TileSeries, protocol.ResolutionStepMs[res], from, to) + if err != nil { + return protocol.HistChunk{}, err + } + for name, samples := range windows { + windows[name] = slices.DeleteFunc(slices.Clone(samples), func(s protocol.HistorySample) bool { + return s.StartMs < from || s.StartMs >= to + }) + } + return protocol.PackTile(res, 1, start, cloudwire.TileSeries, windows, to), nil +} + +// pack is PackTile under the read timeout. A ledger that does not answer +// holds tiles back a minute. +func (w *writer) pack(hs HistorySite, ts *tileState, start, from, to int64) (protocol.HistChunk, error) { + ctx, cancel := context.WithTimeout(context.Background(), tilesReadTimeout) + defer cancel() + c, err := PackTile(ctx, hs, start, from, to) + if err != nil { + w.link.log.Debug("FTW Cloud tiles: the ledger did not answer", "err", err) + ts.retryAt = time.Now().Add(tilesRetryAfter) + } + return c, err +} + +// sendTile sends one tile under the byte budget, if the grant in force +// still allows tiles: a revoke may land while the ledger is read. Past the +// budget, tiles wait a minute and go again from where they were. +func (w *writer) sendTile(ts *tileState, body *cloudwire.Tiles) (bool, error) { + snap := w.link.current() + if !snap.allowed(cloudwire.MsgTiles) { + return false, nil + } + sent, err := w.sendBody(snap, cloudwire.MsgTiles, body, false) + if err == nil && !sent { + ts.retryAt = time.Now().Add(tilesRetryAfter) + } + return sent, err +} diff --git a/go/internal/cloudlink/tiles_test.go b/go/internal/cloudlink/tiles_test.go new file mode 100644 index 000000000..55a888781 --- /dev/null +++ b/go/internal/cloudlink/tiles_test.go @@ -0,0 +1,434 @@ +package cloudlink + +import ( + "bytes" + "context" + "crypto/ecdsa" + "encoding/binary" + "errors" + "slices" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/protocol" + "github.com/srcfl/ftw/go/protocol/cloudwire" +) + +// tileWall is 2026-10-12 17:12:00 UTC: the trailing tile started at 12:00, +// and the bucket that ended at 17:10 has settled. +var tileWall = time.UnixMilli(1_791_806_400_000 + (5*60+12)*60_000) + +// historySite is the fake house with a ledger that has a bucket for every +// five minutes up to the wall clock, the one still filling included, as +// the real one has. +type historySite struct { + *fakeSite + mu sync.Mutex + wall time.Time + // bump is added to every value from bumpFrom on, as a ledger repair + // would change them. + bump float64 + bumpFrom int64 + failing bool + reads int +} + +func (s *historySite) now() time.Time { + s.mu.Lock() + defer s.mu.Unlock() + return s.wall +} + +func (s *historySite) set(wall time.Time) { + s.mu.Lock() + s.wall = wall + s.mu.Unlock() +} + +// value is the ledger's mean for one series and bucket. +func (s *historySite) value(name string, start int64) float64 { + i := float64(start/300_000%10_000) + float64(len(name)) + if start >= s.bumpFrom { + i += s.bump + } + switch name { + case "pv_w": + return -i + case "battery_w": + return 2 * i + } + return i +} + +func (s *historySite) HistoryWindow(_ context.Context, names []string, stepMs, fromMs, toMs int64) (map[string][]protocol.HistorySample, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.reads++ + if s.failing { + return nil, errors.New("the ledger is busy") + } + out := map[string][]protocol.HistorySample{} + for at := fromMs / stepMs * stepMs; at < toMs && at <= s.wall.UnixMilli(); at += stepMs { + for _, name := range names { + out[name] = append(out[name], protocol.HistorySample{StartMs: at, W: s.value(name, at)}) + } + } + return out, nil +} + +// noise is the tile the box serves over Noise at the same moment. +func (s *historySite) noise(t *testing.T, start int64) protocol.HistChunk { + t.Helper() + span := protocol.ResolutionTileSpanMs[cloudwire.TileRes] + w, err := s.HistoryWindow(context.Background(), cloudwire.TileSeries, 300_000, start, start+span) + if err != nil { + t.Fatal(err) + } + return protocol.PackTile(cloudwire.TileRes, 1, start, cloudwire.TileSeries, w, s.now().UnixMilli()) +} + +func newTileHarness(t *testing.T, svc *fakeService, tweak func(*Options)) (*harness, *historySite) { + t.Helper() + hs := &historySite{wall: tileWall, bumpFrom: 1 << 62} + h := newHarnessWith(t, svc, []*ecdsa.PublicKey{&svc.key.PublicKey}, nil, func(o *Options) { + hs.fakeSite = o.Site.(*fakeSite) + o.Site, o.Now = hs, hs.now + if tweak != nil { + tweak(o) + } + }) + return h, hs +} + +// tilesSent decodes every tiles body the service has received, in order. +func tilesSent(t *testing.T, svc *fakeService) []cloudwire.Tiles { + t.Helper() + var out []cloudwire.Tiles + for _, env := range svc.received() { + if env.T != cloudwire.MsgTiles { + continue + } + var b cloudwire.Tiles + if err := cloudwire.Unmarshal(env.B, &b); err != nil { + t.Fatal(err) + } + if err := b.Check(); err != nil { + t.Fatalf("the box sent a tile of no shape: %v", err) + } + out = append(out, b) + } + return out +} + +// waitTiles waits until the service has n tiles bodies, and a little more, +// so one too many shows too. +func waitTiles(t *testing.T, svc *fakeService, n int) []cloudwire.Tiles { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for len(tilesSent(t, svc)) < n && time.Now().Before(deadline) { + time.Sleep(10 * time.Millisecond) + } + time.Sleep(150 * time.Millisecond) + got := tilesSent(t, svc) + if len(got) != n { + t.Fatalf("the service has %d tiles, want %d: %v", len(got), n, tileNames(got)) + } + return got +} + +func tileNames(ts []cloudwire.Tiles) []string { + var out []string + for _, b := range ts { + out = append(out, b.TileID+"@"+time.UnixMilli(b.StartMs).UTC().Format("02 15:04")) + } + return out +} + +func point(b cloudwire.Tiles, series string, i int) int32 { + s := slices.Index(b.Series, series) + return int32(binary.LittleEndian.Uint32(b.Data[(s*b.Points()+i)*4:])) +} + +func tileID(start int64) string { return protocol.TileIDFor(cloudwire.TileRes, 1, start) } + +const ( + trailStart = int64(1_791_806_400_000) // 2026-10-12 12:00 UTC + tileSpan = int64(43_200_000) +) + +func grantHistory(t *testing.T, h *harness) { + t.Helper() + h.grant(t, protocol.ScopeLiveRead, protocol.ScopeHistoryRead) +} + +func TestAtConnectTheTrailingTileGoesWholeAndTheClosedOnesAsOverNoise(t *testing.T) { + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, nil) + grantHistory(t, h) + got := waitTiles(t, svc, 5) + + whole := got[0] + if whole.TileID != tileID(trailStart) || !whole.Partial || !whole.Whole() || whole.Points() != 144 { + t.Fatalf("first tile %v, want the trailing tile whole", tileNames(got[:1])) + } + // 17:05 has closed; 17:10 has not settled, and the bucket still filling + // in the ledger stays out. + at1705, at1710 := int((17*60+5-12*60)/5), int((17*60+10-12*60)/5) + if v := point(whole, "grid_w", at1705); v != protocol.ClampSample(hs.value("grid_w", trailStart+int64(at1705)*300_000)) { + t.Fatalf("17:05 reads %d", v) + } + if point(whole, "grid_w", at1710) != protocol.MissingSample || point(whole, "load_w", 143) != protocol.MissingSample { + t.Fatal("a bucket that has not closed went as a value") + } + + // Then the closed tiles of the last 48 hours, newest first, each the + // Noise tile byte for byte. + want := []int64{trailStart - tileSpan, trailStart - 2*tileSpan, trailStart - 3*tileSpan, trailStart - 4*tileSpan} + for i, start := range want { + b := got[i+1] + n := hs.noise(t, start) + if b.Partial || b.TileID != n.TileID || b.Etag != n.Etag || !bytes.Equal(b.Data, n.Data) || b.StartMs != n.StartMs { + t.Fatalf("closed tile %d is %v, not the Noise tile %s", i, tileNames([]cloudwire.Tiles{b}), n.TileID) + } + } + // Caps say so. + var hello cloudwire.Hello + payload, _ := cloudwire.DecodeFrame(svc.hello) + env, _ := cloudwire.DecodeEnvelope(payload) + if cloudwire.Unmarshal(env.B, &hello) != nil || !slices.Contains(hello.Caps, cloudwire.MsgTiles) { + t.Fatalf("hello.caps %v lacks tiles", hello.Caps) + } +} + +func TestEachClosedBucketGoesAsAnAppendUntilTheTileGoesWholeAgain(t *testing.T) { + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, nil) + grantHistory(t, h) + waitTiles(t, svc, 5) + + // The bucket 17:10–17:15 settles at 17:15:30. + hs.set(tileWall.Add(3*time.Minute + 29*time.Second)) + time.Sleep(200 * time.Millisecond) + if n := len(tilesSent(t, svc)); n != 5 { + t.Fatalf("a tile went before its bucket settled: %d", n) + } + for i := 1; i <= cloudwire.TileAppendsMax+1; i++ { + hs.set(tileWall.Add(time.Duration(i)*5*time.Minute - 2*time.Minute + 31*time.Second)) + got := waitTiles(t, svc, 5+i) + b := got[len(got)-1] + if i <= cloudwire.TileAppendsMax { + start := trailStart + int64((17*60+10-12*60+5*(i-1)))*60_000 + if !b.Partial || b.Whole() || b.Points() != 1 || b.StartMs != start { + t.Fatalf("append %d is %v", i, tileNames([]cloudwire.Tiles{b})) + } + if point(b, "pv_w", 0) != protocol.ClampSample(hs.value("pv_w", start)) { + t.Fatalf("append %d carries %d", i, point(b, "pv_w", 0)) + } + continue + } + if !b.Partial || !b.Whole() || b.Points() != 144 { + t.Fatalf("after %d appends the tile went as %v, not whole", cloudwire.TileAppendsMax, tileNames([]cloudwire.Tiles{b})) + } + } + // The pieces lay into the tile the box holds. + laid := cloudwire.Lay(tilesSent(t, svc)) + if last := laid[len(laid)-1]; last.TileID != tileID(trailStart) || !last.Partial { + t.Fatalf("laid %v", last.TileID) + } +} + +func TestATileGoesClosedWhenItsLastBucketCloses(t *testing.T) { + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, nil) + hs.set(time.UnixMilli(trailStart + tileSpan - 2*60_000)) // 23:58 + grantHistory(t, h) + waitTiles(t, svc, 5) + + hs.set(time.UnixMilli(trailStart + tileSpan + 31_000)) // 00:00:31 + got := waitTiles(t, svc, 6) + closed := got[5] + n := hs.noise(t, trailStart) + if closed.Partial || closed.TileID != tileID(trailStart) || closed.Etag != n.Etag { + t.Fatalf("at midnight %v went, not the closed tile", tileNames(got[5:])) + } + // Nothing of the new tile until its first bucket settles. + hs.set(time.UnixMilli(trailStart + tileSpan + 5*60_000 + 31_000)) + got = waitTiles(t, svc, 7) + first := got[6] + if first.TileID != tileID(trailStart+tileSpan) || !first.Whole() || !first.Partial || + point(first, "grid_w", 0) == protocol.MissingSample || point(first, "grid_w", 1) != protocol.MissingSample { + t.Fatalf("the new tile began as %v", tileNames(got[6:])) + } +} + +func TestTheServicesHaveSavesItsClosedTiles(t *testing.T) { + svc := newFakeService(t) + probe := &historySite{wall: tileWall, bumpFrom: 1 << 62} + held := probe.noise(t, trailStart-tileSpan) + svc.have = map[string]string{ + held.TileID: held.Etag, + tileID(trailStart - 3*tileSpan): "00000000", // an older etag + "5m/1/1": "ffffffff", // nothing the box has + } + h, _ := newTileHarness(t, svc, nil) + grantHistory(t, h) + got := waitTiles(t, svc, 4) + ids := []string{got[1].TileID, got[2].TileID, got[3].TileID} + if slices.Contains(ids, held.TileID) || !slices.Contains(ids, tileID(trailStart-3*tileSpan)) { + t.Fatalf("sent %v with have %v", ids, svc.have) + } +} + +func TestAResyncOfTilesSendsThemAgainOnceInFiveMinutes(t *testing.T) { + svc := newFakeService(t) + conns := make(chan uplinkConn, 4) + svc.afterHello = func(conn uplinkConn, _ string) { conns <- conn } + h, _ := newTileHarness(t, svc, nil) + grantHistory(t, h) + waitTiles(t, svc, 5) + conn := <-conns + + resync := func(named ...string) { + svc.write(conn, svc.key, svc.home, cloudwire.MsgResync, cloudwire.Resync{What: []string{cloudwire.ResyncTiles}, Tiles: named}) + } + resync() + got := waitTiles(t, svc, 10) + if !got[5].Whole() || !got[5].Partial || got[6].Partial || got[9].TileID != tileID(trailStart-4*tileSpan) { + t.Fatalf("a resync sent %v", tileNames(got[5:])) + } + // A second within five minutes is ignored. + resync() + waitTiles(t, svc, 10) + + // One that names a tile sends that one alone. + h.link.mu.Lock() + h.link.resyncAt = time.Time{} + h.link.mu.Unlock() + resync(tileID(trailStart - 2*tileSpan)) + got = waitTiles(t, svc, 11) + if got[10].TileID != tileID(trailStart-2*tileSpan) || got[10].Partial { + t.Fatalf("a named resync sent %v", tileNames(got[10:])) + } +} + +func TestAReconnectSendsWhatTheServiceLacks(t *testing.T) { + svc := newFakeService(t) + svc.kick = 1 + h, _ := newTileHarness(t, svc, nil) + grantHistory(t, h) + // The first connection drops right after hello_ok, before any tile; the + // second sends the trailing tile and the closed ones. + got := waitTiles(t, svc, 5) + if svc.conns < 2 || !got[0].Whole() || !got[0].Partial { + t.Fatalf("after a reconnect: %d connections, %v", svc.conns, tileNames(got)) + } +} + +func TestTheHourlyPassResendsATileTheLedgerRepaired(t *testing.T) { + old := tilesRecheckEvery + tilesRecheckEvery = 300 * time.Millisecond + t.Cleanup(func() { tilesRecheckEvery = old }) + + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, nil) + grantHistory(t, h) + waitTiles(t, svc, 5) + + // The ledger fills a gap two tiles back. + repaired := trailStart - 2*tileSpan + hs.mu.Lock() + hs.bump, hs.bumpFrom = 7, repaired+tileSpan/2 + hs.mu.Unlock() + // The trailing tile changes too, but it goes only as it closes. + got := waitTiles(t, svc, 7) + again := got[5:] + ids := []string{again[0].TileID, again[1].TileID} + if !slices.Contains(ids, tileID(repaired)) || !slices.Contains(ids, tileID(repaired+tileSpan)) { + t.Fatalf("after the repair %v went", tileNames(again)) + } + time.Sleep(500 * time.Millisecond) + if n := len(tilesSent(t, svc)); n != 7 { + t.Fatalf("an unchanged tile went again: %d", n) + } +} + +func TestNoTilesWithoutTheHistoryScopeAClockOrTheBudget(t *testing.T) { + cases := map[string]struct { + scopes []string + tweak func(*Options) + budget bool + }{ + "live only": {scopes: []string{protocol.ScopeLiveRead}}, + "no clock": {scopes: []string{protocol.ScopeLiveRead, protocol.ScopeHistoryRead}, + tweak: func(o *Options) { o.ClockSynced = func() bool { return false } }}, + "past the budget": {scopes: []string{protocol.ScopeLiveRead, protocol.ScopeHistoryRead}, budget: true}, + } + for name, c := range cases { + t.Run(name, func(t *testing.T) { + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, c.tweak) + if c.budget { + h.link.mu.Lock() + h.link.budget.add(h.link.uptimeMs(), budgetBytes) + h.link.mu.Unlock() + } + h.grant(t, c.scopes...) + svc.waitFor(cloudwire.MsgRoster, 5*time.Second) + time.Sleep(time.Second) + if n := len(tilesSent(t, svc)); n != 0 { + t.Fatalf("%d tiles went", n) + } + if name == "live only" { + hs.mu.Lock() + reads := hs.reads + hs.mu.Unlock() + if reads != 0 { + t.Fatalf("the ledger was read %d times for nothing", reads) + } + } + }) + } +} + +func TestALedgerThatFailsHoldsTilesBackAndNothingElse(t *testing.T) { + svc := newFakeService(t) + h, hs := newTileHarness(t, svc, nil) + hs.failing = true + grantHistory(t, h) + svc.waitFor(cloudwire.MsgLive, 5*time.Second) + svc.waitFor(cloudwire.MsgHB, 20*time.Second) + if n := len(tilesSent(t, svc)); n != 0 { + t.Fatalf("%d tiles went from a failing ledger", n) + } + hs.mu.Lock() + reads := hs.reads + hs.mu.Unlock() + if reads > 2 { + t.Fatalf("a failing ledger was read %d times within its retry", reads) + } +} + +// History taken out of the grant and put back: the service dropped what it +// held, so every tile goes again, as at connect, on the same link. +func TestHistoryTakenOutAndPutBackSendsEveryTileAgain(t *testing.T) { + svc := newFakeService(t) + h, _ := newTileHarness(t, svc, nil) + grantHistory(t, h) + waitTiles(t, svc, 5) + conns := svc.conns + + h.grant(t, protocol.ScopeLiveRead) + time.Sleep(300 * time.Millisecond) + if n := len(tilesSent(t, svc)); n != 5 { + t.Fatalf("%d tiles went without history in the grant", n) + } + grantHistory(t, h) + got := waitTiles(t, svc, 10) + if svc.conns != conns { + t.Fatalf("the link dialled again: %d connections", svc.conns) + } + if !got[5].Whole() || !got[5].Partial || got[6].Partial || got[9].TileID != tileID(trailStart-4*tileSpan) { + t.Fatalf("after history came back: %v", tileNames(got[5:])) + } +} diff --git a/go/internal/cloudlink/tunnel_test.go b/go/internal/cloudlink/tunnel_test.go index 9b44b13c8..7b1400ce9 100644 --- a/go/internal/cloudlink/tunnel_test.go +++ b/go/internal/cloudlink/tunnel_test.go @@ -627,3 +627,101 @@ func TestANoticeGoesPastTheBudget(t *testing.T) { } svc.waitFor(cloudwire.MsgNotice, 3*time.Second) } + +// fakeExchange is the box's exchange, as the link sees it. +type fakeExchange struct{ ch chan cloudwire.X } + +func (f *fakeExchange) Deliver(x cloudwire.X) { f.ch <- x } + +// x goes both ways unsigned and reaches the Exchange unread, and the box says +// x in hello.caps only when it has an Exchange beside its Tunnel. +func TestXGoesBothWaysAndIsSaidOnlyWithAnExchange(t *testing.T) { + svc := newFakeService(t) + svc.afterHello = func(conn uplinkConn, home string) { + svc.write(conn, nil, home, cloudwire.MsgX, cloudwire.X{Dev: "wAtCh123", ID: 4, D: []byte{1, 2, 3}}) + } + exchange := &fakeExchange{ch: make(chan cloudwire.X, 4)} + h := newHarnessWith(t, svc, []*ecdsa.PublicKey{&svc.key.PublicKey}, nil, func(o *Options) { o.Exchange = exchange }) + h.setRoster(func(r *fakeRosterState) { r.tun = true }) + + select { + case got := <-exchange.ch: + if got.Dev != "wAtCh123" || got.ID != 4 || !bytes.Equal(got.D, []byte{1, 2, 3}) { + t.Fatalf("the Exchange got %+v", got) + } + case <-time.After(5 * time.Second): + t.Fatal("no x reached the Exchange") + } + if !h.link.TakesX() { + t.Fatal("a link with an Exchange beside its Tunnel does not take x") + } + var hello cloudwire.Hello + svc.mu.Lock() + payload, _ := cloudwire.DecodeFrame(svc.hello) + svc.mu.Unlock() + env, _ := cloudwire.DecodeEnvelope(payload) + if err := cloudwire.Unmarshal(env.B, &hello); err != nil { + t.Fatal(err) + } + if !slices.Contains(hello.Caps, cloudwire.CapX) { + t.Fatalf("hello caps %v name no x", hello.Caps) + } + + if err := h.link.SendX(cloudwire.X{Dev: "wAtCh123", ID: 4, D: []byte{4, 5}}); err != nil { + t.Fatalf("SendX refused with the uplink up: %v", err) + } + up := svc.waitFor(cloudwire.MsgX, 3*time.Second) + var x cloudwire.X + if err := cloudwire.Unmarshal(up.B, &x); err != nil || x.ID != 4 || x.Dev != "wAtCh123" || !bytes.Equal(x.D, []byte{4, 5}) || len(up.Sig) != 0 { + t.Fatalf("the service got %+v: %v", x, err) + } + + // Without an Exchange the box says no x. + plain := newFakeService(t) + h2 := newHarness(t, plain) + h2.setRoster(func(r *fakeRosterState) { r.tun = true }) + h2.nextRoster(t) + plain.mu.Lock() + payload, _ = cloudwire.DecodeFrame(plain.hello) + plain.mu.Unlock() + env, _ = cloudwire.DecodeEnvelope(payload) + if err := cloudwire.Unmarshal(env.B, &hello); err != nil { + t.Fatal(err) + } + if slices.Contains(hello.Caps, cloudwire.CapX) || h2.link.TakesX() { + t.Fatalf("a box with no Exchange says x: %v", hello.Caps) + } +} + +// x keeps to its share: a tenth of the tunnel's budget, and nothing once the +// tunnel has spent nine tenths, so the phones keep the last of it. Past +// either, an answer may not leave, and the exchange runs nothing. +func TestXKeepsToItsShareOfTheTunnelBudget(t *testing.T) { + svc := newFakeService(t) + exchange := &fakeExchange{ch: make(chan cloudwire.X, 4)} + h := newHarnessWith(t, svc, []*ecdsa.PublicKey{&svc.key.PublicKey}, nil, func(o *Options) { o.Exchange = exchange }) + h.setRoster(func(r *fakeRosterState) { r.tun = true }) + h.nextRoster(t) + if !h.link.XAllowed() || h.link.SendX(cloudwire.X{Dev: "w", ID: 1}) != nil { + t.Fatal("x refused with its share unspent") + } + + h.link.mu.Lock() + h.link.xBudget.add(h.link.uptimeMs(), xBudgetBytes) + h.link.mu.Unlock() + if h.link.XAllowed() || h.link.SendX(cloudwire.X{Dev: "w", ID: 2}) == nil { + t.Fatal("x left past its own share") + } + + h.link.mu.Lock() + h.link.xBudget = budget{} + h.link.tunBudget.add(h.link.uptimeMs(), int64(xStopsAt*tunBudgetBytes)) + h.link.mu.Unlock() + if h.link.XAllowed() { + t.Fatal("x left with the phones' last share of the tunnel") + } + // The phones still have theirs. + if h.link.SendTun(cloudwire.Tun{Dev: "p", Sid: 1, D: []byte{1}}, false) != nil { + t.Fatal("a phone's tun was refused at nine tenths") + } +} diff --git a/go/internal/config/config.go b/go/internal/config/config.go index 8f470691f..731455fdc 100644 --- a/go/internal/config/config.go +++ b/go/internal/config/config.go @@ -452,7 +452,9 @@ type Loadpoint struct { // PhaseMode selects how the controller picks between 1Φ and 3Φ // delivery: "3p" (default) | "1p" | "auto". Empty == "3p" for - // backward compat with pre-switching configs. See loadpoint.Config. + // backward compat with pre-switching configs. "auto" charges on three + // phases and uses one only for surplus charging below the 3Φ minimum. + // See loadpoint.Config. PhaseMode string `yaml:"phase_mode,omitempty" json:"phase_mode,omitempty"` PhaseSplitW float64 `yaml:"phase_split_w,omitempty" json:"phase_split_w,omitempty"` MinPhaseHoldS int `yaml:"min_phase_hold_s,omitempty" json:"min_phase_hold_s,omitempty"` diff --git a/go/internal/config/ocpp_test.go b/go/internal/config/ocpp_test.go index abe403e34..11a5d431f 100644 --- a/go/internal/config/ocpp_test.go +++ b/go/internal/config/ocpp_test.go @@ -1,6 +1,11 @@ package config -import "testing" +import ( + "os" + "path/filepath" + "strings" + "testing" +) // The OCPP listener cannot be pinned to one interface — the library builds its // listen address from the port alone. Basic auth is therefore the only thing @@ -142,3 +147,39 @@ func TestConfigValidateCoversOCPP(t *testing.T) { t.Fatalf("config with a credentialed ocpp section should validate, got %v", err) } } + +// Only the default heartbeat changed (60 s to 30 s). An unset value must stay +// unset through a save, and a saved 60 must stay 60. +func TestOCPPHeartbeatSurvivesSave(t *testing.T) { + for _, tc := range []struct { + yaml string + want int + }{ + {"ocpp:\n enabled: false\n", 0}, + {"ocpp:\n enabled: false\n heartbeat_interval_s: 60\n", 60}, + } { + dir := t.TempDir() + path := filepath.Join(dir, "c.yaml") + c, err := Parse([]byte(minimalYAML+tc.yaml), dir) + if err != nil { + t.Fatal(err) + } + if err := SaveAtomic(path, c); err != nil { + t.Fatal(err) + } + saved, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if tc.want == 0 && strings.Contains(string(saved), "heartbeat_interval_s") { + t.Fatalf("an unset heartbeat was written:\n%s", saved) + } + c2, err := Load(path) + if err != nil { + t.Fatal(err) + } + if c2.OCPP == nil || c2.OCPP.HeartbeatIntervalS != tc.want { + t.Fatalf("heartbeat after save: got %+v, want %d", c2.OCPP, tc.want) + } + } +} diff --git a/go/internal/control/control_test.go b/go/internal/control/control_test.go index ab4260693..44ab142ed 100644 --- a/go/internal/control/control_test.go +++ b/go/internal/control/control_test.go @@ -1656,8 +1656,8 @@ func TestJointFuseAllocatorScalesBothBatteryAndEV(t *testing.T) { if got < 0 { t.Errorf("TargetW = %.0f W — battery shouldn't be discharging here", got) } - if !st.FuseSaturated { - t.Errorf("FuseSaturated = false; want true after joint scaling") + if !st.FuseSaturated || st.FuseSaturatedByImport { + t.Errorf("FuseSaturated = %v, by import %v; want the fuse after joint scaling", st.FuseSaturated, st.FuseSaturatedByImport) } // EV cap is the post-republish verdict: after applyFuseGuard + // forceFuseDischarge constrain battery further (and may swing it @@ -1676,6 +1676,23 @@ func TestJointFuseAllocatorScalesBothBatteryAndEV(t *testing.T) { } } +// The same site under a 9 kW hourly import ceiling: the hour's limit, not +// the fuse, sets the ceiling the allocator hits, and says so. +func TestJointFuseAllocatorNamesTheHoursImportLimit(t *testing.T) { + now := time.Now() + dir := SlotDirective{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), BatteryEnergyWh: 1250, Strategy: "arbitrage"} + store := seedStore(8200, []struct { + name string + currentW, soc float64 + }{{"ferroamp", 0, 0.5}}) + st := newStateWithEnergyDispatch(dir, "ferroamp") + st.EVChargingW, st.BatteryCoversEV, st.PeakImportCeilingW = 8000, true, 9000 + ComputeDispatch(store, st, caps(map[string]float64{"ferroamp": 15200}), 11000) + if !st.FuseSaturated || !st.FuseSaturatedByImport { + t.Fatalf("FuseSaturated = %v, by import %v; want the hour's import limit", st.FuseSaturated, st.FuseSaturatedByImport) + } +} + // Counter-test: when battery wants to discharge, no scaling — discharge // helps the fuse rather than competing with the EV. func TestJointFuseAllocatorIgnoresDischarge(t *testing.T) { @@ -3498,6 +3515,43 @@ func TestInverterAffinity_FallsBackToProportionalOnOverflow(t *testing.T) { } } +// PV on an inverter whose battery cannot charge is not DC-local to any +// battery that can. It reaches them over the AC bus, so the fleet total +// is split by capacity and nothing is lost. Before, that group's PV share +// of the total went to no battery (golden seeded_siblings/039 lost 3,316 W +// of a 9,600 W target). +func TestInverterAffinity_PVWithoutItsBatteryKeepsTheTotal(t *testing.T) { + online := []batteryInfo{ + {driver: "sungrow", capacityWh: 15200, soc: 0.5, online: true, group: "sg"}, + {driver: "pixii", capacityWh: 15200, soc: 0.5, online: true, group: "px"}, + } + blocked := append([]batteryInfo{ + {driver: "ferroamp", capacityWh: 10000, soc: 0.5, online: true, group: "fa", chargeBlocked: true}, + }, online...) + for _, tc := range []struct { + name string + bats []batteryInfo + targetW float64 + wantEach float64 + }{ + {"battery offline", online, 6000, 3000}, + {"battery cannot charge", blocked, 6000, 3000}, + // Below the down group's PV, master gave the whole target to it: 0 W each. + {"target below that PV", online, 2000, 1000}, + } { + t.Run(tc.name, func(t *testing.T) { + groupPV := map[string]float64{"fa": 3316} + got := map[string]float64{} + for _, tg := range distributeProportional(tc.bats, tc.targetW, groupPV) { + got[tg.Driver] = tg.TargetW + } + if math.Abs(got["sungrow"]-tc.wantEach) > 1 || math.Abs(got["pixii"]-tc.wantEach) > 1 || got["ferroamp"] != 0 { + t.Errorf("targets = %v, want sungrow and pixii %.0f each, ferroamp 0", got, tc.wantEach) + } + }) + } +} + // With no inverter groups configured, the algorithm must produce // identical results to today's capacity-proportional split — the // backward-compat invariant. diff --git a/go/internal/control/dispatch.go b/go/internal/control/dispatch.go index 28b5ca5a8..388e73afc 100644 --- a/go/internal/control/dispatch.go +++ b/go/internal/control/dispatch.go @@ -551,6 +551,9 @@ type State struct { // loadpoint controller's read of FuseEVMaxW and as the trigger for // an MPC reactive replan from main.go. FuseSaturated bool + // FuseSaturatedByImport says the clock hour's import limit, not the + // fuse, set the ceiling FuseSaturated hit. + FuseSaturatedByImport bool // DriverLimits maps driver name → per-battery charge/discharge cap. // Missing entries and unset fields fall through to the global @@ -1140,6 +1143,7 @@ type batteryInfo struct { // batteryDirectionBlocks + distributeProportional. dischargeBlocked bool chargeBlocked bool + socMissing bool // no fresh SoC reading; soc is 0 } // chargeCap returns the effective per-battery charge ceiling, falling @@ -1474,7 +1478,8 @@ func ComputeDispatch( soc := 0.0 lim := state.DriverLimits[name] dischargeBlocked, chargeBlocked := batteryDirectionBlocks(r.Data) - if r.SoC != nil { + socMissing := r.SoC == nil + if !socMissing { soc = *r.SoC } else { // Never reported SoC: do not discharge a pack we cannot @@ -1499,6 +1504,7 @@ func ComputeDispatch( maxDischargeWSet: lim.MaxDischargeWSet, dischargeBlocked: dischargeBlocked, chargeBlocked: chargeBlocked, + socMissing: socMissing, }) } onlineBats := make([]batteryInfo, 0, len(batteries)) @@ -2040,7 +2046,7 @@ func ComputeDispatch( // The ceiling gives up the worst phase's imbalance: a charger's own load // balancer cuts the car on that phase while the total still fits. state.FuseEVMaxW = 0 - state.FuseSaturated = false + state.FuseSaturated, state.FuseSaturatedByImport = false, false plannedEVW := scheduledEVPowerW(state) evInMeterW := state.evInMeterW() if fuseMaxW > 0 && math.Max(evInMeterW, plannedEVW) > 0 { @@ -2069,7 +2075,7 @@ func ComputeDispatch( totalCorrection = Bn + B*scale - currentTotal state.FuseEVMaxW = E * scale } - state.FuseSaturated = true + state.FuseSaturated, state.FuseSaturatedByImport = true, state.importLimitSets(ceilingW) } } @@ -3081,8 +3087,10 @@ func distributeProportional(bats []batteryInfo, totalCorrection float64, groupPV // same split over a subset of batteries. func splitByCapacityAndPV(bats []batteryInfo, desiredTotal float64, groupPV map[string]float64) []DispatchTarget { var totalCap float64 + capByGroup := map[string]float64{} for _, b := range bats { totalCap += b.capacityWh + capByGroup[b.group] += b.capacityWh } if totalCap <= 0 { return nil @@ -3091,9 +3099,15 @@ func splitByCapacityAndPV(bats []batteryInfo, desiredTotal float64, groupPV map[ // Discharge, idle, or no PV locality info → capacity-only split. // Discharge energy flows to the AC bus regardless of where it // originated, so DC-locality has no win for the negative branch. + // PV in a group with no battery here (offline, faulted or unable to + // charge) is not DC-local to any of them; it reaches them over the + // AC bus, so it stays out of the local pool. Counting it would hand + // its share of the total to no battery. var totalPV float64 - for _, w := range groupPV { - totalPV += w + for g, w := range groupPV { + if capByGroup[g] > 0 { + totalPV += w + } } if desiredTotal <= 0 || totalPV <= 0 { return distributeByCapacity(bats, desiredTotal, totalCap) @@ -3112,11 +3126,6 @@ func splitByCapacityAndPV(bats []batteryInfo, desiredTotal float64, groupPV map[ localCap := math.Min(desiredTotal, totalPV) overflow := desiredTotal - localCap - capByGroup := map[string]float64{} - for _, b := range bats { - capByGroup[b.group] += b.capacityWh - } - out := make([]DispatchTarget, 0, len(bats)) for _, b := range bats { var localShare float64 @@ -3524,6 +3533,9 @@ func clampTargetsToPowerLimits(targets []DispatchTarget, limits map[string]Power return targets } +// dischargeFloorSoC is where Core stops a battery's discharge on its own. +const dischargeFloorSoC = 0.05 + // clampWithSoC applies the hard safety clamps for one battery command: // - don't discharge below SoC 5 % (site: don't make target < 0 when SoC < 0.05); // BMS handles fine-grained SoC but we never ask it to pull an empty pack. @@ -3537,7 +3549,7 @@ func clampWithSoC(target float64, b batteryInfo) (float64, bool) { clamped := target wasClamped := false // Block discharge when the battery is empty. - if b.soc < 0.05 && target < 0 { + if b.soc < dischargeFloorSoC && target < 0 { clamped = 0 wasClamped = true } @@ -3772,6 +3784,13 @@ func (s *State) fuseSafetyMarginW() float64 { return s.SiteFuseSafetyA * s.SiteFuseVoltage * float64(s.SiteFusePhases) } +// importLimitSets reports whether the clock hour's import limit is the +// ceiling ceilingW, rather than the fuse. +func (s *State) importLimitSets(ceilingW float64) bool { + limitW, _, ok := s.importHourLimitsW() + return ok && limitW <= ceilingW +} + // effectiveImportCeilingW returns the binding ceiling for grid import in // watts: the fuse limit minus its safety margin, further capped by the // clock hour's import limit when the operator has opted into a tariff @@ -4081,7 +4100,7 @@ func fuseTargetBounds(r *telemetry.DerReading, lim PowerLimits) (lower, upper fl lower = -lim.dischargeCap() upper = lim.chargeCap() dischargeBlocked, chargeBlocked := batteryDirectionBlocks(r.Data) - if r.SoC == nil || *r.SoC < 0.05 || dischargeBlocked { + if r.SoC == nil || *r.SoC < dischargeFloorSoC || dischargeBlocked { lower = 0 } if chargeBlocked { diff --git a/go/internal/control/ev_battery_sequence_test.go b/go/internal/control/ev_battery_sequence_test.go index 01eed478c..8100ee000 100644 --- a/go/internal/control/ev_battery_sequence_test.go +++ b/go/internal/control/ev_battery_sequence_test.go @@ -53,13 +53,16 @@ func TestJoinedEVWaitsForMeasuredBatteryBackoffThenResumes(t *testing.T) { }) controller.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) controller.SetManualHold(config.ID, loadpoint.ManualHold{PowerW: desiredEVW, PhaseMode: "3p", Persistent: true}) - controller.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return sampledPhaseA, sampledPhaseA, sampledPhaseA, true }) - controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(store, st, mgr.States(), mgr.Configs(), 17250) }) + controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(store, st, mgr.States(), controller.FuseConfigs(), 17250) }) + controller.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(st) }) emit := func(driver string, kind telemetry.DerType, watts float64) { d := map[string]any{"control_power_available": true, "control_power_observed_at": now.Format(time.RFC3339Nano)} if kind == telemetry.DerMeter { d["l1_a"], d["l2_a"], d["l3_a"] = sampledPhaseA, sampledPhaseA, sampledPhaseA } + if kind == telemetry.DerEV { + d["power_max_age_s"] = 5 // A local charger that answers within seconds. + } data, _ := json.Marshal(d) soc := .5 store.Update(driver, kind, watts, &soc, data) @@ -216,11 +219,10 @@ func newEVBatterySiteAt(t *testing.T, houseW float64, start time.Time) *evBatter }) s.ctrl.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) s.ctrl.SetManualHold(config.ID, loadpoint.ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - s.ctrl.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { - a := s.gridW() / 690 - return a, a, a, true + s.ctrl.SetFuseEVMax(func() (float64, bool) { + return EVFuseBudget(s.store, s.state, s.mgr.States(), s.ctrl.FuseConfigs(), 17250) }) - s.ctrl.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(s.store, s.state, s.mgr.States(), s.mgr.Configs(), 17250) }) + s.ctrl.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(s.state) }) return s } diff --git a/go/internal/control/ev_deadline_runtime_test.go b/go/internal/control/ev_deadline_runtime_test.go index ff2950f04..1f4f12047 100644 --- a/go/internal/control/ev_deadline_runtime_test.go +++ b/go/internal/control/ev_deadline_runtime_test.go @@ -75,8 +75,8 @@ func TestJoinedNoBatteryNoPlanGoalContinuity(t *testing.T) { return nil }) controller.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - controller.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return phaseA, phaseA, phaseA, true }) - controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(tel, st, mgr.States(), mgr.Configs(), 17250) }) + controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(tel, st, mgr.States(), controller.FuseConfigs(), 17250) }) + controller.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(st) }) controller.TickWithDispatch(ctx, now, true) if len(commands) == 0 || commands[len(commands)-1] <= 0 { t.Fatalf("goal stopped at integration boundary: %v", commands) @@ -172,9 +172,8 @@ func TestJoinedMixedPhaseNoCreditBeforeMeasuredReduction(t *testing.T) { return nil }) controller.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - var sampledWorst float64 - controller.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return sampledWorst, actualB / 690, actualB / 690, true }) - controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(tel, st, mgr.States(), mgr.Configs(), 17250) }) + controller.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(tel, st, mgr.States(), controller.FuseConfigs(), 17250) }) + controller.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(st) }) emit := func(driver string, kind telemetry.DerType, at time.Time, w float64, amps bool) { d := map[string]any{"control_power_observed_at": at.Format(time.RFC3339Nano), "control_power_available": true} if amps { @@ -192,7 +191,6 @@ func TestJoinedMixedPhaseNoCreditBeforeMeasuredReduction(t *testing.T) { at := wall.Add(-9*time.Second + time.Duration(step)*1500*time.Millisecond) emit(cfgA.DriverName, telemetry.DerEV, at, actualA, false) emit(cfgB.DriverName, telemetry.DerEV, at, actualB, false) - sampledWorst = actualA/230 + actualB/690 emit("meter", telemetry.DerMeter, at.Add(500*time.Millisecond), actualA+actualB, true) emit(cfgA.DriverName, telemetry.DerEV, at.Add(time.Second), actualA, false) emit(cfgB.DriverName, telemetry.DerEV, at.Add(time.Second), actualB, false) diff --git a/go/internal/control/ev_fuse_budget.go b/go/internal/control/ev_fuse_budget.go index ff0755b1d..c9f194f2b 100644 --- a/go/internal/control/ev_fuse_budget.go +++ b/go/internal/control/ev_fuse_budget.go @@ -20,9 +20,14 @@ func evHeadroomW(ceiling, gridW, poolW float64) float64 { func finiteFuseValue(w float64) bool { return !math.IsNaN(w) && !math.IsInf(w, 0) } -// evSettle is how long one of our commands may take to reach the site meter. +// evSettle is how long one of our commands may take to reach the site meter +// when the device declares nothing longer. const evSettle = loadpoint.CommandSettle +// evUnknownSettle is how long an EV rise counts as flowing when the charger +// declares no response time: the slowest we know, Easee Cloud's 180 s. +const evUnknownSettle = 3 * time.Minute + // evRestartDwell is how long the meter must keep showing room before an idle // pool starts again. It stays above Easee's ~30 s contactor minimum. const evRestartDwell = 35 * time.Second @@ -32,10 +37,18 @@ const evRestartDwell = 35 * time.Second // and rise thresholds. const evPhaseMarginA = 1.0 +// fuseHourFactor is the phase current, as a multiple of the fuse rating, +// that a gG fuse of 16-63 A carries for an hour. A cut on its way covers the +// overage it was sent for only while every phase's filtered current stays +// at or below it. +const fuseHourFactor = 1.25 + // sentCommand is one of our own commands, never a device reading. type sentCommand struct { at time.Time w float64 + a float64 // EV offers: current on each phase it may land on + n int // EV offers: the phases it lands on, 1 or the site's } // commandLog keeps a device's commands for evSettle, plus the one in force @@ -43,9 +56,14 @@ type sentCommand struct { type commandLog []sentCommand func (l commandLog) add(now time.Time, w float64) commandLog { - l = append(l, sentCommand{now, w}) + return l.keep(sentCommand{at: now, w: w}, now.Add(-evSettle)) +} + +// keep appends c and drops the commands replaced before from. +func (l commandLog) keep(c sentCommand, from time.Time) commandLog { + l = append(l, c) i := 0 - for i+1 < len(l) && now.Sub(l[i+1].at) >= evSettle { + for i+1 < len(l) && !l[i+1].at.After(from) { i++ } if i > 0 { @@ -76,20 +94,234 @@ func (l commandLog) span(now time.Time) (low, high, latest float64, ok bool) { return low, high, latest, ok } +// unshown is how much more a charger may draw than a reading taken at `at` +// shows, in watts and in current per phase: the highest offer sent since +// settle before the reading and not replaced by a lower one, less the +// lowest offer the reading may show. An offer set before that window is in +// the reading. A cut needs nothing: the reading shows the draw until the +// cut lands, and the charger moves to its latest order, so a cut also ends +// the reservation of the higher offers before it. Before our first offer +// the charger counts as 0. +func (l commandLog) unshown(at time.Time, settle time.Duration) (w, a float64) { + from := at.Add(-settle) + var settledW, settledA float64 + lowW, lowA, highW, highA := math.Inf(1), math.Inf(1), math.Inf(-1), math.Inf(-1) + for _, c := range l { + if !c.at.After(from) { + settledW, settledA = c.w, c.a + continue + } + if c.w < highW { + highW = c.w + } + if c.a < highA { + highA = c.a + } + highW, highA = math.Max(highW, c.w), math.Max(highA, c.a) + if !c.at.After(at) { + lowW, lowA = math.Min(lowW, c.w), math.Min(lowA, c.a) + } + } + lowW, lowA = math.Min(lowW, settledW), math.Min(lowA, settledA) + return math.Max(0, highW-lowW), math.Max(0, highA-lowA) +} + +// toCome is what a charger's latest offer will still change in a reading +// taken at `at`, as a cut counts it, in watts and in current per phase: +// positive is a raise on its way, negative a cut on its way. The reading may +// show the offer in force at from or one sent since, up to `at`, but a cut +// and every offer after it count as not shown yet. Of the offers before the +// first cut, a raise counts from the lowest. An offer on other phases than +// the latest says nothing of the latest's phases, so the latest then counts +// in full as a raise on each. Before the first logged offer the charger +// counts as before. +func (l commandLog) toCome(from, at time.Time, before sentCommand) (w, a float64) { + settled, latest := before, before + for _, c := range l { + latest = c + if !c.at.After(from) { + settled = c + } + } + low, prev, cutW, cutA := settled, settled, false, false + otherPhases := settled.a > 0 && settled.n != latest.n + for _, c := range l { + if !c.at.After(from) || c.at.After(at) { + continue + } + otherPhases = otherPhases || c.a > 0 && c.n != latest.n + cutW, cutA = cutW || c.w < prev.w, cutA || c.a < prev.a + if !cutW { + low.w = math.Min(low.w, c.w) + } + if !cutA { + low.a = math.Min(low.a, c.a) + } + prev = c + } + if otherPhases { + low.a = 0 + } + return latest.w - low.w, latest.a - low.a +} + +// chargerOffers is one charger's offers, kept for the time its driver +// declares its power takes to show. +type chargerOffers struct { + log commandLog + settle time.Duration + // found is the offer in force before the limit's first site reading, + // after a restart: Core did not send it in this run. A rise counts the + // charger as 0 before it; a cut takes it as shown, since the reading is + // all Core knows of it. + found sentCommand +} + +// pendingLoad is load we ordered that a site reading may not show yet: +// each EV offer sent above what the reading may show, until its charger +// can have applied it. A single-phase offer may land on any phase. Rises +// and starts count it on top of the meter; a cut never waits for it. +// stepEVMeterLimit adds our unshown battery orders (withBatteryOrders). +type pendingLoad struct { + w float64 // in total import + phaseA [3]float64 // on each phase +} + +func (s *State) pendingLoad(at time.Time) pendingLoad { + var p pendingLoad + for _, c := range s.evLimit.offers { + w, a := c.log.unshown(at, c.settle) + p.w += w + for i := range p.phaseA { + p.phaseA[i] += a + } + } + return p +} + +// unshownOrders is what a cut counts beside a reading taken at `at`: rise, +// the load our latest offers add that the reading may not show yet, and +// cover, the load it may still show that our cuts on their way take off +// (toCome). latestW is the latest offers. A cut covers only within the run +// over the limit it was sent in, for its charger's response time: once the +// meter has shown every phase under the limit for evSettle, a cut sent +// before counts as shown. Once the run's first cut has outlasted that time, a cut covers for +// evSettle, as before; so it does without phase currents, where Core +// cannot see the hottest phase. A single-phase cut covers on the phase +// it lands on, which Core does not know, so with other chargers in the pool +// it covers no phase. +func (s *State) unshownOrders(at time.Time, perPhase bool) (rise, cover pendingLoad, latestW float64) { + lim := &s.evLimit + for _, c := range lim.offers { + w, a := c.log.toCome(at.Add(-c.settle), at, c.found) + coverFor := c.settle + if !perPhase || !lim.fuseCutAt.IsZero() && at.Sub(lim.fuseCutAt) > c.settle { + coverFor = evSettle + } + from := at.Add(-coverFor) + if lim.fuseOverAt.After(from) { + from = lim.fuseOverAt + } + runW, runA := c.log.toCome(from, at, c.found) + coverA := math.Max(0, math.Min(-a, -runA)) + var latest sentCommand + if n := len(c.log); n > 0 { + latest = c.log[n-1] + } + if latest.n == 1 && len(lim.offers) > 1 { + coverA = 0 + } + rise.w += math.Max(0, w) + cover.w += math.Max(0, math.Min(-w, -runW)) + for i := range rise.phaseA { + rise.phaseA[i] += math.Max(0, a) + cover.phaseA[i] += coverA + } + latestW += latest.w + } + return rise, cover, latestW +} + +// recordEVOffers keeps each plugged charger's offers, with the current each +// puts on the phases its command names, for the longer of evSettle and the +// power_max_age_s its driver declares, or evUnknownSettle when it declares +// none. Our own commands only: a charger's power reading never counts. +func (s *State) recordEVOffers(store *telemetry.Store, points []loadpoint.State) { + now := s.now() + read := now + if !s.evLimit.readingAt.IsZero() && s.evLimit.readingAt.Before(now) { + read = s.evLimit.readingAt + } + offers := make(map[string]chargerOffers, len(points)) + for _, point := range points { + if !point.PluggedIn || !point.CommandedKnown || !finiteFuseValue(point.CommandedW) { + continue + } + c := s.evLimit.offers[point.ID] + c.settle = evUnknownSettle + if r := store.Get(point.DriverName, telemetry.DerEV); r != nil { + if declared := telemetry.DeclaredPowerMaxAge(r.Data); declared > 0 { + c.settle = max(evSettle, declared) + } + } + w := math.Max(0, point.CommandedW) + a, phases := s.offerPhaseA(w, point.CommandedPhases) + sent := sentCommand{at: now, w: w, a: a, n: phases} + if n := len(c.log); n == 0 || c.log[n-1].w != sent.w || c.log[n-1].a != sent.a || c.log[n-1].n != sent.n { + if n == 0 && !s.evLimit.set { + c.found = sent + } + c.log = c.log.keep(sent, read.Add(-c.settle)) + } + offers[point.ID] = c + } + s.evLimit.offers = offers +} + +// offerPhaseA is the current an offer of w puts on each phase it may land +// on, and how many phases it lands on. Unknown phases count as one. +func (s *State) offerPhaseA(w float64, phases int) (float64, int) { + n := 1 + if phases == 3 && s.SiteFusePhases > 1 { + n = s.SiteFusePhases + } + if s.SiteFuseVoltage <= 0 { + return 0, n + } + return w / (s.SiteFuseVoltage * float64(n)), n +} + +// EVFusePhaseRoomA is the current the chargers may still add on the +// busiest phase: below the fuse less its margin, the last reading's +// current or heat, and every charger's offers that reading may not show. +// false when the site meter reports no phase currents. +func EVFusePhaseRoomA(state *State) (float64, bool) { + if state == nil || !state.evLimit.set || !state.evLimit.phaseRoomKnown { + return 0, false + } + p := state.pendingLoad(state.evLimit.readingAt) + return state.evLimit.phaseRoomA - max(p.phaseA[0], p.phaseA[1], p.phaseA[2]), true +} + // evMeterLimit is the managed EVs' common hard cap. It reads only the site // meter, which measures the protected quantities, and our own commands. No // charger, car or battery reading takes part in a cut, a rise or a start. type evMeterLimit struct { - set bool - capW float64 - changedAt time.Time // last cut, rise or idle level - readingAt time.Time // newest site reading seen - roomSince time.Time // first reading of the current run with room to start - raised bool - raisedGridW float64 // site import when the last rise was sent - overSince time.Time // first reading of a run over the hour's import limit - overOffers map[string]float64 // each charger's offer when that run began - idleWh float64 // import since roomSince, for the ceiling's start test + set bool + capW float64 + cutAt time.Time // last cut + readingAt time.Time // newest site reading seen + roomSince time.Time // first reading of the current run with room to start + offers map[string]chargerOffers // each plugged charger's, by loadpoint + phaseRoomA float64 // below the working limit on the busiest phase at readingAt + phaseRoomKnown bool // the meter reports phase currents + overSince time.Time // first reading of a run over the hour's import limit + fuseOverAt time.Time // first reading of a run with a phase, or the total, over the fuse's working limit + fuseCutAt time.Time // the first cut of that run + fuseUnderAt time.Time // first reading of a stretch under that limit + overOffers map[string]float64 // each charger's offer when that run began + idleWh float64 // import since roomSince, for the ceiling's start test + byImport bool // the hour's import limit, not the fuse, left less room at readingAt } type evPool struct { @@ -103,10 +335,18 @@ type evPool struct { // shared freshness gate must already allow dispatch. A fuse overage in total // import cuts at once; a phase overage cuts once that phase's heat budget is // spent, or at once above fuseInstantFactor. Otherwise the cap holds; it -// rises one step only on a fresh reading with room for two, once the meter -// has shown the last change. An idle pool starts only after the meter has -// kept showing room for its minimum plus a step. Each reading acts once, so -// a silent meter can never raise or restart a charge. +// rises one step on a fresh reading with room for two, at least evSettle +// after the last cut. Room counts every +// offer sent above what the reading may show as flowing until its charger +// can have applied it; a cut never frees room before the meter shows it. +// A cut counts our orders both ways: a raise on its way adds to the overage, +// and a cut on its way covers the overage it was sent for until its charger +// can have applied it, within its run over the limit and while the phases +// stay at or below fuseHourFactor (unshownOrders). +// EVFusePhaseRoomA gives the same room per phase to the split between cars. +// An idle pool starts only after the meter has kept showing +// room for its minimum plus a step. Each reading acts once, so a silent +// meter can never raise or restart a charge. func EVFuseBudget(store *telemetry.Store, state *State, points []loadpoint.State, configs []loadpoint.Config, fuseMaxW float64) (float64, bool) { if store == nil || state == nil || !finiteFuseValue(fuseMaxW) || fuseMaxW <= 0 { return 0, true @@ -129,6 +369,7 @@ func EVFuseBudget(store *telemetry.Store, state *State, points []loadpoint.State if pool.offerKnown { state.evOffers = state.evOffers.add(state.now(), pool.offerW) } + state.recordEVOffers(store, points) if !lim.set || meter.UpdatedAt.After(lim.readingAt) { state.stepEVMeterLimit(meter, gridW, pool, fuseMaxW) } @@ -149,13 +390,55 @@ func (s *State) stepEVMeterLimit(meter *telemetry.DerReading, gridW float64, poo if _, high, _, ok := s.evOffers.span(now); ok { peakW = math.Max(peakW, high) } - r := s.evMeterRoom(meter, gridW, pool.phases, fuseMaxW) + // Our own battery orders the meter may not show yet count beside the + // chargers' offers, in total import and on each phase. + pending := s.pendingBatteryCommandsW(now) + r := s.evMeterRoom(meter, gridW, pool.phases, fuseMaxW, s.withBatteryOrders(s.pendingLoad(at), pending)) + _, fresh := qualifiedFuseObservation(meter, now) + fresh = fresh && r.measured + lim.phaseRoomA, lim.phaseRoomKnown = 0, r.phaseRoomKnown + if fresh { + lim.phaseRoomA = r.phaseRoomA + } step := evPhaseMarginA * s.SiteFuseVoltage * pool.phases - // The fuse cuts on every reading, by its heat budget where the meter - // reports phase currents. The hour's budget absorbs a short overage of - // less than a step: it cuts once the overage has lasted evSettle, so one - // reading over the ceiling changes nothing. - cutW := r.fuseCutW + // The label follows whichever limit leaves less room. + lim.byImport = math.Min(r.limitW, r.riseW) < r.fuseW + // The fuse cuts on every reading over it, by its heat budget where the + // meter reports phase currents. It takes off the latest offers only what + // the cuts the reading may still show do not cover, plus the rises it may + // not show yet. Over the fuse, the cap never rises. The hour's budget + // absorbs a short overage of less than a step: it cuts once the overage + // has lasted evSettle, so one reading over the ceiling changes nothing. + // A run over the fuse's working limit ends once every phase has read + // under it for evSettle; a short dip, as battery relief gives, does not. + switch { + case r.overLimit: + lim.fuseUnderAt = time.Time{} + if lim.fuseOverAt.IsZero() || gap { + lim.fuseOverAt, lim.fuseCutAt = at, time.Time{} + } + case lim.fuseUnderAt.IsZero(): + lim.fuseUnderAt = at + case at.Sub(lim.fuseUnderAt) >= evSettle: + lim.fuseOverAt, lim.fuseCutAt = time.Time{}, time.Time{} + } + rise, cover, latestW := s.unshownOrders(at, r.perPhase) + fuseCutW := r.fuseOverW + rise.w - cover.w + if r.perPhase { + // A single-phase order may land on any phase, so the cut takes the + // most a rise adds to a phase and the least a cut takes off one. + // Past the current a gG fuse carries for an hour, nothing on its + // way holds back a cut. + coverA := min(cover.phaseA[0], cover.phaseA[1], cover.phaseA[2]) + if s.fuseHeat.worstA() > fuseHourFactor*s.SiteFuseAmps { + coverA = 0 + } + fuseCutW = (r.fuseOverA + max(rise.phaseA[0], rise.phaseA[1], rise.phaseA[2]) - coverA) * s.SiteFuseVoltage * pool.phases + } + over, capW, byImport := r.fuseOverW > 0, math.Inf(1), false + if over { + capW = latestW - math.Max(0, fuseCutW) + } if r.limitW >= 0 { lim.overSince = time.Time{} } else { @@ -163,30 +446,33 @@ func (s *State) stepEVMeterLimit(meter *telemetry.DerReading, gridW float64, poo lim.overSince, lim.overOffers = at, pool.offers } if !r.hourly || gap || -r.limitW >= step || at.Sub(lim.overSince) >= evSettle { - cutW = math.Min(cutW, r.limitW) + // The meter may still show any offer of the last evSettle. + // Taking the overage from the highest cuts each measured watt once. + hourW := peakW + r.limitW + byImport, capW, over = !over || hourW < capW, math.Min(capW, hourW), true } } - if cutW < 0 { - // The meter may still show any offer of the last evSettle. Taking - // the overage from the highest cuts each measured watt once. - lim.capW, lim.roomSince = math.Min(prev, math.Max(0, peakW+cutW)), time.Time{} + if over { + lim.byImport = byImport + lim.capW, lim.roomSince = math.Min(prev, math.Max(0, capW)), time.Time{} if lim.capW < prev { - lim.changedAt, lim.raised = now, false + lim.cutAt = now + if r.fuseOverW > 0 && lim.fuseCutAt.IsZero() { + lim.fuseCutAt = now + } slog.Info("EV hard limit: site meter over limit", "grid_w", gridW, "worst_phase_a", r.worstA, "fuse_heat_a", s.fuseHeat.worstA(), "import_limit_w", s.effectiveImportCeilingW(fuseMaxW), - "overage_w", -cutW, "offer_w", peakW, "cap_w", lim.capW) + "overage_w", r.fuseOverW, "import_overage_w", math.Max(0, -r.limitW), + "unshown_rise_w", rise.w, "unshown_cut_w", cover.w, "offer_w", latestW, "cap_w", lim.capW) } return } // An increase needs a fresh reading and room below the level increases // may reach, left after our own battery commands that the meter may not - // show yet. Between that level and the import limit the cap holds. - pending := s.pendingBatteryCommandsW(now) - _, fresh := qualifiedFuseObservation(meter, now) - fresh = fresh && r.measured + // show yet (in r). Between that level and the import limit the cap holds. room := 0.0 if fresh { - room = math.Max(0, math.Min(r.fuseW, r.riseW)-pending) + room = math.Max(0, math.Min(r.fuseW, r.riseW)) } if pool.offerW <= 0 { s.stepEVIdleStart(r, gridW, pool, first, gap, fresh, at, prevAt, pending, step) @@ -195,17 +481,10 @@ func (s *State) stepEVMeterLimit(meter *telemetry.DerReading, gridW float64, poo // Keeping the current offers is never a cut. Headroom above them stays // within what this reading measured. lim.capW, lim.roomSince = math.Min(prev, peakW+room), time.Time{} - if first { - lim.changedAt = now - return - } - // The meter shows the last change once import rose with the step, or - // in any reading taken evSettle after it. - seen := at.Sub(lim.changedAt) >= evSettle || - (lim.raised && at.After(lim.changedAt) && gridW >= lim.raisedGridW+step/2) - if lim.capW < pool.wantW && room >= 2*step && seen { - lim.capW += step - lim.changedAt, lim.raised, lim.raisedGridW = now, true, gridW + // Right after a cut the meter may show battery relief that covers it. + if lim.capW < pool.wantW && room >= 2*step && now.Sub(lim.cutAt) >= evSettle { + // One step at most, and a step of room stays free. + lim.capW = math.Max(lim.capW, math.Min(lim.capW+step, peakW+room-step)) } } @@ -216,14 +495,18 @@ func (s *State) stepEVMeterLimit(meter *telemetry.DerReading, gridW float64, poo // the minimum needs no step of margin there: a short overage waits. func (s *State) stepEVIdleStart(r evMeterRoom, gridW float64, pool evPool, first, gap, fresh bool, at, prevAt time.Time, pending, step float64) { lim := &s.evLimit - lim.capW, lim.changedAt, lim.raised = 0, s.now(), false + lim.capW = 0 perReadingW := r.fuseW if !r.hourly { perReadingW = math.Min(perReadingW, r.riseW) + } else { + // The start test below holds an idle car to the ceiling itself, + // whatever slack the hour has: name it by this reading. + lim.byImport = r.startLevelW-gridW-pending < r.fuseW } room := 0.0 if fresh { - room = math.Max(0, perReadingW-pending) + room = math.Max(0, perReadingW) // r holds our battery orders. } if room < pool.minW+step { lim.roomSince = time.Time{} @@ -248,30 +531,45 @@ func (s *State) stepEVIdleStart(r evMeterRoom, gridW float64, pool evPool, first } ceilingRoomW := r.startLevelW - lim.idleWh/dwell.Hours() - pending if ceilingRoomW < pool.minW { - lim.capW, lim.roomSince, lim.idleWh = 0, at, 0 // Judge the next dwell. + lim.capW, lim.roomSince, lim.idleWh, lim.byImport = 0, at, 0, true // Judge the next dwell. return } lim.capW = math.Min(lim.capW, ceilingRoomW) } +// EVLimitReason names the limit behind the managed EVs' cap for the +// loadpoints' reason: "import_limit" when the clock hour's import ceiling +// left less room than the fuse at the last site reading, else "fuse_limit". +func EVLimitReason(state *State) string { + if state != nil && state.evLimit.set && state.evLimit.byImport { + return "import_limit" + } + return "fuse_limit" +} + // evMeterRoom is the measured room in EV watts. The fuse's part holds on // every reading. The import ceiling's parts are +Inf while it is off. type evMeterRoom struct { - fuseW float64 // below the fuse less its margin and the worst phase, read or heat; rises and starts - fuseCutW float64 // what the fuse needs cut now, negative; see fuseHeat - limitW float64 // below the hour's import limit; negative is an overage - riseW float64 // below the level an increase may reach - startLevelW float64 // import an idle start may reach: the rise level, at most the ceiling - hourly bool // the hour's import is measured, so a short overage may wait - measured bool // false when the meter reports phase currents that cannot be read - worstA float64 + fuseW float64 // below the fuse less its margin and the worst phase, read or heat, and what we sent; rises and starts + fuseOverW float64 // the fuse's measured overage that cuts now, 0 under it; see fuseHeat + fuseOverA float64 // the same on the worst phase, with perPhase + perPhase bool // the phase currents measure the fuse + overLimit bool // a phase, or the total without phases, reads over the fuse's working limit + limitW float64 // below the hour's import limit; negative is an overage + riseW float64 // below the level an increase may reach, less what we sent + startLevelW float64 // import an idle start may reach: the rise level, at most the ceiling + hourly bool // the hour's import is measured, so a short overage may wait + measured bool // false when the meter reports phase currents that cannot be read + worstA float64 + phaseRoomA float64 // below the working limit on the busiest phase, read or heat + phaseRoomKnown bool // the meter reports phase currents } -func (s *State) evMeterRoom(meter *telemetry.DerReading, gridW, phases, fuseMaxW float64) evMeterRoom { +func (s *State) evMeterRoom(meter *telemetry.DerReading, gridW, phases, fuseMaxW float64, sent pendingLoad) evMeterRoom { inf, totalW := math.Inf(1), math.Max(0, fuseMaxW-s.fuseSafetyMarginW())-gridW - r := evMeterRoom{fuseW: totalW, fuseCutW: totalW, limitW: inf, riseW: inf, startLevelW: inf} + r := evMeterRoom{fuseW: totalW - sent.w, fuseOverW: math.Max(0, -totalW), overLimit: totalW < 0, limitW: inf, riseW: inf, startLevelW: inf} if limitW, riseW, ok := s.importHourLimitsW(); ok { - r.limitW, r.riseW = limitW-gridW, riseW-gridW + r.limitW, r.riseW = limitW-gridW, riseW-gridW-sent.w // A car that could run only on one hour's slack would stop when // the next hour starts at the ceiling. r.startLevelW = math.Min(riseW, s.PeakImportCeilingW) @@ -279,6 +577,7 @@ func (s *State) evMeterRoom(meter *telemetry.DerReading, gridW, phases, fuseMaxW } amps, present, valid := phaseCurrentsA(meter, s) if present && !valid { + r.phaseRoomKnown = true // No room on phases it cannot read. return r } r.measured = true @@ -287,9 +586,11 @@ func (s *State) evMeterRoom(meter *telemetry.DerReading, gridW, phases, fuseMaxW for _, a := range amps { r.worstA = math.Max(r.worstA, a) } - r.fuseW = math.Min(r.fuseW, (limitA-s.fuseHeat.levelA(amps))*s.SiteFuseVoltage*phases) + r.fuseW = math.Min(r.fuseW, (limitA-s.fuseHeat.levelA(amps, sent.phaseA))*s.SiteFuseVoltage*phases) + r.phaseRoomA, r.phaseRoomKnown = limitA-s.fuseHeat.levelA(amps, [3]float64{}), true // The phases measure the fuse itself; total import stands in only without them. - r.fuseCutW = -s.fuseHeat.overA(amps, limitA, fuseInstantFactor*s.SiteFuseAmps) * s.SiteFuseVoltage * phases + r.fuseOverA = s.fuseHeat.overA(amps, limitA, fuseInstantFactor*s.SiteFuseAmps) + r.fuseOverW, r.perPhase, r.overLimit = r.fuseOverA*s.SiteFuseVoltage*phases, true, r.worstA > limitA } return r } @@ -304,6 +605,21 @@ func worstPhaseA(meter *telemetry.DerReading, state *State) (worstA float64, pre return worstA, present, valid } +// withBatteryOrders adds battery increase we ordered that the meter may not +// show yet, w, to the chargers' unshown offers: in total import, and as a +// three-phase battery puts it, the same current on each phase. A one-phase +// pool's room is one phase's, so the battery's whole power there would count +// it three times. +func (s *State) withBatteryOrders(p pendingLoad, w float64) pendingLoad { + p.w += w + if s.SiteFuseVoltage > 0 && s.SiteFusePhases > 0 { + for i := range p.phaseA { + p.phaseA[i] += w / (s.SiteFuseVoltage * float64(s.SiteFusePhases)) + } + } + return p +} + // pendingBatteryCommandsW is battery increase we ordered that the meter may // not show yet. Signed, so a stopped discharge counts too. func (s *State) pendingBatteryCommandsW(now time.Time) float64 { @@ -398,13 +714,15 @@ func qualifiedFuseObservation(r *telemetry.DerReading, now time.Time) (telemetry func publishBatterylessEVFuseBudget(store *telemetry.Store, state *State, fuseMaxW float64) { meter := store.Get(state.SiteMeterDriver, telemetry.DerMeter) - state.FuseEVMaxW, state.FuseSaturated = 0, false + state.FuseEVMaxW, state.FuseSaturated, state.FuseSaturatedByImport = 0, false, false evInMeterW := state.evInMeterW() demand := math.Max(evInMeterW, scheduledEVPowerW(state)) if meter == nil || demand <= 0 { return } - cap := evHeadroomW(state.phaseImportCeilingW(fuseMaxW, meter), math.Max(meter.RawW, meter.SmoothedW), evInMeterW) + ceilingW := state.phaseImportCeilingW(fuseMaxW, meter) + cap := evHeadroomW(ceilingW, math.Max(meter.RawW, meter.SmoothedW), evInMeterW) state.FuseEVMaxW = math.Min(demand, cap) state.FuseSaturated = demand > cap + state.FuseSaturatedByImport = state.FuseSaturated && state.importLimitSets(ceilingW) } diff --git a/go/internal/control/ev_fuse_budget_test.go b/go/internal/control/ev_fuse_budget_test.go index 0a0364619..add26c76a 100644 --- a/go/internal/control/ev_fuse_budget_test.go +++ b/go/internal/control/ev_fuse_budget_test.go @@ -3,7 +3,9 @@ package control import ( "context" "encoding/json" + "fmt" "math" + "slices" "testing" "time" @@ -39,6 +41,7 @@ type evMeterSite struct { mgr *loadpoint.Manager ctrl *loadpoint.Controller chargerDelay, telemetryDelay time.Duration + chargerMaxAgeS float64 // the driver's power_max_age_s, as Easee Cloud declares 180 readings evReadings houseW float64 housePhaseA [3]float64 @@ -123,6 +126,8 @@ func (s *evMeterSite) boot() { s.ctrl.SetFuseEVMax(func() (float64, bool) { return EVFuseBudget(s.store, s.state, s.mgr.States(), s.mgr.Configs(), 17250) }) + s.ctrl.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(s.state) }) + s.ctrl.SetFuseLimitReason(func() string { return EVLimitReason(s.state) }) s.ctrl.SetManualHold(cfg.ID, loadpoint.ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) if s.onBoot != nil { s.onBoot() @@ -157,7 +162,11 @@ func (s *evMeterSite) emit(driver string, kind telemetry.DerType, w float64, ext // report delivers one poll. The receipt time is all the driver knows. func (s *evMeterSite) report(w float64) { s.reportedW, s.reportedAt = w, s.now - s.emit("easee", telemetry.DerEV, w, nil) + var extra map[string]any + if s.chargerMaxAgeS > 0 { + extra = map[string]any{"power_max_age_s": s.chargerMaxAgeS} + } + s.emit("easee", telemetry.DerEV, w, extra) } func (s *evMeterSite) cloudW() float64 { @@ -393,48 +402,135 @@ func TestEVHardLimitCutsOnNextReadingWhenCarResumesIntoHouseLoad(t *testing.T) { } } -// After a cut the cap rises one step at a time. The next rise follows as -// soon as the meter shows import rising with the last one; a car that does -// not follow gets one step per settle time at most. -func TestEVHardLimitRiseWaitsForMeterOrDwell(t *testing.T) { - for _, paused := range []bool{false, true} { - s := newEVMeterSite(t, 7500, 900) - s.steady(5520) - s.setHouse(2900) // 920 W over: cut to 4,140 W - s.run(10 * time.Second) - if s.offer() != 4140 { - t.Fatalf("offer %.0f W after a 920 W overage", s.offer()) - } - s.setHouse(900) - s.carPaused = paused - recoveredAt := s.now - s.run(2 * time.Minute) - var rises []evTimed - got := s.offersSince(recoveredAt) - for i := 1; i < len(got); i++ { - if got[i].w-got[i-1].w > 690 { - t.Fatalf("paused=%v: rose more than one step at once: %v", paused, offerWatts(got)) - } - if got[i].w > got[i-1].w { - rises = append(rises, got[i]) - } - } - if len(rises) < 2 { - t.Fatalf("paused=%v: rises %v, want at least two", paused, offerWatts(got)) +// After a cut the cap rises once evSettle has passed, then one step per +// reading. It does not wait for the meter to show each rise: every rise +// counts as flowing until the charger can have applied it, so the car +// stops a step below the room. +func TestEVHardLimitRisesOnceCutApplied(t *testing.T) { + s := newEVMeterSite(t, 7500, 900) + s.steady(5520) + stepAt := s.now + s.setHouse(2900) // 920 W over: cut to 4,140 W + s.run(10 * time.Second) + cuts := s.offersSince(stepAt) + if len(cuts) == 0 || s.offer() != 4140 { + t.Fatalf("offer %.0f W after a 920 W overage", s.offer()) + } + var cutAt time.Time + for _, o := range cuts { + if o.w < 5520 { + cutAt = o.at + break } - gap := rises[1].at.Sub(rises[0].at) - if !paused && gap >= evSettle { - t.Fatalf("a rise the meter confirmed still waited %v", gap) + } + s.setHouse(900) + recoveredAt := s.now + s.run(time.Minute) + got := s.offersSince(recoveredAt) + var firstRise time.Time + for i := 1; i < len(got); i++ { + if got[i].w-got[i-1].w > 690 { + t.Fatalf("rose more than one step at once: %v", offerWatts(got)) } - if paused && gap < evSettle { - t.Fatalf("an unconfirmed rise followed after %v", gap) + if got[i].w > got[i-1].w && firstRise.IsZero() { + firstRise = got[i].at } - if !paused && (s.offer() != 5520 || s.houseW+s.evW > 7500-690) { - t.Fatalf("recovered to %.0f W at %.0f W grid, want 5,520 W with a step of margin", s.offer(), s.houseW+s.evW) + } + if firstRise.IsZero() || firstRise.Sub(cutAt) < evSettle { + t.Fatalf("first rise %v after the cut, want at least %v", firstRise.Sub(cutAt), evSettle) + } + if s.offer() != 5520 || s.houseW+s.evW > 7500-690 { + t.Fatalf("recovered to %.0f W at %.0f W grid, want 5,520 W with a step of margin", s.offer(), s.houseW+s.evW) + } +} + +// Easee Cloud may take one to three minutes to apply a new current and +// declares power_max_age_s 180. Beside 15 A of house load on the 25 A fuse, +// rises sent before the meter showed the last one stacked: a 60 s delay +// peaked at 25.0 A, a 120 s delay at 28.0 A. Each rise now counts as flowing +// until the charger can have applied it, and a charger that declares nothing +// counts as just as slow. The car settles in the measured room and climbs +// again when the house lets go. +func TestEVHardLimitSlowChargerRisesDoNotStack(t *testing.T) { + for _, declaredS := range []float64{180, 0} { + for _, delay := range []time.Duration{60 * time.Second, 120 * time.Second} { + t.Run(fmt.Sprintf("declared_%.0fs/delay_%v", declaredS, delay), func(t *testing.T) { + s := newEVMeterSite(t, 0, 15*690) + s.chargerDelay, s.chargerMaxAgeS = delay, declaredS + limitA := s.state.SiteFuseAmps - evPhaseMarginA + var maxA float64 + var overSince time.Time + watch := func(d time.Duration) { + t.Helper() + for end := s.now.Add(d); s.now.Before(end); { + s.step() + a := s.phaseA() + worst := math.Max(a[0], math.Max(a[1], a[2])) + maxA = math.Max(maxA, worst) + switch { + case worst <= limitA: + overSince = time.Time{} + case overSince.IsZero(): + overSince = s.now + case s.now.Sub(overSince) > delay+3*time.Second: + t.Fatalf("worst phase %.1f A above %.0f A for %v", worst, limitA, s.now.Sub(overSince)) + } + } + } + watch(10 * time.Minute) + // 24 - 15 A leaves 9 A; one step stays free. + if s.offer() != 5520 || s.evW != 5520 { + t.Fatalf("offer %.0f W draw %.0f W, want 5,520 W", s.offer(), s.evW) + } + s.setHouse(10 * 690) + watch(15 * time.Minute) + // 24 - 10 A leaves 14 A; one step stays free. + if s.offer() != 8970 || s.evW != 8970 { + t.Fatalf("after the house fell: offer %.0f W draw %.0f W, want 8,970 W", s.offer(), s.evW) + } + t.Logf("worst phase peaked at %.1f A", maxA) + }) } } } +// A fast local charger that declares no response time counts its offers +// as flowing for evUnknownSettle, beside what the meter already shows. It +// still takes the whole room in one move, and the pending count never cuts +// it: only a measured overage does. +func TestEVHardLimitUndeclaredFastChargerTakesRoomAtOnce(t *testing.T) { + s := newEVMeterSite(t, 0, 15*690) + for s.offer() == 0 { + s.step() + } + startedAt := s.now + s.run(s.chargerDelay + 2*time.Second) + // 24 - 15 A leaves 9 A; one step stays free. + if s.offer() != 5520 || s.evW != 5520 { + t.Fatalf("started at %.0f W, drawing %.0f W; want the 5,520 W room in one move", s.offer(), s.evW) + } + // 23.5 A measured, under the 24 A working limit, while the start still + // counts as flowing. + s.setHouse(15.5 * 690) + s.run(time.Minute) + if got := offerWatts(s.offersSince(startedAt)); slices.Min(got) != 5520 { + t.Fatalf("the pending start cut the car: %v", got) + } + + // Under a 7.5 kW import ceiling the house falls by 1,380 W: one run of + // rises, a step per reading, takes the car to the room less a step. + s = newEVMeterSite(t, 7500, 2280) + s.run(5 * time.Minute) + if s.offer() != 4140 { + t.Fatalf("setup: offer %.0f W", s.offer()) + } + s.setHouse(900) + s.run(s.chargerDelay + 5*time.Second) + if s.offer() != 5520 || s.evW != 5520 { + t.Fatalf("offer %.0f W draw %.0f W after the house fell, want 5,520 W", s.offer(), s.evW) + } +} + // A charger that ignores a cut gets the next one once its old current // should have reached the meter. func TestEVHardLimitCutsAgainAfterSettleWhenChargerIgnores(t *testing.T) { @@ -655,6 +751,57 @@ func TestEVHardLimitIdleStartNeedsRoomOnDeclaredPhases(t *testing.T) { } } +// Our own battery order the meter may not show yet counts as a +// three-phase battery puts it: the same current on each phase. A one-phase +// car's room is one phase's, so the whole order there counted it three +// times: 4,703 W of battery order left 661 W below one phase's limit, under +// the 1,610 W a start needs, and the car stayed at 0 W (fuse_limit) until +// the order had shown for evSettle and the meter had shown room for +// evRestartDwell (the rig of 22 Oct 2026, 42 s). The cap counts 6.8 A on +// each phase instead; a three-phase car's cap does not change. +func TestEVHardLimitBatteryOrderCountsOnEachPhase(t *testing.T) { + for _, tc := range []struct { + phase string + want float64 + }{ + {"1p", (24 - 0.68 - 4703.0/690 - 1) * 230}, + {"auto", (24 - 0.68 - 4703.0/690 - 1) * 230}, + {"3p", (24 - 0.68 - 4703.0/690 - 1) * 690}, + } { + t.Run(tc.phase, func(t *testing.T) { + store, state, points, configs := evMeterBudgetFixture(t, -466, []float64{-0.68, -0.68, -0.68}, 0, tc.phase) + now := time.Now() + state.batteryCommands = map[string]commandLog{"ferroamp": commandLog{}.add(now, 4703)} + got, known := EVFuseBudget(store, state, points, configs, 17250) + if !known || math.Abs(got-tc.want) > .01 { + t.Fatalf("cap %.1f W, want %.1f W", got, tc.want) + } + }) + } +} + +// While the hour's import ceiling holds an idle car back, the label names +// the ceiling. The fuse's room counts our battery order on its way, so the +// ceiling's room in the comparison must count it too, or a 5,500 W order +// with 2,750 W of fuse room read fuse_limit from the second reading on. +func TestEVHardLimitIdleCeilingLabelCountsBatteryOrder(t *testing.T) { + store, state, points, configs := evMeterBudgetFixture(t, 9000, []float64{13, 13, 13}, 0, "3p") + now := time.Now() + state.clock = func() time.Time { return now } + state.PeakImportCeilingW = 12000 + state.importHour = hourImport{start: state.importHourStart(now), ceilingW: 12000, limitW: 12000, riseW: 12000, + changedAt: now, seen: true, seenCeiling: 12000, lastAt: now.Add(-time.Second)} + state.batteryCommands = map[string]commandLog{"battery": commandLog{}.add(now, 5500)} + for reading := 1; reading <= 2; reading++ { + store.Get("meter", telemetry.DerMeter).UpdatedAt = now + got, _ := EVFuseBudget(store, state, points, configs, 17250) + if reason := EVLimitReason(state); got != 0 || reason != "import_limit" { + t.Fatalf("reading %d: cap %.1f W %s, want 0 W import_limit", reading, got, reason) + } + now = now.Add(2 * time.Second) + } +} + // A lone 0 W or unavailable charger reading is not a reason to cut, and // before our first command a charging car's reading earns no credit: the // meter alone decides. diff --git a/go/internal/control/ev_fuse_command_phase_test.go b/go/internal/control/ev_fuse_command_phase_test.go new file mode 100644 index 000000000..9db9876ce --- /dev/null +++ b/go/internal/control/ev_fuse_command_phase_test.go @@ -0,0 +1,116 @@ +package control + +import ( + "context" + "encoding/json" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// phaseSite runs one loadpoint against EVFuseBudget on a 25 A, 3 × 230 V +// site, wired as main.go wires it. Each tick the meter shows the house plus +// the last offer, spread over the phases the offer used. +type phaseSite struct { + now time.Time + store *telemetry.Store + state *State + mgr *loadpoint.Manager + ctrl *loadpoint.Controller + houseA float64 + offerW float64 + offer1p bool +} + +func newPhaseSite(t *testing.T, cfg loadpoint.Config, houseA float64) *phaseSite { + t.Helper() + s := &phaseSite{now: time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC), store: telemetry.NewStore(), houseA: houseA} + s.state = NewState(0, 0, "meter") + s.state.SiteFuseAmps, s.state.SiteFuseVoltage, s.state.SiteFusePhases = 25, 230, 3 + s.state.clock = func() time.Time { return s.now } + s.mgr = loadpoint.NewManager() + s.mgr.Load([]loadpoint.Config{cfg}) + s.ctrl = loadpoint.NewController(s.mgr, + func(time.Time) (loadpoint.Directive, bool) { return loadpoint.Directive{}, false }, + func(string) (loadpoint.EVSample, bool) { + return loadpoint.EVSample{Connected: true, RequestActive: true, PowerW: s.offerW, PowerAt: s.now}, true + }, + func(_ context.Context, _ string, payload []byte) error { + var cmd struct { + Action string `json:"action"` + PowerW float64 `json:"power_w"` + PhaseMode string `json:"phase_mode"` + } + if err := json.Unmarshal(payload, &cmd); err != nil { + return err + } + if cmd.Action == "ev_set_current" { + s.offerW = cmd.PowerW + s.offer1p = loadpoint.PhaseFor(cmd.PhaseMode, cmd.PowerW, 3680) == 1 + } + return nil + }) + s.ctrl.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + s.ctrl.SetFuseEVMax(func() (float64, bool) { + return EVFuseBudget(s.store, s.state, s.mgr.States(), s.ctrl.FuseConfigs(), 25*230*3) + }) + s.ctrl.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(s.state) }) + return s +} + +// worstA is the busiest phase: a one-phase offer may land on any of them. +func (s *phaseSite) worstA() float64 { + if s.offer1p { + return s.houseA + s.offerW/230 + } + return s.houseA + s.offerW/690 +} + +func (s *phaseSite) tick() { + s.now = s.now.Add(5 * time.Second) + a := s.worstA() + data, _ := json.Marshal(map[string]any{"control_power_available": true, + "control_power_observed_at": s.now.Format(time.RFC3339Nano), "l1_a": a, "l2_a": a, "l3_a": a}) + s.store.Update("meter", telemetry.DerMeter, a*690, nil, data) + s.store.Get("meter", telemetry.DerMeter).UpdatedAt = s.now + s.store.RecordDriverSuccess("meter") + s.ctrl.Tick(context.Background(), s.now) +} + +// R2 against the real cap: a saved but unarmed battery-SoC unlock must not +// change the phases the cap is measured on. +func TestChargeNowWithUnarmedUnlockHoldsUnderEVFuseBudget(t *testing.T) { + cfg := loadpoint.Config{ID: "garage", DriverName: "easee", MinChargeW: 4140, MaxChargeW: 11040, + AllowedStepsW: []float64{0, 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040}} + s := newPhaseSite(t, cfg, 3) + s.mgr.SetSchedule(cfg.ID, loadpoint.Schedule{SurplusUnlockBatSoC: 0.9}) + s.ctrl.SetBatSoCProvider(func() (float64, bool) { return 0.5, true }) + s.ctrl.SetSiteSurplusForEV(func() (float64, bool) { return -2000, true }) + s.ctrl.SetManualHold(cfg.ID, loadpoint.ManualHold{PowerW: 11040, Persistent: true}) + for tick := 0; tick < 20; tick++ { + s.tick() + st, _ := s.mgr.State(cfg.ID) + if s.offerW != 11040 || st.CommandedReason == "fuse_limit" { + t.Fatalf("tick %d: offer %.0f W, reason %q; want 11040 W", tick, s.offerW, st.CommandedReason) + } + } +} + +// An auto offer below the phase split lands on one phase. At 20 A of house +// load the cap must be measured on that phase, not on the three the config +// leaves unnamed: 2,070 W on one phase would put 29 A on a 25 A fuse. +func TestAutoOfferGetsOnePhaseOfRoomFromEVFuseBudget(t *testing.T) { + cfg := loadpoint.Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 11040, + AllowedStepsW: []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680, + 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040}} + s := newPhaseSite(t, cfg, 20) + s.ctrl.SetManualHold(cfg.ID, loadpoint.ManualHold{PowerW: 2760, PhaseMode: "auto", Persistent: true}) + for tick := 0; tick < 12; tick++ { + s.tick() + if s.worstA() > 24 { + t.Fatalf("tick %d: offer %.0f W puts %.1f A on one phase", tick, s.offerW, s.worstA()) + } + } +} diff --git a/go/internal/control/ev_fuse_cut_netting_test.go b/go/internal/control/ev_fuse_cut_netting_test.go new file mode 100644 index 000000000..575c8ad5b --- /dev/null +++ b/go/internal/control/ev_fuse_cut_netting_test.go @@ -0,0 +1,317 @@ +package control + +import ( + "encoding/json" + "fmt" + "math" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// cutsSince returns the offers below the one before each, from `at` on. +func (s *evMeterSite) cutsSince(at time.Time) []evTimed { + var out []evTimed + prev := math.Inf(1) + for _, o := range s.offers { + if !o.at.Before(at) && o.w < prev { + out = append(out, o) + } + prev = o.w + } + return out +} + +// Field matrix, netting rows: Easee Cloud applies a cut after 30-120 s. A +// house step on L1 takes the car from 15 A to 13 A; while that cut travels, +// a kettle adds 3 A on L1. Core cuts once for the step and once for the +// kettle's part. On master the second cut came 20 s after the first, before +// either could show, and the kettle's took the car lower still. +func TestEVFuseSlowChargerCutCoversItsOverage(t *testing.T) { + for _, tc := range []struct { + delay time.Duration + declared float64 + }{ + {40 * time.Second, 180}, {2 * time.Minute, 180}, {time.Minute, 0}, + } { + t.Run(fmt.Sprintf("delay_%v/declared_%.0fs", tc.delay, tc.declared), func(t *testing.T) { + s := fuseHeatSite(t) + s.chargerDelay, s.chargerMaxAgeS = tc.delay, tc.declared + s.run(time.Minute) + stepAt := s.now + s.addHouseA(6, 0, 0) // L1 26 A + took, offer := s.firstCut(2 * time.Minute) + if took < 0 || offer != 8970 { + t.Fatalf("step cut after %v to %.0f W, want 8,970 W", took, offer) + } + s.run(15 * time.Second) + s.addHouseA(3, 0, 0) // L1 29 A while the cut travels + s.run(tc.delay + time.Minute) + cuts := offerWatts(s.cutsSince(stepAt)) + if len(cuts) != 2 || cuts[0] != 8970 || cuts[1] != 6900 { + t.Fatalf("cuts %v, want 8,970 W for the step and 6,900 W for the kettle", cuts) + } + if a := s.phaseA(); s.evW != 6900 || a[0] > 24 { + t.Fatalf("car %.0f W, L1 %.1f A after both cuts landed", s.evW, a[0]) + } + }) + } +} + +// cutRig feeds EVFuseBudget site readings one second apart and applies its +// cap to the charger's offer, in 1 A steps, as the controller does. +type cutRig struct { + t *testing.T + store *telemetry.Store + state *State + now time.Time + offerW float64 + phases int // the phases the offer lands on +} + +func newCutRig(t *testing.T, offerW, declaredS float64) *cutRig { + t.Helper() + store, state, _, _ := evMeterBudgetFixture(t, 0, nil, offerW, "3p") + r := &cutRig{t: t, store: store, state: state, now: time.Unix(1_792_246_834, 0), offerW: offerW, phases: 3} + state.clock = func() time.Time { return r.now } + data, _ := json.Marshal(map[string]any{"power_max_age_s": declaredS}) + store.Update("charger", telemetry.DerEV, 0, nil, data) + return r +} + +// read delivers one reading with phase currents and returns the cap. +func (r *cutRig) read(phaseA [3]float64) float64 { + r.t.Helper() + data, _ := json.Marshal(map[string]any{"l1_a": phaseA[0], "l2_a": phaseA[1], "l3_a": phaseA[2]}) + return r.deliver((phaseA[0]+phaseA[1]+phaseA[2])*230, data) +} + +// readGrid delivers one reading of total import only. +func (r *cutRig) readGrid(gridW float64) float64 { + r.t.Helper() + return r.deliver(gridW, []byte(`{}`)) +} + +func (r *cutRig) deliver(gridW float64, data []byte) float64 { + r.t.Helper() + r.now = r.now.Add(time.Second) + r.store.Update("meter", telemetry.DerMeter, gridW, nil, data) + r.store.Get("meter", telemetry.DerMeter).UpdatedAt = r.now + _, _, points, configs := evMeterBudgetFixture(r.t, 0, nil, r.offerW, "3p") + points[0].CommandedPhases = r.phases + capW, _ := EVFuseBudget(r.store, r.state, points, configs, 17250) + if capW < r.offerW { + r.offerW = math.Floor(capW/690) * 690 + } + return capW +} + +// A raise on its way adds to the cut. The car draws 10 A while its 14 A +// raise travels, and the house adds 8 A on L1: 26 A on the 24 A working +// limit. Once the heat budget is spent, the cut takes the 2 A the meter +// shows and the 4 A it does not show yet, so L1 ends at 24 A when the raise +// lands. Taking the overage from the raise alone left L1 at 28 A. +func TestEVFuseRaiseOnItsWayAddsToCut(t *testing.T) { + r := newCutRig(t, 6900, 180) + for range 300 { + r.read([3]float64{18, 15, 15}) // 8 A of house on L1, 5 A on the others + } + r.offerW = 9660 // the raise to 14 A, sent with room for it + r.read([3]float64{18, 15, 15}) + var cutAt int + for i := 1; i <= 150 && r.offerW == 9660; i++ { + r.read([3]float64{26, 15, 15}) + cutAt = i + } + if r.offerW != 5520 { + t.Fatalf("offer %.0f W after %d s over, want 9,660 W less 2 A measured and 4 A on its way = 5,520 W", r.offerW, cutAt) + } + for range 60 { + if r.read([3]float64{26, 15, 15}); r.offerW != 5520 { + t.Fatalf("cut again to %.0f W while the first cut travels", r.offerW) + } + } +} + +// A cut that has not shown once its charger can have applied it covers +// nothing more, and the next reading over the limit cuts again: after the +// declared response time, or after evUnknownSettle when nothing is declared. +func TestEVFuseCutRetriedAfterResponseTime(t *testing.T) { + for _, tc := range []struct { + declared float64 + want time.Duration + }{ + {30, 30 * time.Second}, {180, 3 * time.Minute}, {0, evUnknownSettle}, + } { + t.Run(fmt.Sprintf("declared_%.0fs", tc.declared), func(t *testing.T) { + s := fuseHeatSite(t) + s.chargerMaxAgeS, s.chargerIgnores = tc.declared, true + s.run(time.Minute) + stepAt := s.now + s.addHouseA(6, 0, 0) // L1 26 A, under 1.25 x 25 A + if took, offer := s.firstCut(2 * time.Minute); took < 0 || offer != 8970 { + t.Fatalf("step cut after %v to %.0f W", took, offer) + } + s.run(tc.want + 10*time.Second) + cuts := s.cutsSince(stepAt) + if len(cuts) != 2 { + t.Fatalf("cuts %v, want a second one after %v", offerWatts(cuts), tc.want) + } + if gap := cuts[1].at.Sub(cuts[0].at); gap < tc.want || gap > tc.want+5*time.Second { + t.Fatalf("second cut %v after the first, want %v", gap, tc.want) + } + }) + } +} + +// Severe overage cuts whatever is on its way. Above the instant factor the +// first cut goes at once. While that cut travels, L1 stays at 33 A; once its +// filtered current passes 1.25 x 25 A, the current a gG fuse carries for an +// hour, Core cuts again without waiting the charger's 180 s. +func TestEVFuseSevereOverageCutsWhateverIsOnItsWay(t *testing.T) { + s := fuseHeatSite(t) + s.chargerMaxAgeS, s.chargerIgnores = 180, true + s.run(time.Minute) + stepAt := s.now + s.addHouseA(13, 0, 0) // L1 33 A + if took, offer := s.firstCut(5 * time.Second); took != 0 || offer != 4140 { + t.Fatalf("instant cut after %v to %.0f W, want at once to 4,140 W", took, offer) + } + s.run(150 * time.Second) + cuts := s.cutsSince(stepAt) + if len(cuts) != 2 || cuts[1].w != 0 { + t.Fatalf("cuts %v, want a second to 0 W", offerWatts(cuts)) + } + // From 20 A the filtered current passes 31.25 A after 109 s at 33 A. + if gap := cuts[1].at.Sub(cuts[0].at); gap < 100*time.Second || gap > 115*time.Second { + t.Fatalf("second cut %v after the first, want once the heat passes 1.25 x In", gap) + } +} + +// A cut covers only within the run over the limit it was sent in. A fast +// charger that declares nothing applies a cut in 5 s, but counts as slow +// for 180 s. Once the meter has shown L1 under the limit for evSettle, a +// second step on L1 is a new run, and the landed cut covers none of it. +// Within evSettle of the landing it is the same run, and the landed cut +// covers the step until the run's first cut has had 180 s: that is the +// cost of trusting a cut on its way, at under 1.25 x In. +func TestEVFuseLandedCutCoversOnlyItsRun(t *testing.T) { + for _, tc := range []struct { + after time.Duration // the second step after the first cut + maxOver time.Duration + }{ + {30 * time.Second, 30 * time.Second}, {10 * time.Second, evUnknownSettle}, + } { + t.Run(fmt.Sprintf("step_after_%v", tc.after), func(t *testing.T) { + s := fuseHeatSite(t) + s.run(time.Minute) + s.addHouseA(11, 0, 0) // L1 31 A: instant cut of 7 A, lands in 5 s + if took, offer := s.firstCut(5 * time.Second); took != 0 || offer != 5520 { + t.Fatalf("instant cut after %v to %.0f W, want 5,520 W", took, offer) + } + s.run(tc.after) + s.addHouseA(5, 0, 0) // L1 29 A with the car's cut landed + var over time.Duration + var peak float64 + for range 4 * 60 { + s.step() + if a := s.phaseA()[0]; a > 24 { + over += time.Second + peak = math.Max(peak, a) + } + } + if over > tc.maxOver || peak > fuseHourFactor*s.state.SiteFuseAmps { + t.Fatalf("L1 over the working limit for %v, up to %.1f A", over, peak) + } + t.Logf("L1 at %.1f A over the working limit for %v", peak, over) + }) + } +} + +// A car that draws less than its offer hides a cut: the cut comes off the +// offer, not the draw, and changes nothing the meter can show. The first cut +// covers for the charger's 180 s; once it has outlasted that, cuts cover for +// evSettle only, so the next ones follow as on master. With the car at 10 A +// of 15 A and L1 2 A over, L1 stays over for the heat budget (61 s), 180 s +// and three cuts 21 s apart; master took 2 min 36 s. +func TestEVFuseCarUnderItsOfferCutsAgainAfterResponseTime(t *testing.T) { + s := fuseHeatSite(t) + s.carMaxW = 6900 // 10 A of its 15 A offer + s.run(2 * time.Minute) + stepAt := s.now + s.addHouseA(11, 0, 0) // L1 26 A + var over, overFor time.Duration + for range 10 * 60 { + s.step() + if s.phaseA()[0] > 24 { + over += time.Second + overFor = over + } else { + over = 0 + } + } + if s.phaseA()[0] > 24 || overFor > evUnknownSettle+150*time.Second { + t.Fatalf("L1 %.1f A, over the working limit for %v; cuts %v", s.phaseA()[0], overFor, offerWatts(s.cutsSince(stepAt))) + } + t.Logf("L1 over the working limit for %v", overFor) +} + +// Without phase currents Core cannot see the hottest phase, so a cut on its +// way covers for evSettle only, as before: the total import cuts again 20 s +// after the first cut while the charger has not applied it. +func TestEVFuseWithoutPhaseCurrentsCoversForSettle(t *testing.T) { + r := newCutRig(t, 10350, 180) + for range 30 { + r.readGrid(15000) + } + r.readGrid(20000) // 2,750 W over 17,250 W + if r.offerW != 7590 { + t.Fatalf("first cut to %.0f W, want 7,590 W", r.offerW) + } + var again int + for i := 1; i <= 60 && r.offerW == 7590; i++ { + r.readGrid(20000) + again = i + } + if r.offerW >= 7590 || again < int(evSettle/time.Second) { + t.Fatalf("second cut after %d s to %.0f W, want one after evSettle", again, r.offerW) + } +} + +// A change of phases is no cut. A 15 A single-phase order replaced by a 6 A +// three-phase one lowers the current on the order's phase but adds 6 A to +// the other two, so it covers nothing: with L2 at 32 A on the 25 A fuse the +// cut takes the whole offer. +func TestEVFusePhaseChangeCoversNothing(t *testing.T) { + r := newCutRig(t, 3450, 180) + r.phases = 1 + for range 30 { + r.read([3]float64{20, 5, 5}) + } + r.offerW, r.phases = 4140, 3 + r.read([3]float64{20, 5, 5}) + if r.read([3]float64{20, 32, 20}); r.offerW != 0 { + t.Fatalf("offer %.0f W with L2 at 32 A and a 6 A three-phase order on its way, want 0 W", r.offerW) + } +} + +// After a restart, the offer Core finds in force is one it did not send in +// this run. A cut takes it as shown: the first reading over the limit trims +// the overage from it, and while that cut travels to a slow charger no +// further reading counts the found offer as a raise on its way. +func TestEVFuseRestartTakesFoundOfferAsShown(t *testing.T) { + s := fuseHeatSite(t) + s.chargerDelay, s.chargerMaxAgeS = time.Minute, 180 + s.addHouseA(6, 0, 0) + s.run(30 * time.Second) + s.boot() + bootAt := s.now + if took, offer := s.firstCut(5 * time.Second); took != 0 || offer != 8970 { + t.Fatalf("restart at 26 A cut after %v to %.0f W, want the first reading to 8,970 W", took, offer) + } + s.run(50 * time.Second) + if cuts := offerWatts(s.cutsSince(bootAt)); len(cuts) != 1 { + t.Fatalf("cuts %v while the first travels", cuts) + } +} diff --git a/go/internal/control/ev_fuse_phase_room_test.go b/go/internal/control/ev_fuse_phase_room_test.go new file mode 100644 index 000000000..c2347ac39 --- /dev/null +++ b/go/internal/control/ev_fuse_phase_room_test.go @@ -0,0 +1,259 @@ +package control + +import ( + "context" + "encoding/json" + "math" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// roomCharger applies each command delay after it was sent, on the phases +// the command names: a single-phase command lands on L1. +type roomCharger struct { + cfg loadpoint.Config + delay time.Duration + connected bool + queue []roomCommand + drawW float64 + onePhase bool +} + +type roomCommand struct { + at time.Time + w float64 + onePhase bool +} + +// roomSite is a 25 A three-phase site with chargers that declare their +// delay as power_max_age_s. The meter reads every second; the loadpoint +// ticks every 5 s, wired as main.go wires it. +type roomSite struct { + t *testing.T + now time.Time + start time.Time + store *telemetry.Store + state *State + mgr *loadpoint.Manager + ctrl *loadpoint.Controller + houseA [3]float64 + chargers []*roomCharger + static bool // EVFuseBudget reads the configured phase modes + worstA float64 +} + +func newRoomSite(t *testing.T, houseA float64, chargers ...*roomCharger) *roomSite { + t.Helper() + s := &roomSite{t: t, now: time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC), store: telemetry.NewStore(), chargers: chargers} + s.start = s.now + s.houseA = [3]float64{houseA, houseA, houseA} + s.state = NewState(0, 0, "meter") + s.state.SiteFuseAmps, s.state.SiteFuseVoltage, s.state.SiteFusePhases = 25, 230, 3 + s.state.clock = func() time.Time { return s.now } + byDriver := map[string]*roomCharger{} + var configs []loadpoint.Config + for _, c := range chargers { + byDriver[c.cfg.DriverName] = c + configs = append(configs, c.cfg) + } + s.mgr = loadpoint.NewManager() + s.mgr.Load(configs) + s.ctrl = loadpoint.NewController(s.mgr, + func(time.Time) (loadpoint.Directive, bool) { return loadpoint.Directive{}, false }, + func(driver string) (loadpoint.EVSample, bool) { + c := byDriver[driver] + return loadpoint.EVSample{Connected: c.connected, RequestActive: c.connected, PowerW: c.drawW, PowerAt: s.now}, true + }, + func(_ context.Context, driver string, payload []byte) error { + var cmd struct { + Action string `json:"action"` + PowerW float64 `json:"power_w"` + PhaseMode string `json:"phase_mode"` + } + if err := json.Unmarshal(payload, &cmd); err != nil { + return err + } + if cmd.Action == "ev_set_current" { + c := byDriver[driver] + c.queue = append(c.queue, roomCommand{s.now, cmd.PowerW, loadpoint.PhaseFor(cmd.PhaseMode, cmd.PowerW, 3680) == 1}) + } + return nil + }) + s.ctrl.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + s.ctrl.SetFuseEVMax(func() (float64, bool) { + configs := s.ctrl.FuseConfigs() + if s.static { + configs = s.mgr.Configs() + } + return EVFuseBudget(s.store, s.state, s.mgr.States(), configs, 25*230*3) + }) + s.ctrl.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(s.state) }) + return s +} + +func (s *roomSite) phaseA() [3]float64 { + a := s.houseA + for _, c := range s.chargers { + switch { + case !c.connected: + case c.onePhase: + a[0] += c.drawW / 230 + default: + for i := range a { + a[i] += c.drawW / 690 + } + } + } + return a +} + +func (s *roomSite) run(d time.Duration) { + s.t.Helper() + for end := s.now.Add(d); s.now.Before(end); { + s.now = s.now.Add(time.Second) + for _, c := range s.chargers { + for len(c.queue) > 0 && !s.now.Before(c.queue[0].at.Add(c.delay)) { + c.drawW, c.onePhase, c.queue = c.queue[0].w, c.queue[0].onePhase, c.queue[1:] + } + if !c.connected { + c.drawW = 0 + } + data, _ := json.Marshal(map[string]any{"power_max_age_s": c.delay.Seconds()}) + s.store.Update(c.cfg.DriverName, telemetry.DerEV, c.drawW, nil, data) + } + a := s.phaseA() + data, _ := json.Marshal(map[string]any{"control_power_available": true, + "control_power_observed_at": s.now.Format(time.RFC3339Nano), "l1_a": a[0], "l2_a": a[1], "l3_a": a[2]}) + s.store.Update("meter", telemetry.DerMeter, (a[0]+a[1]+a[2])*230, nil, data) + s.store.Get("meter", telemetry.DerMeter).UpdatedAt = s.now + s.store.RecordDriverSuccess("meter") + s.worstA = math.Max(s.worstA, max(a[0], a[1], a[2])) + if s.now.Sub(s.start)%(5*time.Second) == 0 { + s.ctrl.Tick(context.Background(), s.now) + } + } +} + +// requireWithinLimit fails when a phase passed the 24 A working limit. +func (s *roomSite) requireWithinLimit(after string) { + s.t.Helper() + if limitA := s.state.SiteFuseAmps - evPhaseMarginA; s.worstA > limitA+1e-6 { + s.t.Fatalf("%s: a phase reached %.1f A, over the %.0f A working limit", after, s.worstA, limitA) + } +} + +func roomTestCharger(id string, delay time.Duration) *roomCharger { + steps := []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680, + 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040} + return &roomCharger{cfg: loadpoint.Config{ID: id, DriverName: id, MinChargeW: 1380, MaxChargeW: 11040, AllowedStepsW: steps}, + delay: delay, connected: true} +} + +// A 3p offer moved to 1p at nearly the same watts triples the current on +// one phase. The budget counts current per phase, so the switch needs room +// for the rise on that phase. Before, 7,360 W went out on L1 beside 12 A: +// 36 A for 50 s with a 5 s charger and for 90 s with a 90 s charger. +func TestEVFusePhaseSwitchNeedsRoomOnThePhase(t *testing.T) { + for _, delay := range []time.Duration{5 * time.Second, 90 * time.Second} { + t.Run(delay.String(), func(t *testing.T) { + c := roomTestCharger("a", delay) + s := newRoomSite(t, 5, c) + s.houseA[0] = 12 + s.ctrl.SetManualHold("a", loadpoint.ManualHold{PowerW: 8280, PhaseMode: "3p", Persistent: true}) + s.run(5 * time.Minute) + s.ctrl.SetManualHold("a", loadpoint.ManualHold{PowerW: 7360, PhaseMode: "1p", Persistent: true}) + s.run(6 * time.Minute) + s.requireWithinLimit("after 3p to 1p") + if !c.onePhase || c.drawW < 1380 { + t.Fatalf("the car draws %.0f W (1p %v), want a 1p charge", c.drawW, c.onePhase) + } + }) + } +} + +// Two cars at 8 A; the first is raised. Room a slow charger's cut frees is +// not there until that charger can have applied it, and a rise takes only +// the room net of every charger's reservation. Before, the raised car took +// the other's share in one tick: 27 A for 60 s. +func TestEVFuseRaisedCarTakesOnlyFreeRoom(t *testing.T) { + for _, tc := range []struct { + name string + aDelay, bDelay time.Duration + }{ + {"a_fast_b_slow", 5 * time.Second, 90 * time.Second}, + {"both_slow", 60 * time.Second, 120 * time.Second}, + } { + t.Run(tc.name, func(t *testing.T) { + a, b := roomTestCharger("a", tc.aDelay), roomTestCharger("b", tc.bDelay) + s := newRoomSite(t, 3, a, b) + s.ctrl.SetManualHold("a", loadpoint.ManualHold{PowerW: 5520, PhaseMode: "3p", Persistent: true}) + s.ctrl.SetManualHold("b", loadpoint.ManualHold{PowerW: 5520, PhaseMode: "3p", Persistent: true}) + s.run(10 * time.Minute) + s.ctrl.SetManualHold("a", loadpoint.ManualHold{PowerW: 11040, PhaseMode: "3p", Persistent: true}) + s.run(8 * time.Minute) + s.requireWithinLimit("after car a was raised") + if a.drawW <= 5520 { + t.Fatalf("car a stayed at %.0f W beside 3 A of house and car b at %.0f W", a.drawW, b.drawW) + } + }) + } +} + +// A 1p hold over a 3p loadpoint: each offer counts on the phases its own +// command names, never the configured ones, even when the budget is read +// with the configured phase mode. Before, 1p offers counted as 3p and L1 +// reached 39 A. The hold asks 24 A; the charger is 16 A per phase, so the +// car climbs from 9 A beside 15 A to the charger's 16 A once L1 falls. +func TestEVFuseOnePhaseOffersCountOnTheirPhase(t *testing.T) { + for _, delay := range []time.Duration{5 * time.Second, 90 * time.Second} { + t.Run(delay.String(), func(t *testing.T) { + c := roomTestCharger("a", delay) + c.cfg.PhaseMode = "3p" + s := newRoomSite(t, 5, c) + s.static = true + s.houseA[0] = 15 + s.ctrl.SetManualHold("a", loadpoint.ManualHold{PowerW: 5520, PhaseMode: "1p", Persistent: true}) + s.run(5 * time.Minute) + s.houseA[0] = 5 + s.run(6 * time.Minute) + s.requireWithinLimit("after L1 fell") + if !c.onePhase || c.drawW < 3680 { + t.Fatalf("the car draws %.0f W (1p %v), want it to climb on L1", c.drawW, c.onePhase) + } + }) + } +} + +// A silent meter never frees room. An offer sent after the last reading +// counts against it until a newer reading could show the offer. +func TestEVFusePhaseRoomHoldsOnSilentMeter(t *testing.T) { + store, state, points, configs := evMeterBudgetFixture(t, 15*690, []float64{15, 15, 15}, 0, "3p") + now := time.Now().Truncate(time.Second) + state.clock = func() time.Time { return now } + store.Get("meter", telemetry.DerMeter).UpdatedAt = now + points[0].CommandedPhases = 3 + room := func() float64 { + t.Helper() + EVFuseBudget(store, state, points, configs, 17250) + roomA, ok := EVFusePhaseRoomA(state) + if !ok { + t.Fatal("no phase room from a meter with phase currents") + } + return roomA + } + if got := room(); math.Abs(got-9) > 1e-9 { + t.Fatalf("room %.2f A beside 15 A, want 9 A", got) + } + points[0].CommandedW = 5520 + now = now.Add(time.Second) + if got := room(); math.Abs(got-1) > 1e-9 { + t.Fatalf("room %.2f A after an 8 A offer, want 1 A", got) + } + now = now.Add(10 * time.Minute) + if got := room(); math.Abs(got-1) > 1e-9 { + t.Fatalf("room %.2f A after ten silent minutes, want 1 A", got) + } +} diff --git a/go/internal/control/ev_fuse_restart_after_pause_test.go b/go/internal/control/ev_fuse_restart_after_pause_test.go new file mode 100644 index 000000000..881a084b9 --- /dev/null +++ b/go/internal/control/ev_fuse_restart_after_pause_test.go @@ -0,0 +1,48 @@ +package control + +import ( + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" +) + +// A field case on v0.155.0-beta.1 (Easee Cloud, 25 A fuse): a charge at +// 10,350 W, a 25 s pause, then a 4,140 W request. Easee declares 180 s, so +// the rise sent before the pause stayed reserved, and the restart waited +// close to three minutes with fuse_limit while the phases read about 1.3 A. +// A cut ends the reservation of the higher orders before it: the charger +// moves to its latest order, and the meter shows its draw until the cut +// lands. +func TestEVHardLimitRestartAfterPauseIgnoresReplacedRise(t *testing.T) { + s := newEVMeterSite(t, 0, 900) + s.chargerDelay, s.telemetryDelay, s.chargerMaxAgeS = 5*time.Second, 10*time.Second, 180 + s.run(25 * time.Second) + if s.offer() <= 0 { + t.Fatalf("the car did not start before the pause: offers %v", offerWatts(s.offers)) + } + s.ctrl.SetManualHold("garage", loadpoint.ManualHold{PowerW: 0, PhaseMode: "3p", Persistent: true}) + s.run(25 * time.Second) + if s.offer() != 0 { + t.Fatalf("the pause did not stop the offer: %v", offerWatts(s.offers)) + } + askedAt := s.now + s.ctrl.SetManualHold("garage", loadpoint.ManualHold{PowerW: 4140, PhaseMode: "3p", Persistent: true}) + s.run(3 * time.Minute) + var startedAt time.Time + for _, o := range s.offersSince(askedAt) { + if o.w > 0 { + startedAt = o.at + break + } + } + if startedAt.IsZero() { + t.Fatalf("no offer within 3 min after the restart: %v", offerWatts(s.offersSince(askedAt))) + } + // The idle-start dwell (35 s) may still apply; the replaced rise must not. + if wait := startedAt.Sub(askedAt); wait > 60*time.Second { + t.Fatalf("restart waited %v for a 4,140 W offer with the phases near 1.3 A", wait) + } else { + t.Logf("restart offered after %v", wait) + } +} diff --git a/go/internal/control/fuse_heat.go b/go/internal/control/fuse_heat.go index f6fcffbd9..e80c5e70e 100644 --- a/go/internal/control/fuse_heat.go +++ b/go/internal/control/fuse_heat.go @@ -5,6 +5,7 @@ import ( "math" "time" + "github.com/srcfl/ftw/go/internal/loadpoint" "github.com/srcfl/ftw/go/internal/telemetry" ) @@ -71,12 +72,13 @@ func (h *fuseHeat) overA(amps []float64, limitA, instantA float64) float64 { } // levelA is the highest phase current, taking each phase at the higher of -// the reading and its heat. Rises and starts read it, so a rise never spends -// heat the last minute has already used. -func (h *fuseHeat) levelA(amps []float64) float64 { +// the reading and its heat, plus what we sent that it may not show yet. +// Rises and starts read it, so a rise never spends heat the last minute has +// already used, nor room an earlier rise will take. +func (h *fuseHeat) levelA(amps []float64, pendingA [3]float64) float64 { level := 0.0 for i, a := range amps { - level = math.Max(level, math.Max(a, math.Sqrt(h.sq[i]))) + level = math.Max(level, math.Max(a, math.Sqrt(h.sq[i]))+pendingA[i]) } return level } @@ -121,3 +123,21 @@ func phaseCurrentsA(meter *telemetry.DerReading, state *State) (amps []float64, } return amps, true, true } + +// SitePhases is the site meter's last reading of its phase currents, for +// the loadpoints' one-phase car check. false when the meter reports no +// usable phase currents or the site has fewer than three phases. +func SitePhases(store *telemetry.Store, state *State) (loadpoint.SitePhases, bool) { + if store == nil || state == nil || state.SiteFusePhases != 3 { + return loadpoint.SitePhases{}, false + } + meter := store.Get(state.SiteMeterDriver, telemetry.DerMeter) + if meter == nil { + return loadpoint.SitePhases{}, false + } + amps, _, valid := phaseCurrentsA(meter, state) + if !valid || len(amps) != 3 { + return loadpoint.SitePhases{}, false + } + return loadpoint.SitePhases{A: [3]float64(amps), At: meter.UpdatedAt}, true +} diff --git a/go/internal/control/import_hour_test.go b/go/internal/control/import_hour_test.go index b91ff11bf..06741868c 100644 --- a/go/internal/control/import_hour_test.go +++ b/go/internal/control/import_hour_test.go @@ -152,6 +152,29 @@ func TestImportCeilingHouseStepLateInHourStepsCarDown(t *testing.T) { } } +// The car held below its want by the hour's import ceiling says +// import_limit, not fuse_limit: at 23:00 in #1674 the phases sit near 12 A +// on a 25 A fuse. Held by the fuse instead, it says fuse_limit. +func TestImportCeilingSaysImportLimitNotFuse(t *testing.T) { + for _, tc := range []struct { + name string + ceilingW, houseW float64 + want string + }{ + {"hour's ceiling", 7500, 900, "import_limit"}, + {"fuse", 0, 9000, "fuse_limit"}, + } { + t.Run(tc.name, func(t *testing.T) { + h := newHourSite(t, tc.ceilingW, tc.houseW, clock(16, 22, 0)) + h.run(70 * time.Minute) + st, _ := h.mgr.State("garage") + if h.evW <= 0 || h.evW >= 10350 || st.CommandedReason != tc.want { + t.Fatalf("car draws %.0f W of 10,350 W, reason %q, want %q", h.evW, st.CommandedReason, tc.want) + } + }) + } +} + // A Core restart mid-hour keeps the hour's import from state.db. A sauna // spent most of the budget; without the record the box would count the // first half-hour at the ceiling and let the car overrun the hour. @@ -405,6 +428,48 @@ func TestImportCeilingNoStartOnSlackAlone(t *testing.T) { } } +// The field case from the review of #1783: a 7.5 kW hourly ceiling, a 25 A +// fuse, a light first half hour, and a car plugged in at 22:40 beside 6 kW +// (and 8 kW) of house load. The ceiling keeps it idle to the end of the +// hour, and not one tick of that says fuse_limit. +func TestImportCeilingNightNeverSaysFuse(t *testing.T) { + for _, houseW := range []float64{6000, 8000} { + h := newHourSite(t, 7500, 500, clock(16, 22, 0)) + h.unplugged = true + h.run(40 * time.Minute) + h.setHouse(houseW) + h.plugIn() + for h.now.Before(clock(16, 23, 0)) { + h.step() + if st, _ := h.mgr.State("garage"); st.CommandedReason == "fuse_limit" { + t.Fatalf("house %.0f W: %s says fuse_limit with the phases low under a 7.5 kW hourly ceiling", + houseW, h.now.In(hourTestZone).Format("15:04:05")) + } + } + if h.offer() != 0 { + t.Fatalf("house %.0f W: the car started at %.0f W under the ceiling", houseW, h.offer()) + } + } +} + +// A quiet first 40 minutes leave the hour so much slack that its limits sit +// above what the 25 A fuse allows. An 8 kW house still keeps an idle car from +// starting, by the ceiling's own start test, and the car says import_limit. +func TestImportCeilingStartTestSaysImportLimit(t *testing.T) { + h := newHourSite(t, 7500, 0, clock(16, 21, 0)) + h.unplugged = true + h.run(40 * time.Minute) + h.setHouse(8000) + h.plugIn() + h.run(5 * time.Minute) + if h.offer() != 0 { + t.Fatalf("the car started at %.0f W beside 8 kW under a 7.5 kW ceiling", h.offer()) + } + if st, _ := h.mgr.State("garage"); st.CommandedReason != "import_limit" { + t.Fatalf("idle under the ceiling says %q, want import_limit", st.CommandedReason) + } +} + // A reading after a clock jump of years counts only the hour it falls in. func TestImportCeilingClockJumpCountsOnlyItsHour(t *testing.T) { var now time.Time diff --git a/go/internal/control/sitesim_field_api_test.go b/go/internal/control/sitesim_field_api_test.go new file mode 100644 index 000000000..83dd886c1 --- /dev/null +++ b/go/internal/control/sitesim_field_api_test.go @@ -0,0 +1,36 @@ +//go:build !race + +package control_test + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + + "github.com/srcfl/ftw/go/internal/api" + "github.com/srcfl/ftw/go/internal/control" + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// The field matrix reads the EV verdict as the app does, from GET +// /api/loadpoints. The api package imports control, so only this external +// test package can build the server. +func init() { + control.FieldVerdict = func(tel *telemetry.Store, lps *loadpoint.Manager, ctrl *loadpoint.Controller) func() (string, string) { + srv := api.New(&api.Deps{Tel: tel, Loadpoints: lps, LoadpointCtrl: ctrl}) + return func() (string, string) { + rr := httptest.NewRecorder() + srv.Handler().ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/api/loadpoints", nil)) + var body struct { + Loadpoints []struct { + Feedback *struct{ Status, Reason string } `json:"control_feedback"` + } `json:"loadpoints"` + } + if json.Unmarshal(rr.Body.Bytes(), &body) != nil || len(body.Loadpoints) == 0 || body.Loadpoints[0].Feedback == nil { + return "none", "no control_feedback" + } + return body.Loadpoints[0].Feedback.Status, body.Loadpoints[0].Feedback.Reason + } + } +} diff --git a/go/internal/control/sitesim_field_matrix_test.go b/go/internal/control/sitesim_field_matrix_test.go new file mode 100644 index 000000000..3f1b8bd00 --- /dev/null +++ b/go/internal/control/sitesim_field_matrix_test.go @@ -0,0 +1,145 @@ +//go:build !race + +// The race detector makes the matrix 15 times slower (826 s of CPU) and finds +// nothing in it: each case runs on one goroutine. TestSiteSimScenarios still +// runs under -race. + +package control + +import ( + "fmt" + "maps" + "os" + "slices" + "sort" + "strings" + "sync" + "testing" + "testing/synctest" +) + +// fieldKnownFile lists the rows that fail on master: case, kind and the fix +// that turns it green. A listed kind that passes fails the test until its +// line goes, and so does a failure the file does not list. +// FTW_FIELD_KNOWN=write rewrites it from a run and keeps each line's label. +const fieldKnownFile = "testdata/sitesim_field_known.txt" + +type fieldKnown struct { + header []string + fixes map[string]map[string]string // case, kind: fix +} + +func readFieldKnown(t *testing.T) fieldKnown { + t.Helper() + data, err := os.ReadFile(fieldKnownFile) + if err != nil { + t.Fatal(err) + } + k := fieldKnown{fixes: map[string]map[string]string{}} + for i, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if line == "" || strings.HasPrefix(line, "#") { + k.header = append(k.header, line) + continue + } + f := strings.Fields(line) + if len(f) != 3 { + t.Fatalf("%s:%d: want case, kind and fix: %q", fieldKnownFile, i+1, line) + } + if k.fixes[f[0]] == nil { + k.fixes[f[0]] = map[string]string{} + } + k.fixes[f[0]][f[1]] = f[2] + } + return k +} + +func (k fieldKnown) write(t *testing.T, results []fieldResult) { + t.Helper() + lines := slices.Clone(k.header) + for _, r := range results { + for _, kind := range r.failed { + fix := k.fixes[r.name][kind] + if fix == "" { + fix = "new" + } + lines = append(lines, r.name+" "+kind+" "+fix) + } + } + if err := os.WriteFile(fieldKnownFile, []byte(strings.Join(lines, "\n")+"\n"), 0o644); err != nil { + t.Fatal(err) + } +} + +// fieldResult is one case's broken invariants. +type fieldResult struct { + name string + failed []string +} + +func TestSiteSimFieldMatrix(t *testing.T) { + cases, known := fieldCases(), readFieldKnown(t) + writeKnown := os.Getenv("FTW_FIELD_KNOWN") == "write" + names := map[string]bool{} + for _, c := range cases { + names[c.name()] = true + } + for name := range known.fixes { + if !names[name] { + t.Errorf("%s names %s, which the matrix no longer has", fieldKnownFile, name) + } + } + var mu sync.Mutex + var results []fieldResult + t.Cleanup(func() { + sort.Slice(results, func(i, j int) bool { return results[i].name < results[j].name }) + byKind, failing := map[string]int{}, 0 + for _, r := range results { + if len(r.failed) > 0 { + failing++ + } + for _, k := range r.failed { + byKind[k]++ + } + } + t.Logf("field matrix: %d cases, %d fail an invariant; by kind %v", len(results), failing, byKind) + if writeKnown && len(results) == len(cases) { + known.write(t, results) + } + }) + for _, c := range cases { + t.Run(c.name(), func(t *testing.T) { + t.Parallel() + synctest.Test(t, func(t *testing.T) { + var trace func(string, ...any) + if os.Getenv("FTW_FIELD_TRACE") == c.name() { + trace = t.Logf + } + s, f := runFieldCase(c, trace) + failed, fixes := slices.Sorted(maps.Keys(f.failures)), known.fixes[c.name()] + var problems []string + for _, k := range failed { + if fix, ok := fixes[k]; ok { + t.Logf("fails as known (%s): %s", fix, f.failures[k]) + } else { + problems = append(problems, f.failures[k]) + } + } + for _, k := range slices.Sorted(maps.Keys(fixes)) { + if _, ok := f.failures[k]; !ok { + problems = append(problems, fmt.Sprintf("%s passes now: delete its known line (%s)", k, fixes[k])) + } + } + mu.Lock() + results = append(results, fieldResult{name: c.name(), failed: failed}) + mu.Unlock() + t.Logf("peak phase %.1f A, delivered %.1f kWh, car idle with room %v", s.peakA, f.deliveredWh/1000, s.idleWithRoom) + switch { + case len(problems) > 0 && writeKnown: + t.Log(strings.Join(problems, "\n")) + case len(problems) > 0: + t.Fatal(strings.Join(problems, "\n")) + } + }) + }) + } +} diff --git a/go/internal/control/sitesim_field_test.go b/go/internal/control/sitesim_field_test.go new file mode 100644 index 000000000..e77f5a90b --- /dev/null +++ b/go/internal/control/sitesim_field_test.go @@ -0,0 +1,870 @@ +package control + +import ( + "cmp" + "context" + "encoding/json" + "fmt" + "maps" + "math" + "slices" + "strings" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/sitesim" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// The field matrix runs Core's EV charging and fuse protection in the closed +// loop against the settings sites run: each field report's fuse, phase mode, +// step table, charger, car, house and battery, and the axes around them. A +// beta must pass it, not only our defaults (fixed 3p, no step table, a +// charger that answers at once or exactly when it says). +// +// Each case runs in a synctest bubble, so the telemetry store's wall clock is +// the plant's clock: command evidence, the API's verdict and live Home use +// are timed as on a box. + +// fieldTiming is a charger's response: how long an order takes to reach the +// car and the power_max_age_s its driver declares, zero for none. Easee Cloud +// applies an order in 30-60 s and declares 180 s. +type fieldTiming struct { + name string + delay time.Duration + declared time.Duration + cloud bool // easee_cloud.lua: splits auto at the fuse, holds a phase count 90 s, reports 10 s late +} + +var ( + timingFast = fieldTiming{"fast2s", 2 * time.Second, 0, false} + timingEasee = fieldTiming{"easee40s", 40 * time.Second, 3 * time.Minute, true} + timingSlowCloud = fieldTiming{"easee120s", 2 * time.Minute, 3 * time.Minute, true} + timingUndeclared = fieldTiming{"cloud60s", time.Minute, 0, true} + fieldTimings = []fieldTiming{timingFast, timingEasee, timingSlowCloud, timingUndeclared} +) + +// fieldUse is what the household asked for. +type fieldUse string + +const ( + usePlanned fieldUse = "planned" // a ready-by goal; the planner's directive per slot + useNow fieldUse = "chargenow" // Charge now from the app: 3p, or 1p on a 1p charger + useSurplus fieldUse = "surplus" // surplus_only while export steps up and down + usePause fieldUse = "pause" // Charge now, Pause for 25 s, then Start + useRestart fieldUse = "restart" // a planned charge across a 20 s Core restart + usePhaseUp fieldUse = "1p-to-3p" // a held 1p order changed to 3p after 5 min +) + +// fieldCase is one site and one request. The name is stable: the known rows +// refer to it. +type fieldCase struct { + report string // the configuration it comes from + fuseA float64 // per phase, three phases at 230 V + mode string // loadpoint phase_mode + steps string // step table: none, mixed (1p and 3p levels), 3p, 1p, or a report's own + timing fieldTiming + carPhases int + use fieldUse + house [3]float64 // A per phase + houseName string + battery bool // the plan charges the home battery meanwhile + every time.Duration // control_interval_s; zero: 2 s + lp func(*loadpoint.Config) + car func(*sitesim.Car) + + charger string // the charger itself when not what the phase mode says: "1p32A" + planW float64 // > 0: the plan asks this many W each slot, as Wh only + needW float64 // > 0: the goal asks this many W on average + goal bool // surplus-only with a ready-by goal + exportW []float64 // surplus: export each quarter hour + houseStep time.Duration // > 0: L1 +8 A for 10 min from this time into the run + warm bool // the fuse starts warm: the house has carried its load all evening +} + +func (c fieldCase) key() string { + mode := c.mode + if mode == "" { + mode = "empty" + } + name := fmt.Sprintf("%gA/%s/%s/%s/car%dp/%s/%s", c.fuseA, mode, c.steps, c.timing.name, c.carPhases, c.houseName, c.use) + export := "" + if len(c.exportW) > 0 { + export = fmt.Sprintf("export%gW", c.exportW[0]) + } + for _, v := range []struct { + on bool + text string + }{ + {c.battery, "battery"}, {c.charger != "", "charger" + c.charger}, {c.planW > 0, fmt.Sprintf("plan%gW", c.planW)}, {c.needW > 0, fmt.Sprintf("need%gW", c.needW)}, + {c.goal, "goal"}, {export != "", export}, {c.houseStep > 0, "L1step"}, + } { + if v.on { + name += "+" + v.text + } + } + return name +} + +func (c fieldCase) name() string { return c.report + "/" + c.key() } + +// Step tables as sites have them. mixed is Easee's whole amps on one phase, +// then on three (srcfl/ftw#1774's tests, the rig's s5s). +func stepTable(name string) (steps []float64, minW, maxW float64) { + amps := func(phases float64, from, to float64) []float64 { + var out []float64 + for a := from; a <= to; a++ { + out = append(out, a*230*phases) + } + return out + } + switch name { + case "mixed": + return slices.Concat([]float64{0}, amps(1, 6, 16), amps(3, 6, 16)), 1380, 11040 + case "3p": + return slices.Concat([]float64{0}, amps(3, 6, 16)), 4140, 11040 + case "1p": + return slices.Concat([]float64{0}, amps(1, 6, 16)), 1380, 3680 + case "1p-3": // three one-phase levels and a maximum above them + return []float64{0, 1380, 2300, 3680}, 1380, 5520 + } + return nil, 4140, 11000 // What the settings page writes for a new charger. +} + +func fieldLoadpoint(c fieldCase) loadpoint.Config { + steps, minW, maxW := stepTable(c.steps) + if c.mode == "1p" && c.steps == "none" { + minW, maxW = 1380, 3680 + } + cfg := loadpoint.Config{ID: "garage", DriverName: "evse", PhaseMode: c.mode, AllowedStepsW: steps, + MinChargeW: minW, MaxChargeW: maxW, VehicleCapacityWh: 60000, SurplusOnly: c.use == useSurplus} + if c.lp != nil { + c.lp(&cfg) + } + return cfg +} + +var ( + houseBalanced = [3]float64{3, 3, 3} + houseL1 = [3]float64{7, 1, 1} // most of the house on one phase, the phase a 1p car uses +) + +// Field reports. Settings only: no names, hosts or credentials. + +// helpReport is the help report of 22 Oct 2026, v0.155.0-beta.1: 3 × 20 A, +// Easee Cloud, phase_mode auto, a step table with one-phase levels +// 1,600-3,800 W and three-phase 4,800-11,100 W, a deadline goal, +// control_interval_s 5. Pixii is the site meter; its battery sat at its floor +// all night, so it is left out. The house drew about 2 A a phase; the car +// took 3.1-3.4 kW on one phase and was cut to 0 W for the fuse every few +// minutes. +func helpReport(use fieldUse, tm fieldTiming) fieldCase { + return fieldCase{report: "help1022", fuseA: 20, mode: "auto", steps: "own", timing: tm, carPhases: 3, use: use, + house: [3]float64{2, 2, 2}, houseName: "2A", every: 5 * time.Second, + lp: func(c *loadpoint.Config) { + c.AllowedStepsW = []float64{0, 1600, 1900, 2100, 2400, 2600, 2800, 3100, 3300, 3500, 3800, + 4800, 5500, 6200, 6900, 7600, 8300, 9000, 9700, 10400, 11100} + c.MinChargeW, c.MaxChargeW, c.VehicleCapacityWh = 1600, 11100, 84000 + }} +} + +// pr1763Site is the site in srcfl/ftw#1763's comments: 3 × 25 A, Easee Cloud +// and Pixii, the battery charging from the plan while the car charges, +// offers to 15 A of which the car takes about 12 A, pause and restart. +func pr1763Site(use fieldUse, tm fieldTiming) fieldCase { + return fieldCase{report: "pr1763", fuseA: 25, mode: "3p", steps: "3p-15A", timing: tm, carPhases: 3, use: use, + house: [3]float64{6, 2, 2}, houseName: "6-2-2", battery: true, + lp: func(c *loadpoint.Config) { + steps, minW, _ := stepTable("3p") + c.AllowedStepsW, c.MinChargeW, c.MaxChargeW = steps[:len(steps)-1], minW, 10350 + }, + car: func(car *sitesim.Car) { car.MaxW = 8300 }} +} + +// homeBox is the home box: 3 × 16 A, Easee Cloud, phase_mode unset, no step +// table, 4,140-11,000 W. Its own goal is 80 % at 07:00; like every planned +// case here it plans the matrix's need instead. +func homeBox(use fieldUse, tm fieldTiming) fieldCase { + return fieldCase{report: "homebox", fuseA: 16, mode: "", steps: "none", timing: tm, carPhases: 3, use: use, + house: houseBalanced, houseName: "balanced", + lp: func(c *loadpoint.Config) { c.VehicleCapacityWh = 76000 }} +} + +// fieldCases crosses the axes. The field reports run with every charger +// timing. The sweep crosses fuse, phase mode, step table, car and house for +// planned charging and Charge now with an Easee Cloud charger; the timing +// sweep runs every request with every timing on fewer sites. Pruned: a 1p +// phase mode is a one-phase charger, so the car's phases and a 3p table do +// not apply; 20 A runs as the help report's site, between 16 and 25 A. +func fieldCases() []fieldCase { + var out []fieldCase + seen := map[string]bool{} + add := func(c fieldCase) { + if !seen[c.key()] { + seen[c.key()] = true + out = append(out, c) + } + } + for _, use := range []fieldUse{usePlanned, useNow, usePause, useRestart} { + for _, tm := range fieldTimings { + add(helpReport(use, tm)) + add(pr1763Site(use, tm)) + add(homeBox(use, tm)) + } + } + type modeSteps struct{ mode, steps string } + all := []modeSteps{{"", "none"}, {"", "mixed"}, {"", "3p"}, {"auto", "none"}, {"auto", "mixed"}, {"auto", "3p"}, + {"3p", "none"}, {"3p", "mixed"}, {"3p", "3p"}, {"1p", "none"}, {"1p", "1p"}} + houses := []struct { + name string + a [3]float64 + }{{"balanced", houseBalanced}, {"L1", houseL1}} + for _, use := range []fieldUse{usePlanned, useNow} { + for _, fuse := range []float64{16, 25} { + for _, h := range houses { + for _, ms := range all { + for _, car := range []int{3, 1} { + if ms.mode == "1p" && car == 1 { + continue + } + add(fieldCase{report: "sweep", fuseA: fuse, mode: ms.mode, steps: ms.steps, timing: timingEasee, + carPhases: car, use: use, house: h.a, houseName: h.name}) + } + } + } + } + } + for _, use := range []fieldUse{usePlanned, useNow, useSurplus, usePause, useRestart} { + for _, fuse := range []float64{16, 25} { + for _, ms := range []modeSteps{{"", "none"}, {"auto", "mixed"}, {"3p", "3p"}} { + for _, tm := range fieldTimings { + add(fieldCase{report: "timing", fuseA: fuse, mode: ms.mode, steps: ms.steps, timing: tm, + carPhases: 3, use: use, house: houseL1, houseName: "L1"}) + } + } + } + } + for _, c := range probeCases() { + add(c) + } + return out +} + +// probeCases are the edges srcfl/ftw#1799's review named, and the one-phase +// car that fix D is for. +func probeCases() []fieldCase { + var out []fieldCase + two := []fieldTiming{timingFast, timingEasee} + for _, tm := range two { + for _, use := range []fieldUse{usePlanned, useNow} { + // A table of one-phase levels with a maximum above it, on auto. + out = append(out, fieldCase{report: "probe", fuseA: 25, mode: "auto", steps: "1p-3", timing: tm, carPhases: 3, use: use, + house: houseBalanced, houseName: "balanced", charger: "1p32A"}) + } + for _, fuse := range []float64{16, 25} { + // Phase mode unset, a goal, and a plan slot of 2,300 W. + out = append(out, fieldCase{report: "probe", fuseA: fuse, mode: "", steps: "none", timing: tm, carPhases: 3, use: usePlanned, + house: houseBalanced, houseName: "balanced", planW: 2300}) + } + // Surplus-only with a goal and 2.5 kW of export: one phase is allowed. + out = append(out, fieldCase{report: "probe", fuseA: 25, mode: "auto", steps: "mixed", timing: tm, carPhases: 3, use: useSurplus, + house: houseBalanced, houseName: "balanced", goal: true, exportW: []float64{2500, 2500, 3000, 2500}}) + // A house step on one phase under a three-phase car is not a one-phase car. + out = append(out, fieldCase{report: "probe", fuseA: 25, mode: "3p", steps: "3p", timing: tm, carPhases: 3, use: usePlanned, + house: houseBalanced, houseName: "balanced", houseStep: 30 * time.Minute}) + } + for _, tm := range []fieldTiming{timingFast, timingEasee, timingSlowCloud} { + // A held 1p order raised to 3p beside a phase at 20 A. + out = append(out, fieldCase{report: "probe", fuseA: 25, mode: "auto", steps: "mixed", timing: tm, carPhases: 3, use: usePhaseUp, + house: [3]float64{2, 20, 2}, houseName: "2-20-2", warm: true}) + } + for _, tm := range fieldTimings { + for _, use := range []fieldUse{usePlanned, useNow} { + // The help report's site with a one-phase car. + c := helpReport(use, tm) + c.carPhases = 1 + out = append(out, c) + } + } + for _, tm := range []fieldTiming{timingFast, timingEasee} { + // A small goal for the one-phase car: the plan picks its lowest + // level, which must be a third of an order the table has. + c := helpReport(usePlanned, tm) + c.carPhases, c.needW = 1, 1000 + out = append(out, c) + } + c := helpReport(usePlanned, timingEasee) + c.houseStep = 30 * time.Minute + return append(out, c) +} + +// Failure kinds: the invariants. The sitesim adapter's own checks give the +// first five. +const ( + failFuse = "fuse" // the main fuse blew + failLimit = "limit" // over Core's limit after an outside change, longer than the fastest lever + failOwnRaise = "own-raise" // a raise we sent kept a phase over the limit more than 3 s + failIdle = "idle" // the car idle with room for it and a request + failStale = "stale" // a raise on a stale meter + failCut = "cut" // the car's order cut for the fuse with every phase under the limit + failOverCut = "over-cut" // cut again for an overage that a cut the car has not applied yet covers + failEnergy = "energy" // a planned charge under 90 % of its energy by the deadline the fuse allowed + failOnePhase = "one-phase" // planned charging or Charge now put a three-phase charger on one phase + failVerdict = "verdict" // a charger drawing against its order not reported as not following + failHomeUse = "home-use" // Home use not measured while every device reads +) + +func failureKind(msg string) string { + for _, k := range []struct{ text, kind string }{ + {"the main fuse blew", failFuse}, {"a raise we sent kept", failOwnRaise}, + {"over Core's limit", failLimit}, {"the car is idle", failIdle}, {"on a meter", failStale}, + } { + if strings.Contains(msg, k.text) { + return k.kind + } + } + return "other" +} + +// fieldPlan is the planner's side of a deadline goal, the way the directive +// reaches the charger: at each 15 min slot the remaining energy at the +// smallest planning step that reaches the goal by the deadline, as the slot's +// Wh and its on-power. It replans from what the car has taken, as a real +// replan does from Core's session energy. +type fieldPlan struct { + id string + deadline time.Time + needWh float64 + planning func() []float64 // the steps main.go gives the planner + fixedW float64 // > 0: every slot asks this, as Wh only + slotStart time.Time + stepW, slotWh float64 + slotDelivered0 float64 + run *fieldRun +} + +const fieldSlot = 15 * time.Minute + +func (p *fieldPlan) slotEnd() time.Time { + end := p.slotStart.Add(fieldSlot) + if end.After(p.deadline) { + return p.deadline + } + return end +} + +func (p *fieldPlan) directive(now time.Time) (loadpoint.Directive, bool) { + if !now.Before(p.deadline) { + return loadpoint.Directive{}, false + } + if start := now.Truncate(fieldSlot); !start.Equal(p.slotStart) { + p.slotStart, p.slotDelivered0 = start, p.run.deliveredWh + p.stepW, p.slotWh = 0, 0 + left, hours := p.needWh-p.run.deliveredWh, p.deadline.Sub(start).Hours() + if p.fixedW > 0 { + p.stepW, p.slotWh = p.fixedW, p.fixedW*p.slotEnd().Sub(start).Hours() + } else if steps := p.planning(); left > 0 && hours > 0 && len(steps) > 0 { + p.stepW = slices.Max(steps) + for _, w := range steps { + if w > 0 && w >= left/hours { + p.stepW = w + break + } + } + p.slotWh = math.Min(left, p.stepW*p.slotEnd().Sub(start).Hours()) + } + } + d := loadpoint.Directive{SlotStart: p.slotStart, SlotEnd: p.slotEnd(), LoadpointEnergyWh: map[string]float64{p.id: p.slotWh}} + if p.fixedW == 0 { + d.LoadpointMaxPowerW = map[string]float64{p.id: p.stepW} + } + return d, true +} + +// fieldPlanningSteps is what the planner can choose for the session: the +// steps main.go gives it (planningLimitsForLoadpoint), within the minimum +// and maximum it gets with them, as mpc's normalizedSteps keeps them. +func fieldPlanningSteps(s *simSite) []float64 { + st, _ := s.mgr.State(s.lp.ID) + cfg := loadpoint.Config{MinChargeW: st.MinChargeW, MaxChargeW: st.MaxChargeW, AllowedStepsW: st.AllowedStepsW, PhaseMode: s.lp.PhaseMode} + minW, maxW, steps := loadpoint.PlanningLimits(st, cfg, loadpoint.SiteFuse{MaxAmps: s.p.Fuse.RatingA, Voltage: s.p.Voltage, PhaseCnt: 3}) + return slices.DeleteFunc(slices.Clone(steps), func(w float64) bool { return w > 0 && (w < minW || maxW > 0 && w > maxW) }) +} + +// asks reports whether the plan wants the car charging now: its slot has an +// on-power and more energy left than the charger's response time takes. +func (p *fieldPlan) asks(now time.Time, response time.Duration) bool { + if p.stepW <= 0 || !now.Before(p.deadline) { + return false + } + return p.slotWh-(p.run.deliveredWh-p.slotDelivered0) > p.stepW*response.Hours() +} + +// fieldRun holds a case's request and the checks the adapter does not do. +type fieldRun struct { + c fieldCase + holdPhase string + plan *fieldPlan + verdict func() (status, reason string) + paused bool + failures map[string]string // the first failure of each kind + + deliveredWh float64 + orderW float64 // the order last sent, and since when + orderAt time.Time + expectW float64 // the draw it asks for: expected_w for a one-phase car, else the order + mismatchSince time.Time // the car drawing against an order older than its response time + drawW float64 // the car's draw, and when it last changed + drawChangedAt time.Time + heatRoomSince time.Time // the phases' heat has left room to start since + nextLook time.Time + trace func(format string, args ...any) // FTW_FIELD_TRACE names the case + tracedReason string + exportPeak func(window time.Duration) (float64, bool) // surplus: the forecast from now +} + +// FieldVerdict returns a reader of the EV verdict the API shows for a store +// and loadpoints: status and reason. The api package imports control, so +// sitesim_field_api_test.go in package control_test sets it. +var FieldVerdict func(tel *telemetry.Store, lps *loadpoint.Manager, ctrl *loadpoint.Controller) func() (status, reason string) + +func (s *simSite) coreEvery() time.Duration { + if s.every > 0 { + return s.every + } + return simCoreEvery +} + +func (s *simSite) declaredMaxAge() time.Duration { + if s.field == nil { + return 0 + } + return s.field.c.timing.declared +} + +// evData is what a charger driver adds to its reading. +func (s *simSite) evData() map[string]any { + if s.field == nil { + return nil + } + d := map[string]any{"connected": s.ev.Plugged, "control_power_confirmed": true} + if age := s.field.c.timing.declared; age > 0 { + d["power_max_age_s"] = age.Seconds() + } + return d +} + +func (f *fieldRun) fail(s *simSite, kind, format string, args ...any) { + s.fail(format, args...) + f.harvest(s, kind) +} + +// harvest files the adapter's failure under its kind and clears it, so a +// case runs to its end and reports every kind it breaks. +func (f *fieldRun) harvest(s *simSite, kind string) { + if s.failure == "" { + return + } + if kind == "" { + kind = failureKind(s.failure) + } + if _, ok := f.failures[kind]; !ok { + f.failures[kind] = s.failure + } + if kind != failFuse { + s.failure = "" + } +} + +func (f *fieldRun) limitA(s *simSite) float64 { return s.p.Fuse.RatingA - evPhaseMarginA } + +// send records the command as the driver registry does and checks a cut. +func (f *fieldRun) send(ctx context.Context, s *simSite, driver string, payload []byte) error { + var cmd struct { + Action string `json:"action"` + PowerW float64 `json:"power_w"` + ExpectedW *float64 `json:"expected_w"` + } + ours := json.Unmarshal(payload, &cmd) == nil && cmd.Action == "ev_set_current" && driver == s.lp.DriverName + if ours && cmd.PowerW < s.offerW-1 && !s.p.Offline("evse") { + // Surplus follows the export, an outside step; its cuts are not judged. + if st, _ := s.mgr.State(s.lp.ID); st.CommandedReason == "fuse_limit" && !s.surplus { + f.checkCut(s, cmd.PowerW) + } + } + if st, _ := s.mgr.State(s.lp.ID); f.trace != nil && cmd.Action == "ev_set_current" && (cmd.PowerW != s.offerW || st.CommandedReason != f.tracedReason) { + f.tracedReason = st.CommandedReason + e := s.p.EVSE + f.trace("%s order %.0f W (%s); phases %.1f A, offer %.0f A on %d, car %.0f W on %d", + s.p.Now.Format("15:04:05"), cmd.PowerW, st.CommandedReason, s.p.A, e.OfferA, e.OfferPhases, e.DrawW, e.DrawPhases) + } + evidence := s.store.BeginCommand(driver, payload, s.p.Now) + err := s.send(ctx, driver, payload) + result := "accepted" + if err != nil { + result = "failed" + } + s.store.CompleteCommand(evidence, result) + expectW := f.expectW + if ours && err == nil { + expectW = s.offerW + if cmd.ExpectedW != nil { + expectW = *cmd.ExpectedW + } + } + if s.offerW != f.orderW || expectW != f.expectW { + f.orderW, f.expectW, f.orderAt = s.offerW, expectW, s.p.Now + } + return err +} + +// checkCut judges a cut for the fuse against the meter Core read. With +// every phase under the limit nothing measured asks for it: only orders +// that cannot have shown yet, which may hold back a raise and never cut. +// Over the limit, a cut the car has not applied yet that covers the +// overage already answers it; a second one takes the car down twice. +func (f *fieldRun) checkCut(s *simSite, w float64) { + limit, e, over := f.limitA(s), s.p.EVSE, false + for i, a := range s.meter.PhaseA { + over = over || math.Abs(a) >= limit || s.heat[i] >= limit*limit + } + if !over { + f.fail(s, failCut, "cut the car %.0f -> %.0f W for the fuse with every phase under %.0f A (meter %.1f A)", + s.offerW, w, limit, s.meter.PhaseA) + return + } + queuedA, queuedPhases, queued := e.Queued() + if !queued || e.DrawW <= 0 { + return + } + drawA := e.DrawW / s.p.Voltage / float64(e.DrawPhases) + carA := s.p.Car.MaxW / s.p.Voltage / float64(min(max(queuedPhases, 1), e.Phases, s.p.Car.Phases)) + covered := true + for i, a := range s.meter.PhaseA { + after := 0.0 + if i < min(max(queuedPhases, 1), e.Phases, s.p.Car.Phases) && queuedA >= e.MinA { + after = math.Min(queuedA, carA) + } + before := 0.0 + if i < e.DrawPhases { + before = drawA + } + if overA := math.Abs(a) - limit; overA > 0 && before-after < overA { + covered = false + } + } + if covered { + f.fail(s, failOverCut, "cut the car %.0f -> %.0f W while a cut to %.0f A it has not applied covers the overage (meter %.1f A, limit %.0f A)", + s.offerW, w, queuedA, s.meter.PhaseA, limit) + } +} + +// wantsCharge adds the request to the adapter's room test: a planned charge +// asks only while its slot has energy left, and a pause asks for nothing. +// After an overage Core waits for the importing phases' heat (I² over 60 s, +// the fuse's own memory) to leave room for the car's minimum and a step, for +// evRestartDwell, as its start rule does. +func (f *fieldRun) wantsCharge(s *simSite, baseW float64) bool { + p, limit := s.p, f.limitA(s) + room := true + for i := range min(p.EVSE.Phases, p.Car.Phases) { + level := p.A[i] // A phase that exports has room for import, whatever its heat. + if level > 0 { + level = math.Max(level, math.Sqrt(s.heat[i])) + } + room = room && level+p.EVSE.MinA+evPhaseMarginA <= limit + } + switch { + case !room: + f.heatRoomSince = time.Time{} + case f.heatRoomSince.IsZero(): + f.heatRoomSince = p.Now + } + switch { + case f.paused, !room, p.Now.Sub(f.heatRoomSince) < evRestartDwell && p.EVSE.DrawW <= 0: + return false + case f.plan != nil: + return f.plan.asks(p.Now, p.EVSE.Delay+time.Minute) + case s.surplus: + return -baseW >= f.surplusMinW(s) + } + return true +} + +// surplusMinW is the least export surplus charging can use: one phase at +// 6 A where the loadpoint may run on one phase and the next 30 min of +// forecast stay below three phases at 6 A, as pickSurplusSteps decides; +// else the car's phases. +func (f *fieldRun) surplusMinW(s *simSite) float64 { + e, phases := s.p.EVSE, min(s.p.EVSE.Phases, s.p.Car.Phases) + min3 := 3 * s.p.Voltage * e.MinA + onePhase := s.lp.PhaseMode != "3p" && slices.ContainsFunc(s.lp.AllowedStepsW, func(w float64) bool { return w > 0 && w < min3 }) + near, ok := 0.0, false + if f.exportPeak != nil { + near, ok = f.exportPeak(30 * time.Minute) + } + if onePhase && (!ok || near < min3) || s.lp.PhaseMode == "1p" { + phases = 1 + } + return e.MinA * s.p.Voltage * float64(phases) +} + +// response is how long Core may wait for the charger's power to show an +// order: what its driver declares, or Easee's 180 s when it declares nothing. +func (f *fieldRun) response() time.Duration { + if d := f.c.timing.declared; d > 0 { + return max(d, f.c.timing.delay) + } + return max(evUnknownSettle, f.c.timing.delay) +} + +func (f *fieldRun) check(s *simSite) { + p, now := s.p, s.p.Now + f.harvest(s, "") + f.deliveredWh += p.EVSE.DrawW * sitesim.Step.Hours() + if p.EVSE.DrawW != f.drawW { + f.drawW, f.drawChangedAt = p.EVSE.DrawW, now + } + if f.c.use != useSurplus && f.c.use != usePhaseUp && p.EVSE.Phases == 3 && p.EVSE.OfferPhases == 1 && p.EVSE.OfferA > 0 { + f.fail(s, failOnePhase, "%s put the three-phase charger on one phase at %.0f A", f.c.use, p.EVSE.OfferA) + } + up := s.downUntil.IsZero() + if up && p.Car.Plugged && now.Sub(f.orderAt) >= f.response() && math.Abs(p.EVSE.DrawW-f.expectW) > telemetry.ControlToleranceW(f.expectW) { + if f.mismatchSince.IsZero() { + f.mismatchSince = now + } + } else { + f.mismatchSince = time.Time{} + } + if !up || now.Before(f.nextLook) { + return + } + f.nextLook = now.Add(10 * time.Second) + // A charger that reports late leaves the balance negative until its + // reading catches up with the meter; that is not a failed read. + // The plant's PV has no driver of its own, so Home use cannot be + // measured while it produces. + if s.meterAt.Equal(now) && p.PVW == 0 && now.Sub(f.drawChangedAt) > p.EVSE.ReadbackLag+2*sitesim.Step { + if live := s.store.LiveMeasurementNow("meter", telemetry.ForecastOptions{}); live.LoadQuality != telemetry.LoadMeasured { + f.fail(s, failHomeUse, "Home use is %s (%s) while every device reads", live.LoadQuality, live.LoadReason) + } + } + // Core waits its response time for an order to take hold, then two + // minutes of measured disagreement (srcfl/ftw#1795). A car marked + // one-phase is judged against a third of a three-phase order, the + // expected_w Core sends with it (srcfl/ftw#1801). + if f.verdict != nil && !f.mismatchSince.IsZero() && now.Sub(f.mismatchSince) >= 2*time.Minute+30*time.Second { + if status, reason := f.verdict(); status != "not_following" && reason != "one_phase" { + f.fail(s, failVerdict, "the car draws %.0f W against an expected %.0f W (%v), the verdict says %s/%s", + p.EVSE.DrawW, f.expectW, now.Sub(f.orderAt).Round(time.Second), status, reason) + } + } +} + +// fieldRoomW is the most the car can draw under Core's limit with the +// house's steady load: what a deadline goal can be held to. +func fieldRoomW(c fieldCase, e *sitesim.EVSE, car *sitesim.Car) float64 { + phases := min(e.Phases, car.Phases) + worst := 0.0 + for i := range phases { + worst = math.Max(worst, c.house[i]) + } + roomA := math.Min(c.fuseA-evPhaseMarginA-worst, e.MaxA) + if roomA < e.MinA { + return 0 + } + return math.Min(car.MaxW, roomA*230*float64(phases)) +} + +type fieldAction struct { + at time.Duration + do func(*simSite) +} + +// runFieldCase builds the site and runs the request; the run holds the +// failures by kind. It runs inside a synctest bubble. +func runFieldCase(c fieldCase, trace func(string, ...any)) (*simSite, *fieldRun) { + start := time.Date(2026, 10, 21, 22, 0, 0, 0, time.Local) + time.Sleep(start.Sub(time.Now())) // The bubble's clock becomes the plant's. + p := sitesim.New(start, c.fuseA) + phases := 3 + if c.mode == "1p" { + phases = 1 + } + p.EVSE = &sitesim.EVSE{Phases: phases, MinA: 6, MaxA: 16, Delay: c.timing.delay} + if c.charger == "1p32A" { + p.EVSE.Phases, p.EVSE.MaxA = 1, 32 + } + if c.timing.cloud { + p.EVSE.ReadbackLag, p.EVSE.SplitAtA, p.EVSE.PhaseHold = 10*time.Second, c.fuseA, 90*time.Second + } + p.Car = &sitesim.Car{Phases: c.carPhases, MaxW: 11040, Plugged: true} + if c.carPhases == 1 { + p.Car.MaxW = 3680 + } + if c.car != nil { + c.car(p.Car) + } + p.HouseW = houseA(c.house[0], c.house[1], c.house[2]) + if c.warm { + if err := p.Fuse.Preheat(c.house); err != nil { + panic(err) + } + } + if c.battery { + p.Battery = &sitesim.Battery{Phases: 3, MaxChargeW: 10000, MaxDischargeW: 10000, CapacityWh: 20000, SoC: .3, Latency: 3 * time.Second} + } + lp := fieldLoadpoint(c) + f := &fieldRun{c: c, failures: map[string]string{}, holdPhase: "3p", trace: trace} + if c.mode == "1p" { + f.holdPhase = "1p" // The app sends the loadpoint's phases (web/app.js). + } + s := &simSite{p: p, lp: lp, surplus: c.use == useSurplus, every: c.every, field: f} + + run, kettle := 20*time.Minute, []time.Duration{8 * time.Minute} + outside := func(do func(*sitesim.Plant)) func(*sitesim.Plant) { + return func(p *sitesim.Plant) { do(p); s.outsideAt = p.Now } + } + var actions []fieldAction + var exportPlan []float64 // surplus: each quarter hour's export without the car + switch c.use { + case usePlanned, useRestart: + run, kettle = 3*time.Hour, []time.Duration{50 * time.Minute, 140 * time.Minute} + if c.use == useRestart { + run, kettle = 90*time.Minute, []time.Duration{50 * time.Minute} + p.Events = append(p.Events, sitesim.Event{At: 30 * time.Minute, Kind: sitesim.CoreRestart, For: 20 * time.Second}) + } + roomW, askW := fieldRoomW(c, p.EVSE, p.Car), 3300.0 + if c.planW > 0 { + askW = c.planW + } + if c.needW > 0 { + askW = c.needW + } + f.plan = &fieldPlan{id: lp.ID, deadline: start.Add(run), run: f, planning: func() []float64 { return fieldPlanningSteps(s) }, + fixedW: c.planW, needWh: math.Min(askW*run.Hours(), .8*roomW*run.Hours())} + s.plan = f.plan.directive + case usePause: + run, kettle = 15*time.Minute, []time.Duration{11 * time.Minute} + actions = []fieldAction{ + {5 * time.Minute, func(s *simSite) { + f.paused = true + s.ctrl.SetManualHold(lp.ID, loadpoint.ManualHold{PowerW: 0, PhaseMode: f.holdPhase, Persistent: true, StartedAt: s.p.Now, UpdatedAt: s.p.Now}) + }}, + {5*time.Minute + 25*time.Second, func(s *simSite) { + f.paused = false + s.ctrl.SetManualHold(lp.ID, loadpoint.ManualHold{PowerW: lp.MaxChargeW, PhaseMode: f.holdPhase, Persistent: true, StartedAt: s.p.Now, UpdatedAt: s.p.Now}) + }}, + } + case usePhaseUp: + run, kettle = 15*time.Minute, nil + actions = []fieldAction{{5 * time.Minute, func(s *simSite) { + s.ctrl.SetManualHold(lp.ID, loadpoint.ManualHold{PowerW: lp.MaxChargeW, PhaseMode: "3p", Persistent: true, StartedAt: s.p.Now, UpdatedAt: s.p.Now}) + }}} + case useSurplus: + run, kettle = time.Hour, nil + // Export steps up and down, never past the fuse with the car off. + houseW, maxPVW := p.HouseW[0]+p.HouseW[1]+p.HouseW[2], math.Inf(1) + for _, a := range c.house { + maxPVW = math.Min(maxPVW, 3*230*(c.fuseA-2*evPhaseMarginA+a)) + } + exports := []float64{2500, 6000, 9000, 3000} + if len(c.exportW) > 0 { + exports = c.exportW + } + for i, exportW := range exports { + pvW := -math.Min(houseW+exportW, maxPVW) + exportPlan = append(exportPlan, -pvW-houseW) + p.Events = append(p.Events, sitesim.Event{At: time.Duration(i) * 15 * time.Minute, Do: outside(func(p *sitesim.Plant) { p.PVW = pvW })}) + } + } + // A kettle on L1 for three minutes: an outside step Core must make room for. + for _, at := range kettle { + p.Events = append(p.Events, + sitesim.Event{At: at, Do: outside(func(p *sitesim.Plant) { p.HouseW[0] += 6 * 230 })}, + sitesim.Event{At: at + 3*time.Minute, Do: outside(func(p *sitesim.Plant) { p.HouseW[0] -= 6 * 230 })}) + } + if c.houseStep > 0 { + p.Events = append(p.Events, + sitesim.Event{At: c.houseStep, Do: outside(func(p *sitesim.Plant) { p.HouseW[0] += 8 * 230 })}, + sitesim.Event{At: c.houseStep + 10*time.Minute, Do: outside(func(p *sitesim.Plant) { p.HouseW[0] -= 8 * 230 })}) + } + slices.SortStableFunc(p.Events, func(a, b sitesim.Event) int { return cmp.Compare(a.At, b.At) }) + + s.setup = func(s *simSite) { + // The periodic order goes through the outcome sender, as main.go + // wires it to the driver registry, which records command evidence. + s.ctrl.SetOutcomeSender(func(ctx context.Context, driver string, payload []byte, outcome func(error)) error { + err := f.send(ctx, s, driver, payload) + if outcome != nil { + outcome(err) + } + return err + }) + switch c.use { + case useNow, usePause: + // Charge now from the app, which names the loadpoint's phases (web/app.js). + s.ctrl.SetManualHold(lp.ID, loadpoint.ManualHold{PowerW: lp.MaxChargeW, PhaseMode: f.holdPhase, Persistent: true, StartedAt: s.p.Now, UpdatedAt: s.p.Now}) + case usePhaseUp: + s.ctrl.SetManualHold(lp.ID, loadpoint.ManualHold{PowerW: 3680, PhaseMode: "1p", Persistent: true, StartedAt: s.p.Now, UpdatedAt: s.p.Now}) + } + if f.plan != nil || c.goal { + deadline := start.Add(run).Local() + s.mgr.SetSchedule(lp.ID, loadpoint.NewSchedule(1, deadline.Hour()*60+deadline.Minute(), false, 0, false)) + } + if c.battery { + fieldBatteryPlan(s, 6000) + } + if exportPlan != nil { + // The planner's PV forecast, which decides when surplus may use + // one phase: here it knows the export to come. + peak := func(from, to time.Duration) (float64, bool) { + w, ok := 0.0, false + for i, e := range exportPlan { + at := time.Duration(i) * 15 * time.Minute + if at+15*time.Minute > from && at <= to { + w, ok = math.Max(w, e), true + } + } + return w, ok + } + f.exportPeak = func(window time.Duration) (float64, bool) { + return peak(s.p.Now.Sub(start), s.p.Now.Sub(start)+window) + } + s.ctrl.SetNearTermPeakSurplusW(f.exportPeak) + s.ctrl.SetPeakRemainingSurplusW(func() (float64, bool) { return peak(s.p.Now.Sub(start), run) }) + } + if FieldVerdict != nil { + f.verdict = FieldVerdict(s.store, s.mgr, s.ctrl) + } + } + s.boot() + for end := start.Add(run); s.p.Now.Before(end) && f.failures[failFuse] == ""; { + for len(actions) > 0 && s.p.Now.Sub(start) >= actions[0].at && s.downUntil.IsZero() { + actions[0].do(s) + actions = actions[1:] + } + time.Sleep(sitesim.Step) + s.step() + } + if f.plan != nil && f.plan.needWh > 0 && f.deliveredWh < .9*f.plan.needWh { + f.fail(s, failEnergy, "the planned charge delivered %.1f of %.1f kWh by the deadline", f.deliveredWh/1000, f.plan.needWh/1000) + } + return s, f +} + +// fieldBatteryPlan has the plan charge the home battery at w in every slot, +// with the car's slot in the same directive as the planner sends it. +func fieldBatteryPlan(s *simSite, w float64) { + s.state.Mode, s.state.UseEnergyDispatch = ModePlannerArbitrage, true + s.state.SlotDirective = func(now time.Time) (SlotDirective, bool) { + start := now.Truncate(fieldSlot) + d := SlotDirective{SlotStart: start, SlotEnd: start.Add(fieldSlot), BatteryEnergyWh: w * fieldSlot.Hours(), Strategy: "arbitrage"} + if s.plan != nil { + if ev, ok := s.plan(now); ok { + d.LoadpointEnergyWh, d.LoadpointMaxPowerW = maps.Clone(ev.LoadpointEnergyWh), maps.Clone(ev.LoadpointMaxPowerW) + } + } + return d, true + } +} diff --git a/go/internal/control/sitesim_property_test.go b/go/internal/control/sitesim_property_test.go new file mode 100644 index 000000000..7fbb7fbd4 --- /dev/null +++ b/go/internal/control/sitesim_property_test.go @@ -0,0 +1,90 @@ +package control + +import ( + "fmt" + "maps" + "math/rand/v2" + "slices" + "testing" + "time" +) + +// randomHouseKnown names the seeds that break an invariant on master, by +// kind, and the finding. A listed kind that passes fails the test until its +// line goes. The fuse's own kind is never listed. +var randomHouseKnown = map[uint64]map[string]string{ + // 25 A, 40 s charger, no declared response time (Core assumes 180 s). L2 + // steps to 30.8 A with the car at 8 A; a cut of 5 A a phase that landed + // 94 s earlier, in the same run over the limit, still counts as on its + // way, so Core cuts to 6 A, not 0, and L2 holds 28.8 A for another 40 s. + 2: {failLimit: "netting: a landed cut still covers the next overage, ev_fuse_budget.go:436"}, +} + +// Core keeps a random house under the fuse. For each fixed seed the house +// load on each phase steps at random to anywhere up to the fuse less 2 A, +// with the fuse warm from the first load, while a car charges on Charge now +// through a charger that answers in 2, 5 or 40 s. simSite.check holds the +// fuse's invariants every plant second for an hour: the fuse never blows, +// and no phase stays over the fuse less its margin longer than the +// charger's response plus 3 s, or 3 s after a raise of Core's own. Idle +// time with room is logged, not judged: the field matrix judges it with the +// fuse's own memory. +func TestSiteSimRandomHouse(t *testing.T) { + for seed := uint64(1); seed <= 5; seed++ { + t.Run(fmt.Sprint(seed), func(t *testing.T) { + t.Parallel() + rng := rand.New(rand.NewPCG(seed, 0)) + p := simPlant(simStart) + p.Fuse.RatingA = []float64{16, 20, 25}[rng.IntN(3)] + p.EVSE.Delay = []time.Duration{2 * time.Second, 5 * time.Second, 40 * time.Second}[rng.IntN(3)] + house := func() [3]float64 { + var a [3]float64 + for i := range a { + a[i] = rng.Float64() * (p.Fuse.RatingA - 2) + } + return a + } + a := house() + p.HouseW = houseA(a[0], a[1], a[2]) + if err := p.Fuse.Preheat(a); err != nil { + t.Fatal(err) + } + s := &simSite{p: p, lp: simLoadpoint()} + s.boot() + failures, steps, damage := map[string]string{}, 0, 0.0 + next := func() time.Time { return p.Now.Add(time.Duration(15+rng.IntN(166)) * time.Second) } + for at, end := next(), simStart.Add(time.Hour); p.Now.Before(end); { + s.step() + if s.failure != "" { + // Keep each kind's first and run on, so a later blow still shows. + if k := failureKind(s.failure); failures[k] == "" { + failures[k] = s.failure + } + s.failure, s.overSince, s.limitSince, s.needSince = "", time.Time{}, time.Time{}, time.Time{} + } + damage = max(damage, p.Fuse.Damage[0], p.Fuse.Damage[1], p.Fuse.Damage[2]) + if !p.Now.Before(at) { + a, at = house(), next() + p.HouseW, s.outsideAt = houseA(a[0], a[1], a[2]), p.Now + steps++ + } + } + t.Logf("%.0f A fuse, charger %v, %d house steps: peak phase %.1f A, %v over %.0f A, fuse damage at most %.2f, car idle with room %v, %.1f kWh to the car", + p.Fuse.RatingA, p.EVSE.Delay, steps, s.peakA, s.overLimit, p.Fuse.RatingA-evPhaseMarginA, damage, s.idleWithRoom, p.EVSE.SessionWh/1000) + delete(failures, failIdle) + known := randomHouseKnown[seed] + for _, k := range slices.Sorted(maps.Keys(failures)) { + if fix, ok := known[k]; ok { + t.Logf("fails as known (%s): %s", fix, failures[k]) + } else { + t.Error(failures[k]) + } + } + for _, k := range slices.Sorted(maps.Keys(known)) { + if failures[k] == "" { + t.Errorf("%s passes now: delete its known line (%s)", k, known[k]) + } + } + }) + } +} diff --git a/go/internal/control/sitesim_test.go b/go/internal/control/sitesim_test.go new file mode 100644 index 000000000..61e0fba67 --- /dev/null +++ b/go/internal/control/sitesim_test.go @@ -0,0 +1,793 @@ +package control + +import ( + "cmp" + "context" + "encoding/json" + "errors" + "fmt" + "maps" + "math" + "slices" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/loadpoint" + "github.com/srcfl/ftw/go/internal/sitesim" + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// simCoreEvery is control_interval_s's default. The plant steps every second. +const simCoreEvery = 2 * time.Second + +// simSite runs Core's control tick against a sitesim.Plant in main.go's +// order: site import, charger tick, then battery dispatch. Core's settings +// are the config defaults. Every plant second it checks the invariants and +// keeps the first one broken. +type simSite struct { + p *sitesim.Plant + lp loadpoint.Config + surplus bool // the car charges from surplus; otherwise Charge now while plugged + plan loadpoint.PlanFunc // nil: no plan, and the car charges on Charge now + setup func(*simSite) // after each boot + + db memConfigStore // state.db: survives a restart + store *telemetry.Store + state *State + mgr *loadpoint.Manager + ctrl *loadpoint.Controller + + every time.Duration // control_interval_s; zero: simCoreEvery + field *fieldRun // the field matrix's own checks; nil outside it + + downUntil, nextTick time.Time + meter sitesim.MeterReading + meterAt, evAt time.Time // last reading + ev sitesim.EVReading + seen map[string]time.Time // each driver's last reading + offerW, batteryCmdW float64 // our last commands + raisedAt time.Time // our last command that raised import + outsideAt time.Time // the last outside step a scenario marked + defaulted bool + commands []simCommand + + hourStart time.Time // the plant's clock hour + hourWh float64 // its import, as the plant measured it + + heat [3]float64 // the phases' I², filtered as Core's heat model does + heatSet bool + overSince, needSince time.Time + limitSince time.Time // a phase above the fuse less its margin since + ownRaise bool // that run followed a raise we sent + peakA float64 + overLimit time.Duration // time with a phase over the fuse less its margin + idleWithRoom time.Duration // time the car was idle with room for it + overAt time.Time // last second a phase was over the fuse less its margin + cutsWithRoom int // car orders lowered with no phase over that limit within the response time + failure string +} + +type simCommand struct { + at time.Time + device string + w float64 +} + +func (c simCommand) String() string { + return fmt.Sprintf("%s %.0f W at %s", c.device, c.w, c.at.Format("Jan 2 15:04:05")) +} + +// boot starts Core with no memory. The devices keep their last command. +func (s *simSite) boot() { + s.store, s.seen, s.evAt, s.defaulted = telemetry.NewStore(), map[string]time.Time{}, time.Time{}, false + s.state = NewState(0, 42, "meter") + s.state.clock = func() time.Time { return s.p.Now } + s.state.SlewRateW, s.state.MinDispatchIntervalS = 3000, 2 + s.state.SiteFuseAmps, s.state.SiteFuseVoltage, s.state.SiteFusePhases, s.state.SiteFuseSafetyA = s.p.Fuse.RatingA, s.p.Voltage, 3, .5 + if b := s.p.Battery; b != nil { + s.state.DriverLimits = map[string]PowerLimits{"battery": {MaxChargeW: b.MaxChargeW, MaxDischargeW: b.MaxDischargeW}} + } + s.mgr = loadpoint.NewManager() + s.mgr.SetNowFn(s.state.clock) + s.mgr.Load([]loadpoint.Config{s.lp}) + if s.db == nil { + s.db = memConfigStore{} + } + s.mgr.SetSessionStore(s.db) + plan := s.plan + if plan == nil { + plan = func(time.Time) (loadpoint.Directive, bool) { return loadpoint.Directive{}, false } + } + s.ctrl = loadpoint.NewController(s.mgr, plan, s.evSample, s.send) + s.ctrl.SetDriverOnline(func(d string) bool { h := s.store.DriverHealth(d); return h != nil && h.IsOnline() }) + s.ctrl.SetSiteFuse(loadpoint.SiteFuse{MaxAmps: s.p.Fuse.RatingA, Voltage: s.p.Voltage, PhaseCnt: 3}) + s.ctrl.SetFuseEVMax(func() (float64, bool) { + return EVFuseBudget(s.store, s.state, s.mgr.States(), s.ctrl.FuseConfigs(), s.fuseMaxW()) + }) + s.ctrl.SetFusePhaseRoomA(func() (float64, bool) { return EVFusePhaseRoomA(s.state) }) + s.ctrl.SetFuseLimitReason(func() string { return EVLimitReason(s.state) }) + s.ctrl.SetSitePhases(func() (loadpoint.SitePhases, bool) { return SitePhases(s.store, s.state) }) + s.ctrl.SetSiteSurplusForEV(func() (float64, bool) { + if !s.meterFresh() { + return 0, false + } + var batW float64 + for _, r := range s.store.ReadingsByType(telemetry.DerBattery) { + if h := s.store.DriverHealth(r.Driver); h != nil && h.Status != telemetry.StatusOffline { + batW += r.SmoothedW + } + } + meterW := s.store.Get("meter", telemetry.DerMeter).SmoothedW + return loadpoint.SurplusAvailableForEVW(meterW, batW, s.store.SumOnlineEVW(), s.ctrl.AnyLoadpointSurplusActive()), true + }) + s.ctrl.SetBatSoCProvider(func() (float64, bool) { + r := s.store.Get("battery", telemetry.DerBattery) + if h := s.store.DriverHealth("battery"); r == nil || r.SoC == nil || h == nil || !h.IsOnline() { + return 0, false + } + return *r.SoC, true + }) + if s.setup != nil { + s.setup(s) + } +} + +func (s *simSite) fuseMaxW() float64 { return s.p.Fuse.RatingA * s.p.Voltage * 3 } + +// meterFresh is Core's view: a site reading inside the 60 s watchdog. +func (s *simSite) meterFresh() bool { + m := s.store.Get("meter", telemetry.DerMeter) + return m != nil && s.p.Now.Sub(m.UpdatedAt) <= time.Minute +} + +func (s *simSite) evSample(string) (loadpoint.EVSample, bool) { + if s.evAt.IsZero() || s.p.Now.Sub(s.evAt) > time.Minute { + return loadpoint.EVSample{}, false + } + return loadpoint.EVSample{Connected: s.ev.Plugged, RequestActive: true, PowerW: s.ev.W, PowerAt: s.evAt, PowerMaxAge: s.declaredMaxAge(), + DeviceID: "easee:SIM1", SessionWh: s.ev.SessionWh, EnergyAt: s.evAt}, true +} + +func (s *simSite) send(_ context.Context, driver string, payload []byte) error { + var cmd struct { + Action string `json:"action"` + PowerW float64 `json:"power_w"` + PhaseMode string `json:"phase_mode"` + SplitW float64 `json:"phase_split_w"` + MaxA float64 `json:"max_amps_per_phase"` + } + if err := json.Unmarshal(payload, &cmd); err != nil { + return err + } + if driver != s.lp.DriverName || s.p.Offline("evse") { + return errors.New("sitesim: no such charger online") + } + if cmd.Action != "ev_set_current" { + return nil + } + // (c) A site meter older than the watchdog raises nothing. + if age := s.p.Now.Sub(s.meterAt); age > time.Minute && cmd.PowerW > s.offerW { + s.fail("raised the car %.0f -> %.0f W on a meter %v old", s.offerW, cmd.PowerW, age) + } + if cmd.PowerW > s.offerW { + s.raisedAt = s.p.Now + } + if cmd.PowerW < s.offerW && (s.overAt.IsZero() || s.p.Now.Sub(s.overAt) > s.responseTime(true)) { + s.cutsWithRoom++ + } + s.offerW = cmd.PowerW + s.commands = append(s.commands, simCommand{s.p.Now, "evse", cmd.PowerW}) + // Without a split the Easee driver splits auto at the fuse's current + // on one phase (easee_cloud.lua pick_phases). + if cmd.SplitW <= 0 && cmd.MaxA > 0 { + cmd.SplitW = s.p.Voltage * cmd.MaxA + } + s.p.EVSE.Command(s.p.Now, cmd.PowerW, cmd.PhaseMode, cmd.SplitW, s.p.Voltage) + return nil +} + +func (s *simSite) emit(driver string, kind telemetry.DerType, w float64, soc *float64, extra map[string]any) { + d := map[string]any{"control_power_available": true, "control_power_observed_at": s.p.Now.Format(time.RFC3339Nano)} + maps.Copy(d, extra) + data, _ := json.Marshal(d) + s.store.Update(driver, kind, w, soc, data) + r := s.store.Get(driver, kind) + r.UpdatedAt = s.p.Now // Receipt on the plant's clock, as are SoC times: + if soc != nil { // Update stamps both with the wall clock. + r.SoCUpdatedAt = s.p.Now + } + s.store.RecordDriverSuccess(driver) + s.seen[driver] = s.p.Now +} + +func (s *simSite) step() { + tick := s.p.Step() + now := s.p.Now + if tick.Meter != nil { + s.meter, s.meterAt = *tick.Meter, now + } + if tick.Restart > 0 { + s.downUntil = now.Add(tick.Restart) + } + if !s.downUntil.IsZero() && !now.Before(s.downUntil) { + s.downUntil = time.Time{} + s.boot() + } + if s.downUntil.IsZero() { + if m := tick.Meter; m != nil { + s.emit("meter", telemetry.DerMeter, m.W, nil, map[string]any{"l1_a": m.PhaseA[0], "l2_a": m.PhaseA[1], "l3_a": m.PhaseA[2]}) + } + if b := tick.Battery; b != nil { + s.emit("battery", telemetry.DerBattery, b.W, &b.SoC, nil) + } + if e := tick.EV; e != nil { + s.ev, s.evAt = *e, now + s.emit(s.lp.DriverName, telemetry.DerEV, e.W, nil, s.evData()) + } + if !now.Before(s.nextTick) { + s.nextTick = now.Add(s.coreEvery()) + s.control() + } + } + s.check() +} + +func (s *simSite) run(d time.Duration) { + for end := s.p.Now.Add(d); s.p.Now.Before(end) && s.failure == ""; { + s.step() + } +} + +// control is one tick of main.go's control loop. +func (s *simSite) control() { + now := s.p.Now + for d, at := range s.seen { + if h := s.store.DriverHealthMut(d); now.Sub(at) > time.Minute && h.IsOnline() { + h.SetOffline() // The watchdog. + } + } + if _, held := s.ctrl.GetManualHold(s.lp.ID, now); !s.surplus && s.plan == nil && !held && s.ev.Plugged { + s.ctrl.SetManualHold(s.lp.ID, loadpoint.ManualHold{PowerW: s.lp.MaxChargeW, PhaseMode: s.lp.PhaseMode, Persistent: true, StartedAt: now, UpdatedAt: now}) + } + allowed := s.meterFresh() + s.state.ObserveSiteImport(s.store) + s.ctrl.TickWithDispatch(context.Background(), now, allowed) + if !allowed { + if !s.defaulted { + s.batteryCommand(0) // The driver's default mode, sent once. + s.defaulted = true + } + return + } + s.defaulted = false + points := s.mgr.States() + for i := range points { + _, points[i].ManualActive = s.ctrl.GetManualHold(points[i].ID, now) + } + s.state.EVSurplusOnlyReserveW = loadpoint.SurplusReserveW(points, nil) + s.state.EVSurplusOnlyChargingW = loadpoint.SurplusChargingW(points) + s.state.EVCurtailHeadroomW = loadpoint.SurplusPotentialW(points) + s.state.SetEVCommanded(points) + s.state.SetEVFuseDemands(s.ctrl.FuseDemands()) + var capacityWh float64 + if b := s.p.Battery; b != nil { + capacityWh = b.CapacityWh + } + for _, target := range ComputeDispatch(s.store, s.state, map[string]float64{"battery": capacityWh}, s.fuseMaxW()) { + if target.Driver == "battery" { + s.batteryCommand(target.TargetW) + } + } +} + +// batteryCommand sends w. More than an amp of extra import on one phase +// counts as our raise; smaller moves are the PI's trim. +func (s *simSite) batteryCommand(w float64) { + b := s.p.Battery + if b == nil || s.p.Offline("battery") { + return + } + if w-s.batteryCmdW > s.p.Voltage { + s.raisedAt = s.p.Now + } + s.batteryCmdW = w + s.commands = append(s.commands, simCommand{s.p.Now, "battery", w}) + b.Command(s.p.Now, w) +} + +// responseTime is how long a command takes to show on the meter: a control +// tick, the device's delay and a meter reading. slowest picks the slower of +// the charger and the battery. +func (s *simSite) responseTime(slowest bool) time.Duration { + d := s.p.EVSE.Delay + if b := s.p.Battery; b != nil { + d = min(d, b.Latency) + if slowest { + d = max(s.p.EVSE.Delay, b.Latency) + } + } + return d + max(s.p.Meter.Every, sitesim.Step) + s.coreEvery() +} + +// check holds the invariants for this second. +func (s *simSite) check() { + p, now := s.p, s.p.Now + if hour := ImportHourStart(now.Add(-sitesim.Step)); !hour.Equal(s.hourStart) { + s.hourStart, s.hourWh = hour, 0 + } + s.hourWh += math.Max(0, (p.A[0]+p.A[1]+p.A[2])*p.Voltage) * sitesim.Step.Hours() + if p.Fuse.AnyBlown() { + s.fail("the main fuse blew: damage %.2f", p.Fuse.Damage) + } + // (a) A phase above the fuse less its margin after a raise we sent within + // the slowest response time is our own doing and may last 3 s. After an + // outside change, Core's own limit, read from the physical currents as + // fuseHeat reads the meter, may break for the fastest response time plus + // 3 s: a phase may sit above the fuse less its margin only while its + // I²-filtered current is under it, and never above the instant limit. + fuseA := p.Fuse.RatingA + limitA, k := fuseA-evPhaseMarginA, -math.Expm1(-sitesim.Step.Seconds()/fuseHeatTau.Seconds()) + over, overLimit := false, false + for i, a := range p.A { + if s.heatSet { + s.heat[i] += (a*a - s.heat[i]) * k + } else { + s.heat[i] = a * a + } + over = over || math.Abs(a) > fuseInstantFactor*fuseA || (math.Abs(a) > limitA && s.heat[i] > limitA*limitA) + overLimit = overLimit || math.Abs(a) > limitA + s.peakA = math.Max(s.peakA, math.Abs(a)) + } + s.heatSet = true + if overLimit { + s.overAt = now + s.overLimit += sitesim.Step + if s.limitSince.IsZero() { + s.limitSince, s.ownRaise = now, !s.raisedAt.IsZero() && now.Sub(s.raisedAt) <= s.responseTime(true) && !s.outsideAt.After(s.raisedAt) + } + } else { + s.limitSince = time.Time{} + } + if s.ownRaise && !s.limitSince.IsZero() && now.Sub(s.limitSince) > 3*time.Second { + s.fail("a raise we sent kept a phase over %.0f A for more than 3 s", limitA) + } + if !over { + s.overSince = time.Time{} + } else if s.overSince.IsZero() { + s.overSince = now + } + if allow := s.responseTime(false) + 3*time.Second; !s.overSince.IsZero() && now.Sub(s.overSince) > allow { + s.fail("over Core's limit for more than %v (heat %.1f A, limit %.0f A)", allow, math.Sqrt(max(s.heat[0], s.heat[1], s.heat[2])), limitA) + } + // (b) With the meter on schedule, room for the car's minimum and a car + // that asks, the car draws within a minute plus the charger's delay. + if !s.wantsCharge() { + s.needSince = time.Time{} + } else if s.needSince.IsZero() { + s.needSince = now + } + if !s.needSince.IsZero() && p.EVSE.DrawW <= 0 { + s.idleWithRoom += sitesim.Step + if wait := time.Minute + p.EVSE.Delay; now.Sub(s.needSince) > wait { + s.fail("the car is idle after %v with room for it", wait) + } + } + if s.field != nil { + s.field.check(s) + } +} + +// wantsCharge reports whether the car should charge now: it is plugged +// and asks, Core and the meter are running, and the uncontrolled load +// leaves room for the car's minimum on its phases and under the hour's +// import ceiling (and, for surplus charging, exports at least that much). +// Under the ceiling Core starts nothing in an hour's last quarter. +func (s *simSite) wantsCharge() bool { + p, e := s.p, s.p.EVSE + if p.Car == nil || !p.Car.Plugged || p.Offline("evse") || !s.downUntil.IsZero() || p.Now.Sub(s.meterAt) > max(p.Meter.Every, sitesim.Step)+sitesim.Step { + return false + } + phases := min(e.Phases, p.Car.Phases) + var baseW float64 + for i, houseW := range p.HouseW { + w := houseW + p.PVW/3 + baseW += w + if i < phases && math.Abs(w/p.Voltage+e.MinA) > p.Fuse.RatingA-evPhaseMarginA { + return false + } + } + minW := e.MinA * p.Voltage * float64(phases) + if ceilingW := s.state.PeakImportCeilingW; ceilingW > 0 { + left := s.hourStart.Add(time.Hour).Sub(p.Now) + if left < 15*time.Minute || baseW+minW > (ceilingW-s.hourWh)/left.Hours() { + return false + } + } + if s.field != nil { + return s.field.wantsCharge(s, baseW) + } + return !s.surplus || -baseW >= minW +} + +func (s *simSite) fail(format string, args ...any) { + if s.failure != "" { + return + } + st, _ := s.mgr.State(s.lp.ID) + s.failure = fmt.Sprintf("%v: "+format+"\n phases %.1f A, offer %.0f W (%s), car draws %.0f W, battery %.0f W", + append(append([]any{s.p.Now.Format("15:04:05")}, args...), s.p.A, s.offerW, st.CommandedReason, s.p.EVSE.DrawW, s.batteryW())...) +} + +func (s *simSite) batteryW() float64 { + if s.p.Battery == nil { + return 0 + } + return s.p.Battery.W +} + +// simScenario is one house. known names the finding that makes it fail on +// master: the row must keep failing, and fails once the fix lands until the +// fix deletes that line. expect checks the plant at times along the run. +type simScenario struct { + name string + known string + start time.Time // zero: simStart + run time.Duration + surplus bool + plan loadpoint.PlanFunc + plant func(*sitesim.Plant) + lp func(*loadpoint.Config) + setup func(*simSite) + expect []simExpect +} + +type simExpect struct { + at time.Duration + check func(*simSite) error +} + +var simStart = time.Date(2026, 10, 21, 12, 0, 0, 0, time.Local) + +// simPlant is a 25 A site with an Easee Cloud-like charger, which answers +// in 5 s and reports the draw 10 s late, and a 3p car that takes 16 A. +func simPlant(start time.Time) *sitesim.Plant { + p := sitesim.New(start, 25) + p.EVSE = &sitesim.EVSE{Phases: 3, MinA: 6, MaxA: 16, Delay: 5 * time.Second, ReadbackLag: 10 * time.Second} + p.Car = &sitesim.Car{Phases: 3, MaxW: 11040, Plugged: true} + return p +} + +func simLoadpoint() loadpoint.Config { + cfg := loadpoint.Config{ID: "garage", DriverName: "evse", MinChargeW: 4140, MaxChargeW: 11040, PhaseMode: "3p"} + for a := 0.0; a <= 16; a++ { + if a == 0 || a >= 6 { + cfg.AllowedStepsW = append(cfg.AllowedStepsW, a*690) + } + } + return cfg +} + +func houseA(l1, l2, l3 float64) [3]float64 { return [3]float64{l1 * 230, l2 * 230, l3 * 230} } + +// fieldNightLoadpoint is the 22 Oct 2026 field site's Easee: auto, with +// one-phase levels up to 3.8 kW and three-phase levels from 4.8 kW. +func fieldNightLoadpoint(c *loadpoint.Config) { + c.MinChargeW, c.MaxChargeW, c.PhaseMode = 1600, 11100, "auto" + c.AllowedStepsW = []float64{0, 1600, 1900, 2100, 2400, 2600, 2800, 3100, 3300, 3500, 3800, + 4800, 5500, 6200, 6900, 7600, 8300, 9000, 9700, 10400, 11100} +} + +// carPhasesUnknown checks that the session's car was never marked one-phase. +func carPhasesUnknown(s *simSite) error { + if st, _ := s.mgr.State(s.lp.ID); st.CarPhases != 0 { + return fmt.Errorf("car marked one-phase on L%d; car draws %.0f W on %d phases", st.CarPhase, s.p.EVSE.DrawW, s.p.EVSE.DrawPhases) + } + return nil +} + +// planEV is a saved goal and a plan that gives the car w in every quarter +// hour: the path Core takes overnight. +func planEV(w float64) loadpoint.PlanFunc { + return func(now time.Time) (loadpoint.Directive, bool) { + slot := now.Truncate(15 * time.Minute) + return loadpoint.Directive{SlotStart: slot, SlotEnd: slot.Add(15 * time.Minute), + LoadpointEnergyWh: map[string]float64{"garage": w / 4}}, true + } +} + +// planBatteryCharge has the plan charge the battery at w for the hour. +func planBatteryCharge(w float64) func(*simSite) { + return func(s *simSite) { + s.state.Mode, s.state.UseEnergyDispatch = ModePlannerArbitrage, true + slot := SlotDirective{SlotStart: s.p.Now, SlotEnd: s.p.Now.Add(time.Hour), BatteryEnergyWh: w, Strategy: "arbitrage"} + s.state.SlotDirective = func(time.Time) (SlotDirective, bool) { return slot, true } + } +} + +// simDay is a weekday from midnight in Self mode: the car is plugged +// overnight with Charge now, a 9 kW PV peak at 13:00 fills the 20 kWh +// battery by 12:45 and a heavy evening puts L1 at 13.7 A before the car. +// The battery lasts the heavy evening; "battery reaches its floor" is the +// case where it does not. Core restarts at 18:31, the battery driver drops +// out at 20:01 and the meter goes silent at 22:31. +func simDay(p *sitesim.Plant) { + p.Battery = &sitesim.Battery{Phases: 3, MaxChargeW: 5000, MaxDischargeW: 5000, CapacityWh: 20000, SoC: .5, Latency: 2 * time.Second} + for q := range 96 { + h, houseW, pvW := float64(q)/4, 250.0, 0.0 + switch { + case h >= 6.5 && h < 8.5: + houseW += 600 + case h >= 17 && h < 20: + houseW += 2500 + case h >= 20 && h < 22: + houseW += 1000 + } + if h > 7 && h < 19 { + pvW = -9000 * math.Sin(math.Pi*(h-7)/12) + } + plugged := h < 7 || h >= 17 + p.Events = append(p.Events, sitesim.Event{At: time.Duration(q) * 15 * time.Minute, Do: func(p *sitesim.Plant) { + p.HouseW, p.PVW, p.Car.Plugged = [3]float64{houseW + 400, houseW, houseW}, pvW, plugged + }}) + } + p.Events = append(p.Events, + sitesim.Event{At: 18*time.Hour + 31*time.Minute, Kind: sitesim.CoreRestart, For: 20 * time.Second}, + sitesim.Event{At: 20*time.Hour + time.Minute, Kind: sitesim.DriverOffline, Device: "battery", For: 2 * time.Minute}, + sitesim.Event{At: 22*time.Hour + 31*time.Minute, Kind: sitesim.MeterSilent, For: 90 * time.Second}) + slices.SortStableFunc(p.Events, func(a, b sitesim.Event) int { return cmp.Compare(a.At, b.At) }) +} + +func simScenarios() []simScenario { + slowCharger := func(delay time.Duration) func(*sitesim.Plant) { + return func(p *sitesim.Plant) { p.HouseW, p.EVSE.Delay = houseA(15, 15, 15), delay } + } + // 9 kW of export to a charger that reads back at once. + surplus := func(p *sitesim.Plant) { p.HouseW, p.PVW, p.EVSE.ReadbackLag = houseA(2, 2, 2), -9000-3*460, 0 } + surplusPhase := func(mode string) func(*loadpoint.Config) { + return func(c *loadpoint.Config) { + c.PhaseMode, c.SurplusOnly, c.MinChargeW, c.PhaseSplitW = mode, true, 1380, 3680 + steps := []float64{0} + for a := 6.0; a <= 16; a++ { + steps = append(steps, a*230) + } + c.AllowedStepsW = append(steps, c.AllowedStepsW[1:]...) + } + } + return []simScenario{ + {name: "export start", known: "export counted as import, fuse_heat.go:120", run: 5 * time.Minute, + plant: func(p *sitesim.Plant) { p.HouseW, p.PVW = houseA(2, 2, 2), -22*690 }}, + {name: "export start, unsigned phase currents", known: "export counted as import, fuse_heat.go:120", run: 5 * time.Minute, + plant: func(p *sitesim.Plant) { p.HouseW, p.PVW, p.Meter.Unsigned = houseA(2, 2, 2), -22*690, true }}, + {name: "charger's own load balancer", known: "a cut comes off an offer the car does not draw, ev_fuse_budget.go:172", run: 20 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW, p.EVSE.LoadBalancerA = houseA(12, 12, 12), 25 + p.Events = []sitesim.Event{ + {At: 5 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(18, 12, 12) }}, + {At: 10 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(4, 4, 4) }}, + {At: 15 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(20, 20, 20) }}, + } + }}, + {name: "slow cloud charger 60 s", run: 15 * time.Minute, + plant: slowCharger(time.Minute)}, + {name: "slow cloud charger 120 s", run: 15 * time.Minute, + plant: slowCharger(2 * time.Minute)}, + {name: "#1761 replay", run: 20 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW, p.Car.MaxW = houseA(1, 1, 1), 4200 + p.Battery = &sitesim.Battery{Phases: 3, MaxChargeW: 9400, MaxDischargeW: 9400, CapacityWh: 20000, SoC: .3, Latency: 3 * time.Second} + p.Events = []sitesim.Event{{At: 10 * time.Minute, Do: func(p *sitesim.Plant) { p.Car.MaxW = 11040 }}} + }, + lp: func(c *loadpoint.Config) { + c.MaxChargeW, c.AllowedStepsW = 10350, c.AllowedStepsW[:len(c.AllowedStepsW)-1] + }, + setup: planBatteryCharge(9400), + expect: []simExpect{{9*time.Minute + 50*time.Second, func(s *simSite) error { + p := s.p + offerW := p.EVSE.OfferA * p.Voltage * float64(p.EVSE.OfferPhases) + if offerW != 10350 || math.Abs(p.EVSE.DrawW-4200) > 50 || math.Abs(p.Battery.W-9400) > 200 { + return fmt.Errorf("not the #1761 state before the car takes its offer: offer %.0f W, car %.0f W, battery %.0f W", offerW, p.EVSE.DrawW, p.Battery.W) + } + return nil + }}}}, + {name: "meter every 30 s after a cut", known: "a meter slower than evSettle makes every reading a gap, ev_fuse_budget.go:141", run: 10 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW = houseA(3, 3, 3) + p.Events = []sitesim.Event{ + {At: 3 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(23, 23, 23) }}, + {At: 3*time.Minute + 3*time.Second, Do: func(p *sitesim.Plant) { p.HouseW, p.Meter.Every = houseA(3, 3, 3), 30*time.Second }}, + } + }}, + {name: "phase mode 3p, 9 kW surplus", run: 5 * time.Minute, surplus: true, plant: surplus, lp: surplusPhase("3p")}, + {name: "phase mode empty, 9 kW surplus", run: 5 * time.Minute, surplus: true, + plant: surplus, lp: surplusPhase("")}, + {name: "phase mode auto, 9 kW surplus", run: 5 * time.Minute, surplus: true, + plant: surplus, lp: surplusPhase("auto")}, + {name: "9 kW surplus, readback 10 s late", known: "surplus adds back the charger's late readback, surplus_reserve.go:128", run: 5 * time.Minute, surplus: true, + plant: func(p *sitesim.Plant) { surplus(p); p.EVSE.ReadbackLag = 10 * time.Second }, lp: surplusPhase("3p")}, + {name: "surplus_unlock_bat_soc saved, Charge now", run: 5 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW = houseA(3, 3, 3) + p.Battery = &sitesim.Battery{Phases: 3, MaxChargeW: 5000, MaxDischargeW: 5000, CapacityWh: 10000, SoC: .5} + }, + lp: func(c *loadpoint.Config) { c.PhaseMode = "" }, + setup: func(s *simSite) { + planBatteryCharge(0)(s) + s.mgr.SetSchedule(s.lp.ID, loadpoint.Schedule{SurplusUnlockBatSoC: .9}) + }}, + {name: "battery reaches its floor under Charge now", run: 5 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW = houseA(13.7, 12, 12) + p.Battery = &sitesim.Battery{Phases: 3, MaxChargeW: 5000, MaxDischargeW: 5000, CapacityWh: 10000, SoC: .07, Latency: 2 * time.Second} + }}, + {name: "end of a heavy hour, low phases", + start: time.Date(2026, 10, 21, 2, 0, 0, 0, time.Local), run: 65 * time.Minute, + plant: func(p *sitesim.Plant) { + p.HouseW = houseA(10, 10, 10) + p.Events = []sitesim.Event{ + {At: 9 * time.Minute, Kind: sitesim.CoreRestart, For: 20 * time.Second}, + {At: 50 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(1.5, 1.5, 1.5) }}, + } + }, + setup: func(s *simSite) { s.state.PeakImportCeilingW = 6000 }, + expect: []simExpect{{57 * time.Minute, func(s *simSite) error { + st, _ := s.mgr.State(s.lp.ID) + if s.p.EVSE.DrawW > 0 || st.CommandedReason != "import_limit" { + return fmt.Errorf("near the end of the hour: car draws %.0f W, reason %q, want idle on import_limit", s.p.EVSE.DrawW, st.CommandedReason) + } + return nil + }}}}, + // Field report 22 Oct 2026, v0.155.0-beta.1: a 3x20 A site, phase_mode + // auto, an Easee step table with one-phase levels, a plan for 11.1 kW. + // The car ran on one phase at 3.3 kW and FTW cut it with room on every + // phase. Planned charging now runs on three phases and only measured + // overage cuts. + {name: "night plan, auto, one-phase steps", start: time.Date(2026, 10, 22, 3, 0, 0, 0, time.Local), run: 40 * time.Minute, + plant: func(p *sitesim.Plant) { + p.Fuse.RatingA, p.HouseW, p.EVSE.Delay = 20, houseA(1.6, 1.7, 2.3), 40*time.Second + }, + lp: fieldNightLoadpoint, + plan: planEV(11100), + expect: []simExpect{ + {2 * time.Minute, func(s *simSite) error { + if s.p.EVSE.DrawPhases != 3 || s.p.EVSE.DrawW < 4140 { + return fmt.Errorf("car draws %.0f W on %d phases, want three", s.p.EVSE.DrawW, s.p.EVSE.DrawPhases) + } + return nil + }}, + {40*time.Minute - time.Second, func(s *simSite) error { + if s.cutsWithRoom > 0 || s.p.EVSE.SessionWh < 6000 { + return fmt.Errorf("%d cuts with room on every phase, %.2f kWh in 40 min", s.cutsWithRoom, s.p.EVSE.SessionWh/1000) + } + return nil + }}, + }}, + // A one-phase car (a plug-in hybrid) on the field night's table: the + // plan's 4.8 kW three-phase order is 7 A a phase, which the car takes + // on L1 alone, 1.6 kW. Once the meter shows it, the order is the + // current per phase the car should take, and it gets 3.7 kW as fast + // as the fuse limit lets the order rise. + {name: "one-phase car, field table", start: time.Date(2026, 10, 22, 3, 0, 0, 0, time.Local), run: 15 * time.Minute, + plant: func(p *sitesim.Plant) { + p.Fuse.RatingA, p.HouseW, p.EVSE.Delay = 20, houseA(1.6, 1.7, 2.3), 40*time.Second + p.Car.Phases, p.Car.MaxW = 1, 3700 + }, + lp: fieldNightLoadpoint, + plan: planEV(4800), + expect: []simExpect{{8 * time.Minute, func(s *simSite) error { + st, _ := s.mgr.State(s.lp.ID) + if st.CarPhases != 1 || st.CarPhase != 1 || s.p.EVSE.DrawW < 3600 { + return fmt.Errorf("car phases %d (L%d), car draws %.0f W", st.CarPhases, st.CarPhase, s.p.EVSE.DrawW) + } + return nil + }}}}, + // A house load on one phase as a three-phase car starts, and a + // two-phase car: neither is one-phase. + {name: "three-phase car, kettle on L1 at the start", start: time.Date(2026, 10, 22, 3, 0, 0, 0, time.Local), run: 10 * time.Minute, + plant: func(p *sitesim.Plant) { + p.Fuse.RatingA, p.HouseW, p.EVSE.Delay, p.Car.MaxW = 25, houseA(1, 1, 1), 40*time.Second, 7360 + p.Events = []sitesim.Event{ + {At: 5 * time.Second, Do: func(p *sitesim.Plant) { p.HouseW = houseA(10, 1, 1) }}, + {At: 4 * time.Minute, Do: func(p *sitesim.Plant) { p.HouseW = houseA(1, 1, 1) }}, + } + }, + lp: fieldNightLoadpoint, plan: planEV(11100), + expect: []simExpect{{10*time.Minute - time.Second, carPhasesUnknown}}}, + {name: "two-phase car", start: time.Date(2026, 10, 22, 3, 0, 0, 0, time.Local), run: 10 * time.Minute, + plant: func(p *sitesim.Plant) { + p.Fuse.RatingA, p.HouseW, p.EVSE.Delay = 25, houseA(1, 1, 1), 40*time.Second + p.Car.Phases, p.Car.MaxW = 2, 7360 + }, + lp: fieldNightLoadpoint, plan: planEV(11100), + expect: []simExpect{{10*time.Minute - time.Second, carPhasesUnknown}}}, + {name: "a day", start: time.Date(2026, 10, 21, 0, 0, 0, 0, time.Local), run: 24 * time.Hour, plant: simDay}, + } +} + +// runSimScenario runs sc from start and returns the site, with the first +// broken invariant or expectation in failure. +func runSimScenario(sc simScenario, start time.Time) *simSite { + p := simPlant(start) + if sc.plant != nil { + sc.plant(p) + } + lp := simLoadpoint() + if sc.lp != nil { + sc.lp(&lp) + } + s := &simSite{p: p, lp: lp, surplus: sc.surplus, plan: sc.plan, setup: sc.setup} + if sc.plan != nil { + setup := sc.setup + s.setup = func(s *simSite) { + s.mgr.SetSchedule(s.lp.ID, loadpoint.Schedule{SoC: .8}) + if setup != nil { + setup(s) + } + } + } + s.boot() + for _, e := range sc.expect { + s.run(start.Add(e.at).Sub(p.Now)) + if err := e.check(s); err != nil { + s.fail("%v", err) + } + } + s.run(start.Add(sc.run).Sub(p.Now)) + return s +} + +func TestSiteSimScenarios(t *testing.T) { + for _, sc := range simScenarios() { + t.Run(sc.name, func(t *testing.T) { + start := sc.start + if start.IsZero() { + start = simStart + } + s := runSimScenario(sc, start) + t.Logf("peak phase %.1f A, %v over %.0f A; car idle with room %v", s.peakA, s.overLimit, s.p.Fuse.RatingA-evPhaseMarginA, s.idleWithRoom) + switch { + case sc.known == "" && s.failure != "": + t.Fatal(s.failure) + case sc.known != "" && s.failure == "": + t.Fatalf("passes now: delete its known line (%s)", sc.known) + case sc.known != "": + t.Logf("fails as known (%s): %s", sc.known, s.failure) + } + }) + } +} + +// Core reads only the plant's clock: a run that crosses the wall clock does +// what the same run a day earlier or later does. Battery-full latching once +// read SoC times from the wall clock. +func TestSiteSimIgnoresWallClock(t *testing.T) { + sc := simScenario{run: 15 * time.Hour, plant: simDay} + // The battery is full from 12:45, so it is full when the plant passes now. + base := time.Now().Add(-13*time.Hour - 30*time.Minute).Truncate(time.Second) + want := runSimScenario(sc, base) + if want.failure != "" { + t.Fatal(want.failure) + } + for _, shift := range []time.Duration{-24 * time.Hour, 24 * time.Hour} { + got := runSimScenario(sc, base.Add(shift)) + if i := firstCommandDiff(want, got, shift); i >= 0 { + t.Fatalf("shifted %v: command %d is %v, want %v", shift, i, commandAt(got.commands, i), commandAt(want.commands, i)) + } + } +} + +func firstCommandDiff(a, b *simSite, shift time.Duration) int { + for i := range max(len(a.commands), len(b.commands)) { + x, y := commandAt(a.commands, i), commandAt(b.commands, i) + if x.device != y.device || x.w != y.w || !x.at.Add(shift).Equal(y.at) { + return i + } + } + return -1 +} + +func commandAt(log []simCommand, i int) simCommand { + if i < len(log) { + return log[i] + } + return simCommand{} +} diff --git a/go/internal/control/storage_plan.go b/go/internal/control/storage_plan.go index 12daa8996..eafcd8701 100644 --- a/go/internal/control/storage_plan.go +++ b/go/internal/control/storage_plan.go @@ -3,6 +3,7 @@ package control import ( "maps" "math" + "slices" "time" ) @@ -14,9 +15,12 @@ type storageSlotDelivery struct { // Keep each physical battery's share after Core has computed a live site // target. A blocked battery's energy is not reassigned to another battery. -// Manual control and missing plans retain the existing local controller. +// Interchangeable batteries share the live target by available energy +// instead. Manual control, a plan without per-battery energy, an idle slot, +// a single online battery and interchangeable batteries without a known SoC +// retain the existing local controller, which splits by capacity. func distributePlannedStorages(state *State, bats []batteryInfo, desired float64, manual bool) ([]DispatchTarget, bool) { - if manual || !state.Mode.IsPlannerMode() || state.PlanStale { + if manual || !state.Mode.IsPlannerMode() || state.PlanStale || len(bats) < 2 { state.storageDelivery = storageSlotDelivery{} return nil, false } @@ -34,6 +38,13 @@ func distributePlannedStorages(state *State, bats []batteryInfo, desired float64 state.storageDelivery = storageSlotDelivery{} return nil, false } + if interchangeableStorages(bats) { + state.storageDelivery = storageSlotDelivery{} + if slices.ContainsFunc(bats, func(b batteryInfo) bool { return b.socMissing }) { + return nil, false + } + return shareByAvailableEnergy(bats, desired), true + } d := &state.storageDelivery slotS := dir.SlotEnd.Sub(dir.SlotStart).Seconds() if slotS <= 0 || now.Before(dir.SlotStart) || !now.Before(dir.SlotEnd) { @@ -107,3 +118,89 @@ func distributePlannedStorages(state *State, bats []batteryInfo, desired float64 } return out, true } + +// Batteries with the same charge and discharge power per Wh of capacity are +// one pool to the planner. Core gives every storage the site's efficiencies, +// so each Wh costs the same on either battery, and a split by capacity can +// carry any total the plan can. Which battery the plan names is then +// arbitrary: Energyplan often puts a whole slot on one and the next on the +// other. A 5 % spread costs a capacity split at most about 1.3 % of the +// fleet's full power. +const interchangeablePowerSpread = 0.05 + +func interchangeableStorages(bats []batteryInfo) bool { + if len(bats) == 0 { + return false + } + lowC, highC := math.Inf(1), math.Inf(-1) + lowD, highD := math.Inf(1), math.Inf(-1) + for _, b := range bats { + if b.capacityWh <= 0 { + return false + } + c, d := b.chargeCap()/b.capacityWh, b.dischargeCap()/b.capacityWh + lowC, highC = math.Min(lowC, c), math.Max(highC, c) + lowD, highD = math.Min(lowD, d), math.Max(highD, d) + } + return highC-lowC <= interchangeablePowerSpread*highC && + highD-lowD <= interchangeablePowerSpread*highD +} + +// Split desired across interchangeable batteries by the energy each can still +// give (above the discharge floor) or take (below full). The fuller battery +// discharges faster and the emptier one charges faster, so a SoC gap shrinks +// while they work. A blocked battery gets nothing; a battery at its power +// limit stays there and the others take the rest. +func shareByAvailableEnergy(bats []batteryInfo, desired float64) []DispatchTarget { + out := make([]DispatchTarget, len(bats)) + room := make([]float64, len(bats)) + for i, b := range bats { + out[i].Driver = b.driver + if desired < 0 && !b.dischargeBlocked { + room[i] = b.capacityWh * math.Max(0, b.soc-dischargeFloorSoC) + } else if desired > 0 && !b.chargeBlocked { + room[i] = b.capacityWh * math.Max(0, 1-b.soc) + } + // Blocked, empty or full: this battery cannot take part. + out[i].Clamped = desired != 0 && room[i] <= 0 + } + left := math.Abs(desired) + for left > 0 { + var total, pinnedW float64 + for _, r := range room { + total += r + } + if total <= 0 { + break + } + pinned := false + for i, b := range bats { + limit := b.chargeCap() + if desired < 0 { + limit = b.dischargeCap() + } + if room[i] > 0 && left*room[i]/total > limit { + out[i].TargetW = math.Copysign(limit, desired) + out[i].Clamped = true + pinnedW += limit + room[i] = 0 + pinned = true + } + } + if !pinned { + for i := range bats { + if room[i] > 0 { + out[i].TargetW = math.Copysign(left*room[i]/total, desired) + } + } + break + } + left -= pinnedW + } + for i, b := range bats { + var clamped bool + out[i].TargetW, clamped = clampWithSoC(out[i].TargetW, b) + out[i].Clamped = out[i].Clamped || clamped + } + return out +} diff --git a/go/internal/control/storage_plan_test.go b/go/internal/control/storage_plan_test.go index a678483f3..a56e8592e 100644 --- a/go/internal/control/storage_plan_test.go +++ b/go/internal/control/storage_plan_test.go @@ -22,6 +22,8 @@ func TestPhysicalStorageAllocationSurvivesDispatch(t *testing.T) { }{{"a", 0, .5}, {"b", 0, .5}}) s := newStateWithEnergyDispatch(dir, "ferroamp") s.clock = func() time.Time { return now } + // b sits on a smaller inverter, so the plan's split is physical. + s.DriverLimits = map[string]PowerLimits{"b": {MaxChargeW: 1500, MaxDischargeW: 1500, MaxChargeWSet: true, MaxDischargeWSet: true}} out := ComputeDispatch(store, s, caps(map[string]float64{"a": 10000, "b": 10000}), 11040) want := map[string]float64{"a": 1600, "b": 400} if len(out) != 2 { @@ -75,7 +77,8 @@ func TestPhysicalStorageDeliveryAndSiteClamp(t *testing.T) { s := newStateWithEnergyDispatch(dir, "site") s.SlotDirective = func(time.Time) (SlotDirective, bool) { return dir, true } s.clock = func() time.Time { return now } - bats := []batteryInfo{{driver: "a", capacityWh: 10000, soc: .5}, {driver: "b", capacityWh: 10000, soc: .5}} + bats := []batteryInfo{{driver: "a", capacityWh: 10000, soc: .5}, + {driver: "b", capacityWh: 10000, soc: .5, maxChargeW: 1000, maxDischargeW: 1000, maxChargeWSet: true, maxDischargeWSet: true}} out, ok := distributePlannedStorages(s, bats, 1000, false) if !ok || out[0].TargetW != 750 || out[1].TargetW != 250 { t.Fatalf("site clamp=%+v", out) @@ -106,3 +109,266 @@ func TestPhysicalStorageDeliveryAndSiteClamp(t *testing.T) { t.Fatal("stale plan allocated storage") } } + +// Field report, 23 Oct 2026: two identical Pixii batteries took turns. The +// planner is indifferent between them, so Energyplan put a whole slot on one +// battery and the next slot on the other; Core followed each battery's +// share and the SoCs drifted apart (74 % against 60 %). Now both charge in +// every slot, the emptier one faster (room 26 % against 40 %). +func TestInterchangeableStoragesShareThePlanTotal(t *testing.T) { + start := time.Now() + now := start + dir := SlotDirective{DecisionID: "p", SlotStart: start, SlotEnd: start.Add(15 * time.Minute), BatteryEnergyWh: 500, + StorageEnergyWh: map[string]float64{"a": 500, "b": 0}} + store := seedStore(0, []struct { + name string + currentW, soc float64 + }{{"a", 0, .74}, {"b", 0, .60}}) + s := newStateWithEnergyDispatch(dir, "ferroamp") + s.SlotDirective = func(time.Time) (SlotDirective, bool) { return dir, true } + s.clock = func() time.Time { return now } + for slot, plan := range []map[string]float64{{"a": 500, "b": 0}, {"a": 0, "b": 500}} { + dir.SlotStart, dir.SlotEnd = now, now.Add(15*time.Minute) + dir.StorageEnergyWh = plan + out := ComputeDispatch(store, s, caps(map[string]float64{"a": 10000, "b": 10000}), 11040) + if len(out) != 2 { + t.Fatalf("slot %d targets=%+v", slot, out) + } + want := map[string]float64{"a": 2000 * 26. / 66, "b": 2000 * 40. / 66} + for _, v := range out { + if math.Abs(v.TargetW-want[v.Driver]) > 1e-6 { + t.Fatalf("slot %d: targets=%+v, want %v", slot, out, want) + } + } + now = now.Add(15 * time.Minute) + } +} + +// A battery that cannot charge hands its share to its twin instead of +// sending it to the grid. The plan named the blocked battery. +func TestInterchangeableStoragesHandOverABlockedShare(t *testing.T) { + now := time.Now() + dir := SlotDirective{DecisionID: "p", SlotStart: now, SlotEnd: now.Add(15 * time.Minute), BatteryEnergyWh: 500, + StorageEnergyWh: map[string]float64{"a": 500, "b": 0}} + store := seedStore(0, []struct { + name string + currentW, soc float64 + }{{"a", 0, .5}, {"b", 0, .5}}) + soc := .5 + store.Update("a", telemetry.DerBattery, 0, &soc, json.RawMessage(`{"charge_capable":false}`)) + s := newStateWithEnergyDispatch(dir, "ferroamp") + s.clock = func() time.Time { return now } + out := ComputeDispatch(store, s, caps(map[string]float64{"a": 10000, "b": 10000}), 11040) + want := map[string]float64{"a": 0, "b": 2000} + if len(out) != 2 { + t.Fatalf("targets=%+v", out) + } + for _, v := range out { + if math.Abs(v.TargetW-want[v.Driver]) > 1e-6 { + t.Fatalf("targets=%+v, want %v", out, want) + } + } +} + +// The same two batteries on a cover-load slot: the plan put −564 W on one, +// the house drew 3 kW. Both batteries discharge, more than the planned +// 564 W in total, instead of one battery at its planned rate with the rest +// from the grid. The fuller one gives more: 69 % above the floor against 55 %. +func TestInterchangeableStoragesShareALargeLoad(t *testing.T) { + now := time.Now() + dir := SlotDirective{DecisionID: "p", SlotStart: now, SlotEnd: now.Add(15 * time.Minute), BatteryEnergyWh: -141, + StorageEnergyWh: map[string]float64{"a": -141, "b": 0}, PlannedGridW: 0, HasPlannedGridW: true} + store := seedStore(3000, []struct { + name string + currentW, soc float64 + }{{"a", 0, .74}, {"b", 0, .60}}) + s := newStateWithEnergyDispatch(dir, "ferroamp") + s.clock = func() time.Time { return now } + out := ComputeDispatch(store, s, caps(map[string]float64{"a": 10000, "b": 10000}), 11040) + if len(out) != 2 { + t.Fatalf("targets=%+v", out) + } + w := map[string]float64{} + for _, v := range out { + w[v.Driver] = v.TargetW + } + if w["b"] >= 0 || math.Abs(w["a"]/w["b"]-69./55) > 1e-6 || w["a"]+w["b"] > -564-1e-6 { + t.Fatalf("both batteries should share more than the planned 564 W, got %+v", out) + } +} + +// Two equal batteries follow their targets for five hours of discharge, and +// five of charge. The SoC gap shrinks on every tick, neither battery passes +// the floor or full, and both end there together. +func TestInterchangeableStoragesConverge(t *testing.T) { + for _, tc := range []struct { + name string + socA, socB, end float64 + desiredW float64 + }{ + {"discharge from 74 % and 60 %", .74, .60, dischargeFloorSoC, -3000}, + {"charge from 30 % and 50 %", .30, .50, 1, 3000}, + } { + now := time.Now() + s := NewState(0, 0, "site") + s.Mode = ModePlannerArbitrage + s.clock = func() time.Time { return now } + // The plan puts every slot on a; Core still shares it. + wh := tc.desiredW / 4 + s.SlotDirective = func(at time.Time) (SlotDirective, bool) { + return SlotDirective{DecisionID: "p", SlotStart: at, SlotEnd: at.Add(15 * time.Minute), BatteryEnergyWh: wh, + StorageEnergyWh: map[string]float64{"a": wh, "b": 0}}, true + } + bats := []batteryInfo{{driver: "a", capacityWh: 10000, soc: tc.socA}, {driver: "b", capacityWh: 10000, soc: tc.socB}} + gap := math.Abs(tc.socA - tc.socB) + for tick := 0; tick < 3600; tick++ { + out, ok := distributePlannedStorages(s, bats, tc.desiredW, false) + if !ok || len(out) != 2 { + t.Fatalf("%s tick %d: targets=%+v", tc.name, tick, out) + } + for i := range bats { + bats[i].soc += out[i].TargetW * 5 / 3600 / bats[i].capacityWh + bats[i].currentW = out[i].TargetW + } + now = now.Add(5 * time.Second) + next := math.Abs(bats[0].soc - bats[1].soc) + if next > gap+1e-12 || (next >= gap && next > 1e-9) { + t.Fatalf("%s tick %d: gap %.4f did not shrink from %.4f", tc.name, tick, next, gap) + } + gap = next + // One 5 s tick at 3 kW moves a 10 kWh battery 0.04 %. + const tickSoC = 3000. * 5 / 3600 / 10000 + for _, b := range bats { + if b.soc < dischargeFloorSoC-tickSoC || b.soc > 1+tickSoC { + t.Fatalf("%s tick %d: %s at %.4f", tc.name, tick, b.driver, b.soc) + } + } + } + if math.Abs(bats[0].soc-tc.end) > .002 || math.Abs(bats[1].soc-tc.end) > .002 { + t.Fatalf("%s: ended at %.4f and %.4f", tc.name, bats[0].soc, bats[1].soc) + } + } +} + +func TestShareByAvailableEnergyStaysWithinLimits(t *testing.T) { + bats := []batteryInfo{{driver: "a", capacityWh: 10000, soc: .80}, {driver: "b", capacityWh: 10000, soc: .10}} + // By energy a would give 5,625 W of 6 kW; it stops at 5 kW and b gives the rest. + out := shareByAvailableEnergy(bats, -6000) + if out[0].TargetW != -5000 || math.Abs(out[1].TargetW+1000) > 1e-6 { + t.Fatalf("targets=%+v", out) + } + // More than the fleet can give: both at their limits. + out = shareByAvailableEnergy(bats, -12000) + if out[0].TargetW != -5000 || out[1].TargetW != -5000 { + t.Fatalf("targets=%+v", out) + } + // A battery at the floor gives nothing. + bats[1].soc = dischargeFloorSoC + out = shareByAvailableEnergy(bats, -3000) + if out[0].TargetW != -3000 || out[1].TargetW != 0 { + t.Fatalf("targets=%+v", out) + } + // Full batteries take nothing. + bats[0].soc, bats[1].soc = 1, 1 + out = shareByAvailableEnergy(bats, 3000) + if out[0].TargetW != 0 || out[1].TargetW != 0 { + t.Fatalf("targets=%+v", out) + } +} + +func TestShareByAvailableEnergyCharge(t *testing.T) { + // Three batteries charging. c is blocked, a nearly full; b would take + // 6.4 kW of 8 kW by energy, stops at 5 kW, and a takes the rest. + bats := []batteryInfo{ + {driver: "a", capacityWh: 10000, soc: .80}, + {driver: "b", capacityWh: 10000, soc: .20}, + {driver: "c", capacityWh: 10000, soc: .10, chargeBlocked: true}, + } + out := shareByAvailableEnergy(bats, 8000) + want := []DispatchTarget{{Driver: "a", TargetW: 3000}, {Driver: "b", TargetW: 5000, Clamped: true}, {Driver: "c", Clamped: true}} + for i := range want { + if math.Abs(out[i].TargetW-want[i].TargetW) > 1e-6 || out[i].Clamped != want[i].Clamped { + t.Fatalf("targets=%+v, want %+v", out, want) + } + } + // Nothing limits a small charge, so nothing reads as clamped. + out = shareByAvailableEnergy(bats, 1000) + if math.Abs(out[0].TargetW-200) > 1e-6 || math.Abs(out[1].TargetW-800) > 1e-6 || out[0].Clamped || out[1].Clamped || !out[2].Clamped { + t.Fatalf("targets=%+v", out) + } +} + +// An idle slot keeps the local controller, with its inverter-group routing. +// So do batteries whose SoC Core does not know: room cannot be weighed. +func TestInterchangeableStoragesKeepTheLocalControllerWhenIdleOrBlind(t *testing.T) { + now := time.Now() + dir := SlotDirective{DecisionID: "p", SlotStart: now, SlotEnd: now.Add(15 * time.Minute), StorageEnergyWh: map[string]float64{"a": 0, "b": 0}} + s := newStateWithEnergyDispatch(dir, "site") + s.Mode = ModePlannerSelf + s.clock = func() time.Time { return now } + bats := []batteryInfo{{driver: "a", capacityWh: 10000, soc: .5}, {driver: "b", capacityWh: 10000, soc: .5}} + if _, ok := distributePlannedStorages(s, bats, 3000, false); ok { + t.Fatal("idle slot left the local controller") + } + dir.StorageEnergyWh = map[string]float64{"a": 500, "b": 0} + s.SlotDirective = func(time.Time) (SlotDirective, bool) { return dir, true } + bats[1].socMissing, bats[1].soc, bats[1].dischargeBlocked = true, 0, true + if _, ok := distributePlannedStorages(s, bats, 2000, false); ok { + t.Fatal("a battery without SoC was weighed as empty") + } +} + +// A battery on a hybrid inverter with a small limit is physically different: +// splitting 6 kW by capacity would ask it for 3 kW and lose 1.5 kW. The plan +// already put the energy where each inverter can deliver it, so Core keeps +// following each battery's planned energy. +func TestPhysicallyDifferentStoragesFollowThePlan(t *testing.T) { + now := time.Now() + dir := SlotDirective{DecisionID: "p", SlotStart: now, SlotEnd: now.Add(15 * time.Minute), BatteryEnergyWh: 1500, + StorageEnergyWh: map[string]float64{"a": 1125, "b": 375}} + store := seedStore(500, []struct { + name string + currentW, soc float64 + }{{"a", 0, .5}, {"b", 0, .5}}) + s := newStateWithEnergyDispatch(dir, "ferroamp") + s.clock = func() time.Time { return now } + s.DriverLimits = map[string]PowerLimits{ + "a": {MaxChargeW: 5000, MaxDischargeW: 5000, MaxChargeWSet: true, MaxDischargeWSet: true}, + "b": {MaxChargeW: 1500, MaxDischargeW: 1500, MaxChargeWSet: true, MaxDischargeWSet: true}, + } + out := ComputeDispatch(store, s, caps(map[string]float64{"a": 10000, "b": 10000}), 11040) + want := map[string]float64{"a": 4500, "b": 1500} + if len(out) != 2 { + t.Fatalf("targets=%+v", out) + } + for _, v := range out { + if math.Abs(v.TargetW-want[v.Driver]) > 1e-6 { + t.Fatalf("targets=%+v, want %v", out, want) + } + } +} + +func TestInterchangeableStoragesRule(t *testing.T) { + limit := func(charge, discharge float64) batteryInfo { + return batteryInfo{capacityWh: 10000, maxChargeW: charge, maxDischargeW: discharge, maxChargeWSet: true, maxDischargeWSet: true} + } + for _, tc := range []struct { + name string + bats []batteryInfo + want bool + }{ + {"identical defaults", []batteryInfo{{capacityWh: 10000}, {capacityWh: 10000}}, true}, + {"twice the capacity and twice the power", []batteryInfo{limit(3000, 3000), {capacityWh: 20000, maxChargeW: 6000, maxDischargeW: 6000, maxChargeWSet: true, maxDischargeWSet: true}}, true}, + {"4 % apart", []batteryInfo{limit(5000, 5000), limit(4800, 4800), limit(5000, 5000)}, true}, + {"6 % apart", []batteryInfo{limit(5000, 5000), limit(4700, 5000)}, false}, + {"spread across three", []batteryInfo{limit(4850, 5000), limit(5000, 5000), limit(5150, 5000)}, false}, + {"smaller inverter", []batteryInfo{limit(5000, 5000), limit(1500, 1500)}, false}, + {"same power, smaller pack", []batteryInfo{{capacityWh: 1000}, {capacityWh: 19000}}, false}, + {"charge disabled on one", []batteryInfo{limit(5000, 5000), limit(0, 5000)}, false}, + {"unknown capacity", []batteryInfo{{capacityWh: 10000}, {}}, false}, + } { + if got := interchangeableStorages(tc.bats); got != tc.want { + t.Errorf("%s: interchangeable=%v, want %v", tc.name, got, tc.want) + } + } +} diff --git a/go/internal/control/testdata/golden/seeded_siblings.json b/go/internal/control/testdata/golden/seeded_siblings.json index f14522be2..e6339aa3a 100644 --- a/go/internal/control/testdata/golden/seeded_siblings.json +++ b/go/internal/control/testdata/golden/seeded_siblings.json @@ -1,7 +1,7 @@ { "family": "seeded_siblings", "generated_by": "FTW_GOLDEN_DUMP=1 go test ./internal/control/ -run TestGoldenDump", - "ftw_commit": "68c5b6125898bb9da70728b8921259c87e96171d", + "ftw_commit": "b3af71859a912dcae3a512f6c1ca79a21c2bb631", "record_count": 60, "records": [ { @@ -396,12 +396,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 0, + "target_w": 1303.9, "clamped": false }, { "driver": "sungrow", - "target_w": 0, + "target_w": 1303.9, "clamped": false } ], @@ -424,9 +424,9 @@ }, "site_totals": { "battery_current_w": -13, - "battery_target_sum_w": 0, + "battery_target_sum_w": 2607.8, "raw_grid_w": -4368, - "projected_grid_w": -4355 + "projected_grid_w": -1747.1999999999998 } }, { @@ -1060,17 +1060,19 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 2568.0000044676126, + "target_w": 3716.1290322580644, "clamped": false }, { "driver": "sungrow", - "target_w": 4066.0000070737206, - "clamped": false + "target_w": 5000, + "clamped": true } ], "clamp_attributions": { - "clamped_drivers": [], + "clamped_drivers": [ + "sungrow" + ], "all_targets_zero_clamped": false, "plan_sign_intent": 1, "plan_sign_floor_zeroed": false, @@ -1088,9 +1090,9 @@ }, "site_totals": { "battery_current_w": 850, - "battery_target_sum_w": 6634.000011541333, + "battery_target_sum_w": 8716.129032258064, "raw_grid_w": -5208, - "projected_grid_w": 576.0000115413332 + "projected_grid_w": 2658.1290322580644 } }, { @@ -1714,12 +1716,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 402.6, + "target_w": 1459.9333333333334, "clamped": false }, { "driver": "sungrow", - "target_w": 805.2, + "target_w": 2919.866666666667, "clamped": false } ], @@ -1742,9 +1744,9 @@ }, "site_totals": { "battery_current_w": 1639, - "battery_target_sum_w": 1207.8000000000002, + "battery_target_sum_w": 4379.8, "raw_grid_w": -4568, - "projected_grid_w": -4999.2 + "projected_grid_w": -1827.1999999999998 } }, { @@ -2162,7 +2164,7 @@ }, { "driver": "sungrow", - "target_w": 3287.6712395762556, + "target_w": 3287.671232876712, "clamped": false } ], @@ -2188,9 +2190,9 @@ }, "site_totals": { "battery_current_w": -479, - "battery_target_sum_w": 8287.671239576255, + "battery_target_sum_w": 8287.671232876712, "raw_grid_w": -5443, - "projected_grid_w": 3323.671239576255 + "projected_grid_w": 3323.6712328767117 } }, { @@ -2365,12 +2367,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 1122.2709677419355, + "target_w": 2943.5612903225806, "clamped": false }, { "driver": "sungrow", - "target_w": 1776.9290322580644, + "target_w": 4660.63870967742, "clamped": false } ], @@ -2393,9 +2395,9 @@ }, "site_totals": { "battery_current_w": 4063, - "battery_target_sum_w": 2899.2, + "battery_target_sum_w": 7604.200000000001, "raw_grid_w": -5902, - "projected_grid_w": -7065.8 + "projected_grid_w": -2360.7999999999993 } }, { @@ -3016,12 +3018,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 0, + "target_w": 196.83168316831683, "clamped": false }, { "driver": "sungrow", - "target_w": 0, + "target_w": 598.3683168316832, "clamped": false } ], @@ -3044,9 +3046,9 @@ }, "site_totals": { "battery_current_w": -1063, - "battery_target_sum_w": 0, + "battery_target_sum_w": 795.2, "raw_grid_w": -3097, - "projected_grid_w": -2034 + "projected_grid_w": -1238.8 } }, { @@ -3237,12 +3239,12 @@ }, { "driver": "pixii", - "target_w": 3243.243246846847, + "target_w": 3243.2432432432433, "clamped": false }, { "driver": "sungrow", - "target_w": 3113.5135169729733, + "target_w": 3113.5135135135138, "clamped": false } ], @@ -3267,9 +3269,9 @@ }, "site_totals": { "battery_current_w": -4513, - "battery_target_sum_w": 6656.75676381982, + "battery_target_sum_w": 6656.756756756757, "raw_grid_w": -5457, - "projected_grid_w": 5712.75676381982 + "projected_grid_w": 5712.756756756757 } }, { @@ -3664,12 +3666,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 0, + "target_w": 729.5049504950495, "clamped": false }, { "driver": "sungrow", - "target_w": 0, + "target_w": 2217.6950495049505, "clamped": false } ], @@ -3692,9 +3694,9 @@ }, "site_totals": { "battery_current_w": 627, - "battery_target_sum_w": 0, + "battery_target_sum_w": 2947.2, "raw_grid_w": -3867, - "projected_grid_w": -4494 + "projected_grid_w": -1546.8000000000002 } }, { @@ -3772,7 +3774,7 @@ }, { "driver": "sungrow", - "target_w": -3200.0000029653334, + "target_w": -3200, "clamped": false } ], @@ -3797,9 +3799,9 @@ }, "site_totals": { "battery_current_w": -1969, - "battery_target_sum_w": -3200.0000029653334, + "battery_target_sum_w": -3200, "raw_grid_w": 5401, - "projected_grid_w": 4169.999997034667 + "projected_grid_w": 4170 } }, { @@ -4324,12 +4326,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 3142.0000053333333, + "target_w": 4800, "clamped": false }, { "driver": "sungrow", - "target_w": 3142.0000053333333, + "target_w": 4800, "clamped": false } ], @@ -4352,9 +4354,9 @@ }, "site_totals": { "battery_current_w": -3448, - "battery_target_sum_w": 6284.000010666667, + "battery_target_sum_w": 9600, "raw_grid_w": -5478, - "projected_grid_w": 4254.000010666667 + "projected_grid_w": 7570 } }, { @@ -4976,12 +4978,12 @@ "per_driver_targets": [ { "driver": "pixii", - "target_w": 1670.3000000000002, + "target_w": 3996.8, "clamped": false }, { "driver": "sungrow", - "target_w": 1670.3000000000002, + "target_w": 3996.8, "clamped": false } ], @@ -5004,9 +5006,9 @@ }, "site_totals": { "battery_current_w": 5215, - "battery_target_sum_w": 3340.6000000000004, + "battery_target_sum_w": 7993.6, "raw_grid_w": -4631, - "projected_grid_w": -6505.4 + "projected_grid_w": -1852.3999999999996 } }, { @@ -5419,12 +5421,12 @@ }, { "driver": "pixii", - "target_w": 4800.000008448, + "target_w": 4800, "clamped": false }, { "driver": "sungrow", - "target_w": 4800.000008448, + "target_w": 4800, "clamped": false } ], @@ -5449,9 +5451,9 @@ }, "site_totals": { "battery_current_w": -1477, - "battery_target_sum_w": 9600.000016896, + "battery_target_sum_w": 9600, "raw_grid_w": -3251, - "projected_grid_w": 7826.000016896 + "projected_grid_w": 7826 } }, { @@ -6501,12 +6503,12 @@ }, { "driver": "pixii", - "target_w": 1920.0000032, + "target_w": 1920, "clamped": false }, { "driver": "sungrow", - "target_w": 3840.0000064, + "target_w": 3840, "clamped": false } ], @@ -6531,9 +6533,9 @@ }, "site_totals": { "battery_current_w": -3067, - "battery_target_sum_w": 6060.000009599999, + "battery_target_sum_w": 6060, "raw_grid_w": -3266, - "projected_grid_w": 5861.000009599999 + "projected_grid_w": 5861 } } ] diff --git a/go/internal/control/testdata/sitesim_field_known.txt b/go/internal/control/testdata/sitesim_field_known.txt new file mode 100644 index 000000000..71f1a29a3 --- /dev/null +++ b/go/internal/control/testdata/sitesim_field_known.txt @@ -0,0 +1,44 @@ +# Field matrix rows that fail on master: case, invariant and the fix that +# turns the row green. TestSiteSimFieldMatrix fails when a listed row passes +# and when a row fails that is not listed. FTW_FIELD_KNOWN=write rewrites +# this file from a run, keeps each label and marks unlisted failures "new". +# +# B srcfl/ftw#1799: planned charging and Charge now run on three +# phases; the planner gets steps a three-phase order can deliver. +# #1795 srcfl/ftw#1795: a charger drawing against its order reads not +# following. +# export srcfl/ftw#1782, draft: an exporting phase counts as import, so +# surplus never finds room to start. +# readback no fix yet: surplus raises again before the charger applied the +# last raise, then pauses when both land (surplus_reserve.go:128). +# relief srcfl/ftw#1789, draft: the battery ends its relief before the +# car's cut shows. +pr1763/25A/3p/3p-15A/cloud60s/car3p/6-2-2/chargenow+battery own-raise relief +pr1763/25A/3p/3p-15A/cloud60s/car3p/6-2-2/pause+battery own-raise relief +pr1763/25A/3p/3p-15A/easee120s/car3p/6-2-2/chargenow+battery own-raise relief +pr1763/25A/3p/3p-15A/easee40s/car3p/6-2-2/chargenow+battery own-raise relief +pr1763/25A/3p/3p-15A/easee40s/car3p/6-2-2/pause+battery own-raise relief +pr1763/25A/3p/3p-15A/fast2s/car3p/6-2-2/pause+battery own-raise relief +probe/25A/auto/mixed/easee40s/car3p/balanced/surplus+goal+export2500W idle readback +timing/16A/3p/3p/cloud60s/car3p/L1/surplus idle export +timing/16A/3p/3p/easee120s/car3p/L1/surplus idle export +timing/16A/3p/3p/easee40s/car3p/L1/surplus idle export +timing/16A/3p/3p/fast2s/car3p/L1/surplus idle export +timing/16A/auto/mixed/cloud60s/car3p/L1/surplus idle export +timing/16A/auto/mixed/easee120s/car3p/L1/surplus idle export +timing/16A/auto/mixed/easee40s/car3p/L1/surplus idle export +timing/16A/auto/mixed/fast2s/car3p/L1/surplus idle export +timing/16A/empty/none/cloud60s/car3p/L1/surplus idle export +timing/16A/empty/none/easee120s/car3p/L1/surplus idle export +timing/16A/empty/none/easee40s/car3p/L1/surplus idle export +timing/16A/empty/none/fast2s/car3p/L1/surplus idle export +timing/25A/3p/3p/cloud60s/car3p/L1/surplus idle readback +timing/25A/3p/3p/easee120s/car3p/L1/surplus idle readback +timing/25A/3p/3p/easee40s/car3p/L1/surplus idle readback +timing/25A/auto/mixed/cloud60s/car3p/L1/surplus idle readback +timing/25A/auto/mixed/easee120s/car3p/L1/surplus idle readback +timing/25A/auto/mixed/easee40s/car3p/L1/surplus idle readback +timing/25A/empty/none/cloud60s/car3p/L1/surplus idle readback +timing/25A/empty/none/easee120s/car3p/L1/surplus idle readback +timing/25A/empty/none/easee40s/car3p/L1/surplus idle readback +timing/25A/empty/none/fast2s/car3p/L1/surplus idle export diff --git a/go/internal/drivers/registry.go b/go/internal/drivers/registry.go index 07ea3bca4..0d5b11e92 100644 --- a/go/internal/drivers/registry.go +++ b/go/internal/drivers/registry.go @@ -1110,6 +1110,9 @@ func (r *Registry) runLoop(rd *runningDriver) { pollCtx, finishPoll := rd.beginCommand(ctx) returned, err := rd.driver.Poll(pollCtx) finishPoll() + if r.tel != nil { + r.tel.CountPoll(err != nil) + } if err != nil { slog.Warn("driver poll failed", "name", rd.cfg.Name, "err", err) if r.tel != nil { diff --git a/go/internal/drivers/registry_restart_test.go b/go/internal/drivers/registry_restart_test.go index 5f0463fb1..03baa1373 100644 --- a/go/internal/drivers/registry_restart_test.go +++ b/go/internal/drivers/registry_restart_test.go @@ -1332,3 +1332,29 @@ function driver_command(action, w, cmd) end t.Errorf("LastSuccess advanced without any host.emit call: %v", h.LastSuccess) } } + +// The fleet report's share of failed polls counts the poll loop's polls, +// failed or not, and nothing else. +func TestRunLoopCountsPollsAndFailedPolls(t *testing.T) { + r := NewRegistry(telemetry.NewStore()) + src := ` +local n = 0 +function driver_init(config) host.set_poll_interval(50) end +function driver_poll() + n = n + 1 + if n % 2 == 0 then error("meter did not answer") end + return 50 +end +function driver_command(action, w, cmd) end +` + path := writeTestDriver(t, src) + if err := r.Add(context.Background(), config.Driver{Name: "d1", Lua: path}); err != nil { + t.Fatal(err) + } + defer r.Remove("d1") + time.Sleep(400 * time.Millisecond) + polls, errors, _ := r.tel.PollsLastDay(time.Now()) + if polls < 4 || errors < 2 || errors > polls/2+1 { + t.Fatalf("polls %d, failed %d after 400ms of 50ms polls, every second one failing", polls, errors) + } +} diff --git a/go/internal/energyreport/day.go b/go/internal/energyreport/day.go new file mode 100644 index 000000000..29b48c5c0 --- /dev/null +++ b/go/internal/energyreport/day.go @@ -0,0 +1,117 @@ +package energyreport + +import ( + "context" + "math" + "time" + + "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/protocol/energywire" +) + +// Ledger is the part of the store a day is read from. +type Ledger interface { + EnergyAssets() ([]state.EnergyAsset, error) + LoadEnergyHistoryContext(context.Context, state.EnergyHistoryQuery) ([]state.EnergyLedgerPoint, bool, error) +} + +// usable are the ledger qualities that count as a recorded bucket. A reset +// is one: a charger's session counter starts again at every plug-in, and the +// ledger covers that interval from the power reading. A gap, an invalid read +// or anything this list does not name makes the day incomplete. +var usable = map[string]bool{"measured": true, "recovered": true, "integrated": true, "estimated": true, "reset": true} + +// sums maps a ledger flow to its place in energywire.Fields. +var sums = map[state.EnergyFlow]int{ + state.FlowPVGeneration: 0, + state.FlowGridImport: 1, + state.FlowGridExport: 2, + state.FlowBatteryCharge: 3, + state.FlowBatteryDischarge: 4, + state.FlowVehicleCharge: 5, +} + +const dayMS = int64(24 * time.Hour / time.Millisecond) + +// Day reads the closed UTC day that starts at start. ok is false unless the +// whole day is there. +// +// Whole means: a site meter, and every grid meter, solar, battery and charger +// asset that was in use that day recorded every bucket of it, with nothing the +// ledger marks as a gap or invalid read. An asset that arrived or went +// silent during the day makes it incomplete; one that left before the day +// began is not part of it. A box that misses part of a day sends nothing for +// it rather than a sum that looks like a quiet day. +func Day(ctx context.Context, l Ledger, start time.Time) (energywire.Report, bool, error) { + start = start.UTC().Truncate(24 * time.Hour) + from := start.UnixMilli() + to := from + dayMS + assets, err := l.EnergyAssets() + if err != nil { + return energywire.Report{}, false, err + } + var wh [6]float64 + meter := false + for _, a := range assets { + switch a.Kind { + case state.AssetGridMeter, state.AssetPV, state.AssetBattery, state.AssetVehicleCharger: + default: + continue + } + // Gone before the day began, or first seen after it ended. + if a.LastSeenMS < from || a.FirstSeenMS >= to { + continue + } + if a.FirstSeenMS > from || a.LastSeenMS < to { + return energywire.Report{}, false, nil + } + points, truncated, err := l.LoadEnergyHistoryContext(ctx, state.EnergyHistoryQuery{ + AssetID: a.AssetID, SinceMS: from, UntilMS: to, + BucketMS: state.EnergyLedgerBucketMS, Limit: int(dayMS / state.EnergyLedgerBucketMS), + }) + if err != nil { + return energywire.Report{}, false, err + } + if truncated || !covers(points, from, to) { + return energywire.Report{}, false, nil + } + for _, p := range points { + if i, ok := sums[p.Flow]; ok { + wh[i] += p.EnergyWh + } + } + meter = meter || a.Kind == state.AssetGridMeter + } + if !meter { + return energywire.Report{}, false, nil + } + r := energywire.Report{Schema: energywire.Schema, Date: start.Format(time.DateOnly)} + fields := []*int64{&r.PVGeneration, &r.GridImport, &r.GridExport, &r.BatteryCharge, &r.BatteryDischarge, &r.VehicleCharge} + for i, v := range wh { + if math.IsNaN(v) || math.IsInf(v, 0) || v < 0 { + return energywire.Report{}, false, nil + } + *fields[i] = int64(math.Round(v / 1000)) + } + return r, r.Valid(), nil +} + +// covers reports whether the points fill [from, to) with usable buckets of +// one width and hold nothing else. +func covers(points []state.EnergyLedgerPoint, from, to int64) bool { + if len(points) == 0 { + return false + } + width := points[0].BucketLenMS + if width <= 0 || (to-from)%width != 0 { + return false + } + seen := map[int64]bool{} + for _, p := range points { + if p.BucketLenMS != width || p.BucketStartMS < from || p.BucketStartMS >= to || (p.BucketStartMS-from)%width != 0 || !usable[p.Quality] { + return false + } + seen[p.BucketStartMS] = true + } + return int64(len(seen)) == (to-from)/width +} diff --git a/go/internal/energyreport/day_test.go b/go/internal/energyreport/day_test.go new file mode 100644 index 000000000..6cd088e80 --- /dev/null +++ b/go/internal/energyreport/day_test.go @@ -0,0 +1,216 @@ +package energyreport + +import ( + "context" + "os" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/state" +) + +var day = time.Date(2026, 10, 23, 0, 0, 0, 0, time.UTC) + +const tick = 2 * time.Minute + +type asset struct { + kind state.EnergyAssetKind + flow state.EnergyFlow + watts float64 + counter bool +} + +// The test site: a metered grid, solar, a battery and a charger, read every +// two minutes, the grid meter by its counters and the rest by power. +var site = map[string][]asset{ + "meter:1/grid_meter": {{state.AssetGridMeter, state.FlowGridImport, 1000, true}, {state.AssetGridMeter, state.FlowGridExport, 0, true}}, + "inverter:1/pv": {{state.AssetPV, state.FlowPVGeneration, 500, false}}, + "battery:1/battery": {{state.AssetBattery, state.FlowBatteryCharge, 250, false}, {state.AssetBattery, state.FlowBatteryDischarge, 0, false}}, + "charger:1/vehicle_charger": {{state.AssetVehicleCharger, state.FlowVehicleCharge, 1000, false}}, + "site/observed-consumer": {{state.AssetObservedConsumer, state.FlowConsumerUse, 700, false}}, +} + +func openStore(t *testing.T) *state.Store { + t.Helper() + st, err := state.Open(filepath.Join(t.TempDir(), "state.db")) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + return st +} + +// record writes the site from `from` to `to`, in time order and in one +// transaction, leaving out any asset whose id skip returns true for at that +// time. A counter keeps counting while it is not read, as a meter's does. +func record(t *testing.T, st *state.Store, from, to time.Time, skip func(id string, at time.Time) bool) { + t.Helper() + var obs []state.EnergyObservation + for at := from; !at.After(to); at = at.Add(tick) { + for id, flows := range site { + if skip != nil && skip(id, at) { + continue + } + for _, a := range flows { + w, c := a.watts, a.watts*at.Sub(from).Hours() + o := state.EnergyObservation{AssetID: id, DeviceID: id, AssetKind: a.kind, Label: "x", Flow: a.flow, AtMs: at.UnixMilli(), PowerW: &w} + if a.counter { + o.CounterWh, o.PowerW = &c, nil + } + obs = append(obs, o) + } + } + } + if err := st.RecordTickWithOptionalHistory(nil, nil, obs); err != nil { + t.Fatal(err) + } +} + +// sharedDay is the whole test day, written once and only read. +var sharedDay struct { + once sync.Once + dir string + st *state.Store + err error +} + +func TestMain(m *testing.M) { + code := m.Run() + if sharedDay.st != nil { + _ = sharedDay.st.Close() + } + if sharedDay.dir != "" { + _ = os.RemoveAll(sharedDay.dir) + } + os.Exit(code) +} + +func whole(t *testing.T) *state.Store { + t.Helper() + sharedDay.once.Do(func() { + sharedDay.dir, sharedDay.err = os.MkdirTemp("", "energyreport") + if sharedDay.err != nil { + return + } + sharedDay.st, sharedDay.err = state.Open(filepath.Join(sharedDay.dir, "state.db")) + if sharedDay.err == nil { + record(t, sharedDay.st, day.Add(-10*time.Minute), day.Add(24*time.Hour+10*time.Minute), nil) + } + }) + if sharedDay.err != nil { + t.Fatal(sharedDay.err) + } + return sharedDay.st +} + +func wholeDay(t *testing.T, skip func(string, time.Time) bool) *state.Store { + st := openStore(t) + record(t, st, day.Add(-10*time.Minute), day.Add(24*time.Hour+10*time.Minute), skip) + return st +} + +func TestAWholeDayIsSixWholeKWhSums(t *testing.T) { + st := whole(t) + r, ok, err := Day(context.Background(), st, day.Add(5*time.Hour)) + if err != nil || !ok { + t.Fatalf("ok %v err %v", ok, err) + } + if r.Date != "2026-10-23" || r.GridImport != 24 || r.GridExport != 0 || r.PVGeneration != 12 || + r.BatteryCharge != 6 || r.BatteryDischarge != 0 || r.VehicleCharge != 24 || !r.Valid() { + t.Fatalf("%+v", r) + } +} + +func TestAPartDayIsNotSent(t *testing.T) { + for name, skip := range map[string]func(string, time.Time) bool{ + "solar silent for an hour": func(id string, at time.Time) bool { + return id == "inverter:1/pv" && at.After(day.Add(13*time.Hour)) && at.Before(day.Add(14*time.Hour)) + }, + "a battery that stopped at noon": func(id string, at time.Time) bool { + return id == "battery:1/battery" && at.After(day.Add(12*time.Hour)) + }, + "a charger added at noon": func(id string, at time.Time) bool { + return id == "charger:1/vehicle_charger" && at.Before(day.Add(12*time.Hour)) + }, + "no site meter": func(id string, _ time.Time) bool { return id == "meter:1/grid_meter" }, + } { + t.Run(name, func(t *testing.T) { + if r, ok, err := Day(context.Background(), wholeDay(t, skip), day); err != nil || ok { + t.Fatalf("sent %+v (err %v)", r, err) + } + }) + } + // The day after the ledger ends has not happened yet. + if _, ok, _ := Day(context.Background(), whole(t), day.AddDate(0, 0, 1)); ok { + t.Fatal("an unrecorded day was sent") + } +} + +// A meter read by its counters keeps a whole day across an outage: the +// counter carries the three hours' energy over the gap, and the ledger +// spreads it. +func TestACounterCarriesTheMeterAcrossAnOutage(t *testing.T) { + only := map[string]bool{"meter:1/grid_meter": true} + st := wholeDay(t, func(id string, at time.Time) bool { + return !only[id] || (at.After(day.Add(3*time.Hour)) && at.Before(day.Add(6*time.Hour))) + }) + r, ok, err := Day(context.Background(), st, day) + if err != nil || !ok || r.GridImport != 24 || r.PVGeneration != 0 { + t.Fatalf("%+v ok %v err %v", r, ok, err) + } +} + +// An asset that left before the day began is not part of it. +func TestAnAssetRetiredBeforeTheDayIsLeftOut(t *testing.T) { + st := openStore(t) + record(t, st, day.Add(-48*time.Hour), day.Add(24*time.Hour+10*time.Minute), func(id string, at time.Time) bool { + return id == "battery:1/battery" && !at.Before(day.Add(-24*time.Hour)) + }) + r, ok, err := Day(context.Background(), st, day) + if err != nil || !ok || r.BatteryCharge != 0 || r.GridImport != 24 { + t.Fatalf("%+v ok %v err %v", r, ok, err) + } +} + +// An asset first seen after the day ended is not part of it. +func TestAnAssetAddedAfterTheDayLeavesItWhole(t *testing.T) { + st := wholeDay(t, nil) + w := 400.0 + for _, at := range []time.Time{day.Add(24*time.Hour + 30*time.Minute), day.Add(24*time.Hour + 32*time.Minute)} { + o := state.EnergyObservation{AssetID: "battery:2/battery", DeviceID: "battery:2", AssetKind: state.AssetBattery, Label: "x", + Flow: state.FlowBatteryCharge, AtMs: at.UnixMilli(), PowerW: &w} + if err := st.RecordTickWithOptionalHistory(nil, nil, []state.EnergyObservation{o}); err != nil { + t.Fatal(err) + } + } + if r, ok, err := Day(context.Background(), st, day); err != nil || !ok || r.BatteryCharge != 6 { + t.Fatalf("%+v ok %v err %v", r, ok, err) + } +} + +// A charger's session counter starts again at every plug-in. The ledger marks +// a reset and covers the interval from the power reading, so the day stays +// whole and the charge is counted. +func TestASessionCounterThatStartsAgainKeepsTheDay(t *testing.T) { + st := wholeDay(t, func(id string, _ time.Time) bool { return id == "charger:1/vehicle_charger" }) + var obs []state.EnergyObservation + first := day.Add(-10 * time.Minute) + for at := first; !at.After(day.Add(24*time.Hour + 10*time.Minute)); at = at.Add(tick) { + session := first + if !at.Before(day.Add(12 * time.Hour)) { + session = day.Add(12 * time.Hour) + } + w, c := 1000.0, 1000*at.Sub(session).Hours() + obs = append(obs, state.EnergyObservation{AssetID: "charger:2/vehicle_charger", DeviceID: "charger:2", AssetKind: state.AssetVehicleCharger, + Label: "x", Flow: state.FlowVehicleCharge, AtMs: at.UnixMilli(), PowerW: &w, CounterWh: &c}) + } + if err := st.RecordTickWithOptionalHistory(nil, nil, obs); err != nil { + t.Fatal(err) + } + r, ok, err := Day(context.Background(), st, day) + if err != nil || !ok || r.VehicleCharge != 24 { + t.Fatalf("%+v ok %v err %v", r, ok, err) + } +} diff --git a/go/internal/energyreport/energyreport.go b/go/internal/energyreport/energyreport.go new file mode 100644 index 000000000..75ce7451b --- /dev/null +++ b/go/internal/energyreport/energyreport.go @@ -0,0 +1,236 @@ +// Package energyreport sends one anonymous energy report a day, through an +// independent Oblivious HTTP relay, so that neither the relay nor FTW Cloud +// can tell which home sent it. +// +// The report is six whole-kWh sums of one closed UTC day and its date +// (package energywire). Package statsrelay seals it to FTW Cloud's gateway +// key, pinned in this release, and posts it to the relay: +// +// - the relay sees the box's address, the time and a ciphertext of fixed +// length, never the report; +// - the gateway sees the report and the relay's address, never the box's. +// It adds the sums to that day's totals and keeps nothing of the report. +// +// Two things keep the report from being joined to the daily fleet report, +// which carries the box's drivers, zone and size: it carries no field the +// fleet report carries, and it is sent alone at its own time, drawn afresh +// for each day from 1 to 12 hours after the day closes. With a fleet of a +// few boxes those fields can point at one home, so energy sent beside them +// would be that home's again. +// +// It runs only while the box shares statistics (cloudlink.SharesStatistics), +// the one answer that also covers the fleet report, and never for a site a +// partner controls. It adds no question or setting of its own. +package energyreport + +import ( + "context" + "encoding/json" + "errors" + "log/slog" + "time" + + "github.com/srcfl/ftw/go/internal/statsrelay" + "github.com/srcfl/ftw/go/protocol/energywire" +) + +const ( + // The send window after a day closes: late enough that the day's last + // readings are in the ledger, early enough that the gateway still takes + // the date. + firstSend = time.Hour + lastSend = 12 * time.Hour + + // lastDayKey records the last date this box sent, on the box only, so a + // restart does not send a day twice. + lastDayKey = "energy_report.last_date" +) + +// KV is where the box remembers the last date it sent. +type KV interface { + LoadConfig(key string) (string, bool) + SaveConfig(key, value string) error +} + +// Options wires the sender. +type Options struct { + // Enabled is the owner's answer and the partner rule, read at each send. + Enabled func() bool + Ledger Ledger + KV KV + + // Relay defaults to statsrelay.Pinned. Tests set a fake relay. + Relay *statsrelay.Client + Logger *slog.Logger + + // Injectable seams. Production leaves them nil. + Now func() time.Time + Random func() float64 + ClockSynced func() bool +} + +// Sender owns the schedule. +type Sender struct { + enabled func() bool + ledger Ledger + kv KV + relay *statsrelay.Client + log *slog.Logger + now func() time.Time + random func() float64 + synced func() bool +} + +func New(opts Options) (*Sender, error) { + if opts.Ledger == nil { + return nil, errors.New("energyreport: Ledger is required") + } + s := &Sender{enabled: opts.Enabled, ledger: opts.Ledger, kv: opts.KV, relay: opts.Relay, log: opts.Logger, + now: opts.Now, random: opts.Random, synced: opts.ClockSynced} + if s.relay == nil { + s.relay = statsrelay.Pinned() + } + if s.enabled == nil { + s.enabled = func() bool { return false } + } + if s.log == nil { + s.log = slog.Default() + } + s.log = s.log.With("component", "energyreport") + if s.now == nil { + s.now = time.Now + } + if s.random == nil { + s.random = randomFloat + } + if s.synced == nil { + s.synced = func() bool { return true } + } + return s, nil +} + +// Preview is what the box would send for yesterday, for Settings. +type Preview struct { + // Relay is where it goes, empty when this build has no relay. + Relay string `json:"relay"` + // Sending is whether this box sends at all: a relay and a gateway key in + // this build, and the owner's answer now. + Sending bool `json:"sending"` + // Day is the closed UTC day it describes. + Day string `json:"day"` + // Payload is the report, or nil when that day is incomplete. + Payload *energywire.Report `json:"payload"` +} + +// Preview builds yesterday's report with the same call the sender uses. +func (s *Sender) Preview(ctx context.Context) Preview { + day := s.now().UTC().Truncate(24*time.Hour).AddDate(0, 0, -1) + p := Preview{Day: day.Format(time.DateOnly)} + if s.relay != nil { + p.Relay, p.Sending = s.relay.Relay(), s.enabled() + } + if r, ok, err := Day(ctx, s.ledger, day); err == nil && ok { + p.Payload = &r + } + return p +} + +// Run sends once a day until ctx ends. Without a relay and a gateway key it +// returns at once and sends nothing. +func (s *Sender) Run(ctx context.Context) { + if s.relay == nil { + return + } + var p plan + for { + now := s.now() + if !p.live(now) { + p = s.draw(now) + } + if !now.Before(p.at) { + s.send(ctx, p.day) + p.done = true + continue + } + // Wake at least hourly, so a wall clock that NTP moves is noticed + // and the plan drawn again. + timer := time.NewTimer(min(p.at.Sub(now), time.Hour)) + select { + case <-ctx.Done(): + timer.Stop() + return + case <-timer.C: + } + } +} + +// plan is one day's report: the day it describes and when to send it. +type plan struct { + day time.Time + at time.Time + done bool +} + +// live reports whether the plan still belongs to the clock's day. +func (p plan) live(now time.Time) bool { + return !p.done && !p.day.IsZero() && !now.Before(p.day) && now.Before(p.day.Add(24*time.Hour+lastSend)) +} + +// draw plans the next report. Within a day's send window and before that day +// is sent, the send time is drawn from what is left of the window; otherwise +// from the next day's window. The time is drawn afresh each day, so it does +// not settle into a slot that would name the box. +func (s *Sender) draw(now time.Time) plan { + now = now.UTC() + closed := now.Truncate(24 * time.Hour) + day := closed.AddDate(0, 0, -1) + end := closed.Add(lastSend) + if now.Before(end) && !s.sent(day) { + from := closed.Add(firstSend) + if now.After(from) { + from = now + } + return plan{day: day, at: from.Add(time.Duration(s.random() * float64(end.Sub(from))))} + } + next := closed.AddDate(0, 0, 1) + return plan{day: closed, at: next.Add(firstSend + time.Duration(s.random()*float64(lastSend-firstSend)))} +} + +func (s *Sender) sent(day time.Time) bool { + if s.kv == nil { + return false + } + last, ok := s.kv.LoadConfig(lastDayKey) + return ok && last >= day.Format(time.DateOnly) +} + +// send makes one attempt for one day and forgets about it. There is no +// retry: a lost report is one fewer in a count that says it counts reports. +func (s *Sender) send(ctx context.Context, day time.Time) { + if s.relay == nil || !s.synced() || !s.enabled() || s.sent(day) { + return + } + r, ok, err := Day(ctx, s.ledger, day) + if err != nil || !ok { + s.log.Debug("no complete day to report", "day", day.Format(time.DateOnly), "err", err) + return + } + // The answer may have changed while the ledger was read. + if !s.enabled() { + return + } + // Remembered before it goes, so a crash mid-send loses the day rather + // than counting it twice. + if s.kv != nil { + if err := s.kv.SaveConfig(lastDayKey, r.Date); err != nil { + return + } + } + body, err := json.Marshal(r) + if err != nil { + return + } + if err := s.relay.Send(ctx, body); err != nil { + s.log.Debug("energy report did not go through", "err", err) + } +} diff --git a/go/internal/energyreport/energyreport_test.go b/go/internal/energyreport/energyreport_test.go new file mode 100644 index 000000000..b6b19863e --- /dev/null +++ b/go/internal/energyreport/energyreport_test.go @@ -0,0 +1,140 @@ +package energyreport + +import ( + "context" + "net/http" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/state" + "github.com/srcfl/ftw/go/internal/statsrelay/statsrelaytest" +) + +func TestTheSendTimeIsDrawnInTheWindowAfterTheDayCloses(t *testing.T) { + closed := day.AddDate(0, 0, 1) + for name, c := range map[string]struct { + now time.Time + random float64 + sent string + wantDay time.Time + from, to time.Time + }{ + "boot before the window": {closed.Add(20 * time.Minute), 0, "", day, closed.Add(time.Hour), closed.Add(time.Hour)}, + "latest in the window": {closed.Add(20 * time.Minute), 1, "", day, closed.Add(12 * time.Hour), closed.Add(12 * time.Hour)}, + "boot inside the window": {closed.Add(6 * time.Hour), 0.5, "", day, closed.Add(9 * time.Hour), closed.Add(9 * time.Hour)}, + "boot after the window": {closed.Add(13 * time.Hour), 0, "", closed, closed.Add(25 * time.Hour), closed.Add(25 * time.Hour)}, + "that day was sent already": {closed.Add(6 * time.Hour), 0, "2026-10-23", closed, closed.Add(25 * time.Hour), closed.Add(25 * time.Hour)}, + "a later day was sent first": {closed.Add(6 * time.Hour), 0, "2026-10-24", closed, closed.Add(25 * time.Hour), closed.Add(25 * time.Hour)}, + } { + kv := memKV{} + if c.sent != "" { + kv[lastDayKey] = c.sent + } + s := newSender(t, Options{KV: kv, Random: func() float64 { return c.random }}) + p := s.draw(c.now) + if !p.day.Equal(c.wantDay) || p.at.Before(c.from) || p.at.After(c.to) || !p.live(c.now) { + t.Fatalf("%s: plan %+v", name, p) + } + } + // A clock that jumps a year makes the plan stale. + p := plan{day: day, at: day.Add(30 * time.Hour)} + if p.live(day.AddDate(1, 0, 0)) || p.live(day.AddDate(-1, 0, 0)) || !p.live(day.Add(30*time.Hour)) { + t.Fatal("a stale plan stayed live") + } +} + +// Box to relay to gateway: the gateway reads the day's report, and a restart +// does not send the day again. +func TestABoxReportReachesTheGatewayThroughTheRelay(t *testing.T) { + client, gw, relay := statsrelaytest.New(t, day.Add(30*time.Hour)) + kv := memKV{} + s := newSender(t, Options{Ledger: whole(t), KV: kv, Relay: client, Enabled: func() bool { return true }, + Now: func() time.Time { return day.Add(30 * time.Hour) }}) + + s.send(context.Background(), day) + got := gw.Energy() + if len(got) != 1 || got[0].Date != "2026-10-23" || got[0].GridImport != 24 || got[0].VehicleCharge != 24 { + t.Fatalf("gateway got %+v", got) + } + if kv[lastDayKey] != "2026-10-23" || gw.Answered() != http.StatusNoContent || len(relay.Bodies()) != 1 { + t.Fatalf("kv %v answered %d", kv, gw.Answered()) + } + s.send(context.Background(), day) + if len(gw.Energy()) != 1 { + t.Fatal("the same day went twice") + } +} + +func TestNothingIsSentWithoutTheOwnersAnswer(t *testing.T) { + client, _, relay := statsrelaytest.New(t, day.Add(30*time.Hour)) + ledger := whole(t) + var mu sync.Mutex + reads := 0 + for name, o := range map[string]Options{ + "no answer": {Enabled: nil}, + "the answer is no": {Enabled: func() bool { return false }}, + "clock not set": {Enabled: func() bool { return true }, ClockSynced: func() bool { return false }}, + // Withdrawn while the ledger was being read. + "withdrawn mid-read": {Enabled: func() bool { + mu.Lock() + defer mu.Unlock() + reads++ + return reads == 1 + }}, + "incomplete day": {Enabled: func() bool { return true }, Ledger: wholeDay(t, func(id string, at time.Time) bool { return id == "inverter:1/pv" && at.After(day.Add(time.Hour)) })}, + } { + if o.Ledger == nil { + o.Ledger = ledger + } + o.Relay, o.KV = client, memKV{} + newSender(t, o).send(context.Background(), day) + if len(relay.Bodies()) != 0 { + t.Fatalf("%s: sent", name) + } + } +} + +// Without a relay and a gateway key, Run returns and never reads the ledger. +func TestWithoutARelayRunSendsNothing(t *testing.T) { + l := &countingLedger{} + s := newSender(t, Options{Ledger: l, Enabled: func() bool { return true }}) + done := make(chan struct{}) + go func() { s.Run(context.Background()); close(done) }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("Run waited with nowhere to send") + } + if l.n != 0 { + t.Fatal("read the ledger with nowhere to send") + } + if p := s.Preview(context.Background()); p.Relay != "" || p.Day == "" { + t.Fatalf("preview %+v", p) + } +} + +func newSender(t *testing.T, o Options) *Sender { + t.Helper() + if o.Ledger == nil { + o.Ledger = &countingLedger{} + } + s, err := New(o) + if err != nil { + t.Fatal(err) + } + return s +} + +type memKV map[string]string + +func (m memKV) LoadConfig(k string) (string, bool) { v, ok := m[k]; return v, ok } +func (m memKV) SaveConfig(k, v string) error { m[k] = v; return nil } + +type countingLedger struct{ n int } + +func (c *countingLedger) EnergyAssets() ([]state.EnergyAsset, error) { c.n++; return nil, nil } +func (c *countingLedger) LoadEnergyHistoryContext(context.Context, state.EnergyHistoryQuery) ([]state.EnergyLedgerPoint, bool, error) { + c.n++ + return nil, false, nil +} diff --git a/go/internal/energyreport/random.go b/go/internal/energyreport/random.go new file mode 100644 index 000000000..ec5eef048 --- /dev/null +++ b/go/internal/energyreport/random.go @@ -0,0 +1,16 @@ +package energyreport + +import ( + "crypto/rand" + "math/big" +) + +// randomFloat draws from the system source, as fleetping's does. +func randomFloat() float64 { + const precision = 1 << 53 + n, err := rand.Int(rand.Reader, big.NewInt(precision)) + if err != nil { + return 0.5 + } + return float64(n.Int64()) / precision +} diff --git a/go/internal/fleetping/fleetping.go b/go/internal/fleetping/fleetping.go index 812349f48..e3f8db017 100644 --- a/go/internal/fleetping/fleetping.go +++ b/go/internal/fleetping/fleetping.go @@ -5,6 +5,9 @@ // on which version, with which drivers, so engineering effort lands where the // fleet actually is. It runs against the constraint in docs/architecture.md: // the daily totals must not give the relay a stable household record. +// It also says, coarsely, how the box ran over the last day: how slow its +// slowest plan was, whether plans fell back, how often driver polls failed and +// whether an update failed, so the fleet's trouble is found where it is. // // So one ping is a picture of one box's shape and carries nothing that joins // two pings together: @@ -20,10 +23,15 @@ // - a send time that moves. A box that pings at 03:17 every night has told // the endpoint its name in a field nobody thought to look at. // -// What this cannot fix is the source IP, which the endpoint sees and which a -// household often keeps for months. That is the honest limit: this design -// makes the *payload* useless as an identifier, it does not make the -// connection anonymous. The drifting send time is what stops arrival time +// Sent straight to an endpoint, the source IP is what this cannot fix: the +// endpoint sees it, and a household often keeps one for months. A release +// that names a relay (package statsrelay) sends the report to FTW Cloud +// through it instead, sealed with Oblivious HTTP, so FTW Cloud reads the +// report without the address and the relay sees the address without the +// report. The sealed request also carries a random number drawn new for each +// report, and the day it was sent (package reportwire); neither joins two +// pings. Without a relay, this design makes the *payload* useless as an +// identifier; it does not make the connection anonymous. The drifting send time is what stops arrival time // becoming a second selector alongside it — though the drift is a random walk // in six-hour steps, so it takes about four days to spread across the clock, // and until then one box's arrival times are still loosely related. @@ -53,6 +61,7 @@ import ( "github.com/srcfl/ftw/go/internal/config" "github.com/srcfl/ftw/go/internal/prices" + "github.com/srcfl/ftw/go/internal/statsrelay" "github.com/srcfl/ftw/go/protocol/fleetwire" ) @@ -124,6 +133,30 @@ type Facts struct { // InstalledAt is when this box first ran FTW. InstalledAt time.Time + + // Health is how the box ran lately. + Health Health +} + +// Health is how the box ran lately, before it is coarsened. Every field is a +// sum over the whole box: never a driver's name, a plan's time or an +// update's version. +type Health struct { + // Plans is how many plans the planner kept in the last 24 hours, + // SlowestSolve the longest any of them took, and Fallbacks how many were + // a fallback plan. + Plans int + SlowestSolve time.Duration + Fallbacks int + + // Polls is how many driver polls ran in the last 24 hours, all drivers + // together, and PollErrors how many failed. Both stay zero while the + // count covers too little of the day to say anything. + Polls, PollErrors uint64 + + // Update is the outcome of the newest update in the last 7 days: "ok", + // "failed", or empty without one. + Update string } // Payload is the whole ping. Every field is a bucket or a label shared with @@ -138,16 +171,84 @@ type Payload = fleetwire.Report func Build(f Facts, now time.Time) Payload { version, channel := releaseVersion(f.Version) return Payload{ - Schema: Schema, - Version: version, - Channel: channel, - Drivers: driverTypes(f.Drivers, f.Catalogue), - Battery: batteryBucket(f.BatteryWh), - Zone: knownZone(f.PriceZone), - Age: ageBucket(f.InstalledAt, now), + Schema: Schema, + Version: version, + Channel: channel, + Drivers: driverTypes(f.Drivers, f.Catalogue), + Battery: batteryBucket(f.BatteryWh), + Zone: knownZone(f.PriceZone), + Age: ageBucket(f.InstalledAt, now), + Solve: solveBucket(f.Health), + Fallbacks: fallbackBucket(f.Health), + Errors: errorBucket(f.Health), + Update: updateBucket(f.Health.Update), + } +} + +// The health buckets answer where engineering time goes: is the planner fast +// enough on the hardware boxes run, does it fall back, do drivers fail, do +// updates. Each is three or four words wide and describes the last day, or +// for updates the last week, so it changes as the box runs instead of +// becoming one more fixed trait of the household. Most boxes share the +// common answer, which is what keeps the rarer ones in a crowd. + +// solveBucket is the slowest plan of the last 24 hours. Five seconds is the +// edge that matters: past it a small box's planner runs out of its budget. +func solveBucket(h Health) string { + switch s := h.SlowestSolve; { + case h.Plans <= 0: + return "none" + case s < time.Second: + return "0-1s" + case s < 5*time.Second: + return "1-5s" + default: + return "5s+" + } +} + +// fallbackBucket is how many of the last 24 hours' plans fell back. +func fallbackBucket(h Health) string { + switch { + case h.Plans <= 0: + return "none" + case h.Fallbacks <= 0: + return "0" + case h.Fallbacks <= 3: + return "1-3" + default: + return "4+" } } +// errorBucket is the share of the last 24 hours' driver polls that failed. +// A share, not a count, so a driver that polls every second and one that +// polls every minute land in the same bucket for the same trouble. None is +// no polls, or too few hours of them, as just after a restart. +func errorBucket(h Health) string { + switch { + case h.Polls == 0: + return "none" + case h.PollErrors == 0: + return "0" + case h.PollErrors*100 < h.Polls: + return "0-1%" + default: + return "1%+" + } +} + +// updateBucket is the newest update of the last 7 days. A box that does not +// update itself says none, like a box that had no update, so how a box was +// installed does not become one more field. +func updateBucket(outcome string) string { + switch outcome { + case "ok", "failed": + return outcome + } + return "none" +} + // releaseTag matches the two shapes this project publishes: vX.Y.Z and // vX.Y.Z-beta.N, and nothing looser. // @@ -347,6 +448,26 @@ func (h *HTTPProvider) Send(ctx context.Context, p Payload) error { return nil } +// RelayProvider sends the report through the relay (package statsrelay), as +// its own message at its own time: never beside the energy report, whose +// sums a driver set and a price zone could otherwise tie to one home. +type RelayProvider struct { + c *statsrelay.Client +} + +func NewRelayProvider(c *statsrelay.Client) *RelayProvider { return &RelayProvider{c: c} } + +func (p *RelayProvider) Send(ctx context.Context, payload Payload) error { + body, err := json.Marshal(payload) + if err != nil { + return err + } + return p.c.Send(ctx, body) +} + +// EndpointURL is the relay, which is where the box sends. +func (p *RelayProvider) EndpointURL() string { return p.c.Relay() } + // Options wires the pinger. Everything except the injectable seams is required. type Options struct { // Enabled is read at each send rather than captured, so turning the ping @@ -426,6 +547,16 @@ func (p *Pinger) Endpoint() string { return "" } +// Enabled is whether the next report would go, read the way the sender reads +// it, so Settings says what the box does. +func (p *Pinger) Enabled() bool { return p.enabled() } + +// ThroughRelay reports whether the running provider sends through the relay. +func (p *Pinger) ThroughRelay() bool { + _, ok := p.provider.(*RelayProvider) + return ok +} + // Run pings once a day until ctx ends. func (p *Pinger) Run(ctx context.Context) { for { diff --git a/go/internal/fleetping/fleetping_test.go b/go/internal/fleetping/fleetping_test.go index b0848ab9b..e62708daa 100644 --- a/go/internal/fleetping/fleetping_test.go +++ b/go/internal/fleetping/fleetping_test.go @@ -1,6 +1,7 @@ package fleetping import ( + "bytes" "context" "encoding/json" @@ -15,6 +16,8 @@ import ( "time" "github.com/srcfl/ftw/go/internal/config" + "github.com/srcfl/ftw/go/internal/statsrelay/statsrelaytest" + "github.com/srcfl/ftw/go/protocol/fleetwire" ) // The whole point of this package is what it does NOT send. These tests are @@ -51,12 +54,16 @@ func TestPayloadCarriesOnlyTheAgreedFields(t *testing.T) { got := marshal(t, Build(facts(), time.Date(2026, 8, 5, 12, 0, 0, 0, time.UTC))) want := map[string]any{ - "schema": "ftw.fleet/1", - "ftw_version": "v1.4.0-beta.2", - "channel": "beta", - "battery_kwh": "5-15", - "price_zone": "SE3", - "install_age": "6-12m", + "schema": "ftw.fleet/2", + "ftw_version": "v1.4.0-beta.2", + "channel": "beta", + "battery_kwh": "5-15", + "price_zone": "SE3", + "install_age": "6-12m", + "plan_solve": "none", + "plan_fallbacks": "none", + "driver_errors": "none", + "update": "none", } for key, value := range want { if got[key] != value { @@ -451,7 +458,7 @@ func TestHTTPProviderPostsTheJSONAndReadsNothingBack(t *testing.T) { if err := provider.Send(context.Background(), Build(facts(), time.Now())); err != nil { t.Fatalf("Send: %v", err) } - if got := body.Load().(string); !strings.Contains(got, `"schema":"ftw.fleet/1"`) { + if got := body.Load().(string); !strings.Contains(got, `"schema":"ftw.fleet/2"`) { t.Fatalf("posted body = %s", got) } if got := agent.Load().(string); got != "ftw-fleet-ping" { @@ -542,3 +549,59 @@ func TestHTTPProviderDefaultClientHasATimeout(t *testing.T) { t.Fatal("the default client has no timeout") } } + +// Through the relay the gateway reads the same report the box would post +// directly, and the relay sees none of it. +func TestTheReportGoesThroughTheRelayUnchanged(t *testing.T) { + client, gw, relay := statsrelaytest.New(t, time.Now()) + p := NewRelayProvider(client) + facts := Facts{Version: "v1.4.0", Drivers: []string{"sungrow"}, Catalogue: []string{"sungrow"}, PriceZone: "SE3"} + pinger, err := New(Options{Facts: func() Facts { return facts }, Provider: p}) + if err != nil { + t.Fatal(err) + } + want := pinger.Payload() + if err := p.Send(context.Background(), want); err != nil { + t.Fatal(err) + } + got := gw.Fleet() + if len(got) != 1 || got[0].Version != want.Version || got[0].Zone != "SE3" || len(got[0].Drivers) != 1 { + t.Fatalf("gateway got %+v", got) + } + if bytes.Contains(relay.Bodies()[0], []byte("sungrow")) || !pinger.ThroughRelay() || pinger.Endpoint() != relay.URL { + t.Fatal("the relay saw the report, or the pinger does not say where it sends") + } +} + +// ---- health ---- + +func TestHealthTravelsOnlyAsBuckets(t *testing.T) { + // Each case is what a box held over its last day; what leaves is a word + // from a short list, never the count or the time behind it. + for _, c := range []struct { + h Health + solve, fallbacks, errors, update string + }{ + {Health{}, "none", "none", "none", "none"}, + {Health{Plans: 96, SlowestSolve: 420 * time.Millisecond, Polls: 86_400}, "0-1s", "0", "0", "none"}, + {Health{Plans: 96, SlowestSolve: time.Second, Fallbacks: 3, Polls: 1000, PollErrors: 9, Update: "ok"}, "1-5s", "1-3", "0-1%", "ok"}, + {Health{Plans: 12, SlowestSolve: 5 * time.Second, Fallbacks: 4, Polls: 1000, PollErrors: 10, Update: "failed"}, "5s+", "4+", "1%+", "failed"}, + {Health{Plans: 1, SlowestSolve: 47 * time.Second, Polls: 3, PollErrors: 3, Update: "v1.4.0"}, "5s+", "0", "1%+", "none"}, + } { + f := facts() + f.Health = c.h + p := Build(f, time.Now()) + if p.Solve != c.solve || p.Fallbacks != c.fallbacks || p.Errors != c.errors || p.Update != c.update { + t.Errorf("%+v: got %s %s %s %s", c.h, p.Solve, p.Fallbacks, p.Errors, p.Update) + } + raw, _ := json.Marshal(p) + if _, err := fleetwire.Decode(raw); err != nil { + t.Errorf("the service would refuse %s", raw) + } + for _, leak := range []string{"420", "47", "86400", "v1.4.0\"", "1000"} { + if strings.Contains(string(raw), leak) { + t.Errorf("%s carries %s", raw, leak) + } + } + } +} diff --git a/go/internal/ftwcli/backup.go b/go/internal/ftwcli/backup.go index 1e1e1df83..2b41fc15e 100644 --- a/go/internal/ftwcli/backup.go +++ b/go/internal/ftwcli/backup.go @@ -29,6 +29,7 @@ type backupProgress struct { TotalBytes int64 `json:"total_bytes"` Table string `json:"table"` RowsDone int64 `json:"rows_done"` + Error string `json:"error"` } type backupList struct { @@ -87,6 +88,10 @@ func runBackup(args []string, out io.Writer, e env) error { if len(before.Backups) > 0 && before.FreeBytes > 0 && before.FreeBytes < before.Backups[0].SizeBytes { fmt.Fprintf(out, "Warning: less space is free than the last backup took (%s)\n", formatBytes(before.Backups[0].SizeBytes)) } + // An earlier run may have ended after its caller left. + if before.Progress.Phase == "failed" && before.Progress.Error != "" { + fmt.Fprintf(out, "Last try: failed: %s\n", before.Progress.Error) + } } fmt.Fprintln(out, "Making a full backup. Core keeps running; Ctrl-C stops the backup.") start := e.now() diff --git a/go/internal/ftwcli/cli_test.go b/go/internal/ftwcli/cli_test.go index 38d45c431..029d996a2 100644 --- a/go/internal/ftwcli/cli_test.go +++ b/go/internal/ftwcli/cli_test.go @@ -459,6 +459,19 @@ func TestBackupCopiesOnlyAVerifiedMatch(t *testing.T) { } } +// A backup can fail after its caller has left, as on the home box on +// 16 October 2026. The next run names that failure and its own. +func TestBackupNamesTheLastFailureAndItsOwn(t *testing.T) { + const reason = "live backup reached its time limit of 2h0m0s: backup history: copy ts_buckets: context deadline exceeded" + f, srv := newFakeCore(t) + f.on("GET", "/api/backups", reply(200, `{"dir":"/var/lib/ftw/backups","progress":{"phase":"failed","table":"ts_buckets","rows_done":14200000,"error":"`+reason+`"}}`)) + f.on("POST", "/api/backups", reply(500, `{"error":"`+reason+`"}`)) + code, out, errOut := runCLI(t, testEnv(), "backup", "--url", srv.URL) + if code != exitFailed || !strings.Contains(out, "Last try: failed: "+reason) || !strings.Contains(errOut, "HTTP 500: "+reason) { + t.Fatalf("exit %d\n%s%s", code, out, errOut) + } +} + func TestSupportWritesAPrivateReadableZip(t *testing.T) { var archive bytes.Buffer zw := zip.NewWriter(&archive) diff --git a/go/internal/ha/bridge.go b/go/internal/ha/bridge.go index 1faa8e396..9aae52446 100644 --- a/go/internal/ha/bridge.go +++ b/go/internal/ha/bridge.go @@ -492,6 +492,7 @@ func (b *Bridge) publishDiscovery() { {"battery_power", "Battery Power", "W", "power", "measurement", b.stateTopic("bat_w")}, {"ev_power", "EV Power", "W", "power", "measurement", b.stateTopic("ev_w")}, {"v2x_power", "V2X Power", "W", "power", "measurement", b.stateTopic("v2x_w")}, + // Live balance as the dashboard shows it. {"load_power", "Load Power", "W", "power", "measurement", b.stateTopic("load_w")}, {"battery_soc", "Battery SoC", "%", "battery", "measurement", b.stateTopic("bat_soc_pct")}, // Control setpoints @@ -949,8 +950,8 @@ func (b *Bridge) publishState() { b.publishValue("bat_w", batW) b.publishValue("ev_w", evW) b.publishValue("v2x_w", v2xW) - if live.Valid { - b.publishValue("load_w", live.HouseholdW) + if live.LoadKnown { + b.publishValue("load_w", live.LoadW) } else { // HA's numeric MQTT sensors ignore an empty payload. The literal // None replaces the previous value with unknown. diff --git a/go/internal/ha/live_load_test.go b/go/internal/ha/live_load_test.go index e34687aeb..a0c41e42a 100644 --- a/go/internal/ha/live_load_test.go +++ b/go/internal/ha/live_load_test.go @@ -2,6 +2,7 @@ package ha import ( "encoding/json" + "strings" "sync" "testing" "time" @@ -99,9 +100,9 @@ func TestLoadPublishClearsUnknownAndRecovers(t *testing.T) { } check("1000.00") + // A declared battery that never reported is left out, not a blank. options.ExpectedFlows = []telemetry.ForecastFlow{{Driver: "battery", DerType: telemetry.DerBattery}} - check("None") // HA ignores an empty numeric payload; None clears it to unknown. - check("None") + check("1000.00") b.tel.EnsureDriverHealth("battery") b.tel.Update("battery", telemetry.DerBattery, -200, nil, nil) check("1200.00") @@ -110,8 +111,40 @@ func TestLoadPublishClearsUnknownAndRecovers(t *testing.T) { b.tel.Update("battery", telemetry.DerBattery, 0, nil, nil) check("0.00") + // Unresolved topology still has a read site meter: an estimate. options.HouseholdInvalidReason = "unresolved_topology" - check("None") + check("0.00") + b.tel.DriverHealthMut("meter").SetOffline() + check("None") // HA ignores an empty numeric payload; None clears it to unknown. + b.tel.DriverHealthMut("meter").RecordSuccess() options.HouseholdInvalidReason = "" check("0.00") } + +// HA gets one Home use: the live balance as the dashboard shows it, +// negative included. +func TestLoadPublishesOneHomeUse(t *testing.T) { + b, client := liveLoadBridge(nil) + for _, d := range []string{"meter", "battery"} { + b.tel.EnsureDriverHealth(d) + } + b.tel.Update("meter", telemetry.DerMeter, 1200, nil, nil) + b.tel.Update("battery", telemetry.DerBattery, 2000, nil, nil) + b.publishState() + if got := client.last(t, b.stateTopic("load_w")); got != "-800.00" { + t.Fatalf("display load = %s, want the negative balance as is", got) + } + b.publishDiscovery() + var display, energy bool + client.mu.Lock() + for topic, payloads := range client.payloads { + for _, p := range payloads { + display = display || strings.Contains(topic, "load_power/config") && strings.Contains(p, b.stateTopic("load_w")) + energy = energy || strings.Contains(topic, "load_energy_power/config") + } + } + client.mu.Unlock() + if !display || energy { + t.Fatalf("discovery: display=%t energy=%t", display, energy) + } +} diff --git a/go/internal/loadpoint/battery_boost.go b/go/internal/loadpoint/battery_boost.go index 266dc9b4e..40a2e2926 100644 --- a/go/internal/loadpoint/battery_boost.go +++ b/go/internal/loadpoint/battery_boost.go @@ -32,6 +32,7 @@ const ( BatteryBoostStoppedBatteryHold BatteryBoostStopReason = "battery_hold" BatteryBoostStoppedCoreMode BatteryBoostStopReason = "core_mode" BatteryBoostStoppedFuseSafety BatteryBoostStopReason = "fuse_safety_block" + BatteryBoostStoppedImportLimit BatteryBoostStopReason = "import_limit_block" BatteryBoostStoppedRestartInvalid BatteryBoostStopReason = "restart_lease_invalid" ) diff --git a/go/internal/loadpoint/controller.go b/go/internal/loadpoint/controller.go index 472cb253a..b13b7e2d4 100644 --- a/go/internal/loadpoint/controller.go +++ b/go/internal/loadpoint/controller.go @@ -46,25 +46,26 @@ type Controller struct { // fuseEVMax is this tick's EV share of the site fuse. nil means no cap. fuseEVMax func() (float64, bool) + // fusePhaseRoomA is the current chargers may still add on the busiest + // phase. nil, or false, leaves the cap alone to limit them. + fusePhaseRoomA func() (float64, bool) + // fuseLimitReason names the site limit behind the cap. + fuseLimitReason func() string + // sitePhases reads the site meter's phases for the one-phase car check. + sitePhases func() (SitePhases, bool) - // perPhaseMeterAmps returns the live site-meter per-phase currents (L1, - // L2, L3). Wired from telemetry; nil disables the phase increase gate. - perPhaseMeterAmps func() (l1, l2, l3 float64, ok bool) // tickFuseCaps belongs to the serial tick, after one shared budget read. tickFuseCaps map[string]float64 - tickFusePhases int + tickFuseRooms map[string]fuseRoom // serial tick, beside tickFuseCaps + tickFuseReason string + tickChoices map[string]tickChoice // serial tick tickDirectiveActive bool tickDirectiveOK bool tickDirective Directive - // offerPeaks is each loadpoint's highest recent offer (serial tick). - offerPeaks map[string]offerPeak // savedOffers mirrors each loadpoint's last order kept across restarts. - savedOffers map[string]savedOffer - // fuseResumeTicks is owned by the serial tick. A paused charger needs - // two consecutive dispatch ticks with room for a legal step. - fuseResumeTicks map[string]int - fuseDemandMu sync.RWMutex - fuseDemands []FuseDemand + savedOffers map[string]savedOffer + fuseDemandMu sync.RWMutex + fuseDemands []FuseDemand // siteSurplusForEVW is live PV left after house load and battery soak. // A false reading pauses surplus_only rather than guessing an import. @@ -207,8 +208,6 @@ const nearTermLogCooldown = 10 * time.Minute // Easee contactor and winds up the battery PI. const phaseSwitchMinHold = 30 * time.Minute -const fuseResumeSafeTicks = 2 - // CommandSettle is how long a new current may take to reach the site meter. // Easee Cloud needs about 5-15 s (#1674). Until then the meter may still show // an earlier, higher offer. @@ -524,15 +523,43 @@ func (c *Controller) SetFuseEVMax(f func() (float64, bool)) { c.fuseEVMax = f } -// SetPerPhaseMeterAmps wires the live site-meter per-phase current reader -// used by the per-phase fuse clamp. -func (c *Controller) SetPerPhaseMeterAmps(f func() (l1, l2, l3 float64, ok bool)) { +// SetFusePhaseRoomA wires control.EVFusePhaseRoomA: the current the +// chargers may still add on the busiest phase, net of every charger's +// commands the meter may not show yet. Read once per tick, with the cap. +func (c *Controller) SetFusePhaseRoomA(f func() (float64, bool)) { if c == nil { return } - c.perPhaseMeterAmps = f + c.fusePhaseRoomA = f } +// SetFuseLimitReason wires control.EVLimitReason: "import_limit" when the +// clock hour's import ceiling holds the cap, else "fuse_limit". Read once +// per tick, after the cap. +func (c *Controller) SetFuseLimitReason(f func() string) { + if c == nil { + return + } + c.fuseLimitReason = f +} + +// capReason is the reason a command the cap lowered carries. +func (c *Controller) capReason() string { + if c.tickFuseReason != "" { + return c.tickFuseReason + } + if c.fuseLimitReason != nil { + if r := c.fuseLimitReason(); r != "" { + return r + } + } + return "fuse_limit" +} + +// SetPerPhaseMeterAmps does nothing. Deprecated: control.EVFuseBudget reads +// the phase currents; this stays until the last test caller is gone. +func (c *Controller) SetPerPhaseMeterAmps(func() (l1, l2, l3 float64, ok bool)) {} + // SetSiteSurplusForEV wires a per-tick "PV surplus available to the // EV" reader for the surplus_only clamp. The function returns watts // the EV may claim this tick: leftover after house load, minus @@ -595,16 +622,13 @@ func (c *Controller) SetPeakRemainingSurplusW(f func() (float64, bool)) { c.peakRemainingSurplusW = f } -// fusePhaseMarginA is the per-phase amp headroom held below the breaker. -const fusePhaseMarginA = 1.0 - -// applyFuseClampAndCooldown applies the frozen hard cap. A short recovery -// wait is part of that cap; an unknown startup budget never starts a timer. -func (c *Controller) applyFuseClampAndCooldown(now time.Time, lpCfg Config, wantW float64) (float64, string) { +// applyFuseClamp applies this loadpoint's share of the frozen hard cap. It +// snaps to steps that are legal on the phases the command names. +func (c *Controller) applyFuseClamp(lpCfg Config, wantW float64, phaseMode string) (float64, string) { if c == nil { return wantW, "" } - lpCfg = c.fuseConfig(now, lpCfg) + lpCfg = c.fuseConfig(lpCfg, phaseMode) if c.fuseEVMax == nil { return wantW, "" } @@ -615,9 +639,6 @@ func (c *Controller) applyFuseClampAndCooldown(now time.Time, lpCfg Config, want if !finite(cap) { cap = 0 } - if c.fuseResumeTicks[lpCfg.ID] > 0 { - return 0, "fuse_cooldown" - } if wantW <= cap { return wantW, "" } @@ -626,12 +647,12 @@ func (c *Controller) applyFuseClampAndCooldown(now time.Time, lpCfg Config, want if snapped > 0 && snapped >= lpCfg.MinChargeW { slog.Info("loadpoint fuse-clamp: ramped down", "lp", lpCfg.ID, "want_w", wantW, "fuse_cap_w", cap, "snapped_w", snapped) - return snapped, "fuse_limit" + return snapped, c.capReason() } slog.Info("loadpoint fuse-clamp: below minimum step", "lp", lpCfg.ID, "want_w", wantW, "fuse_cap_w", cap, "min_step_w", lpCfg.MinChargeW) - return 0, "fuse_limit" + return 0, c.capReason() } // SetNearTermPeakSurplusW wires the short-horizon "peak surplus over @@ -1171,11 +1192,6 @@ func (c *Controller) TickWithDispatch(ctx context.Context, now time.Time, dispat return } c.clearFuseDemands() - if !dispatchAllowed { - for id := range c.fuseResumeTicks { - c.fuseResumeTicks[id] = 0 - } - } // Stop every affected charger before observing any session. Observation // may restore or sync state.db; a stalled disk must not delay standdown, // including the chargers after the one whose checkpoint is blocked. @@ -1220,13 +1236,86 @@ func (c *Controller) TickWithDispatch(ctx context.Context, now time.Time, dispat if dispatchAllowed { c.tickDirective, c.tickDirectiveOK = c.currentDirective(now) c.tickDirectiveActive = true + defer func() { + c.tickFuseCaps, c.tickFuseRooms, c.tickFuseReason = nil, nil, "" + c.tickChoices = nil + c.tickDirectiveActive = false + }() + } + choices := make(map[string]tickChoice, len(observations)) + for _, observed := range observations { + if !observed.sample.ConnectionUnknown { + choices[observed.config.ID] = c.chooseTick(now, observed.config, observed.sample, dispatchAllowed) + } + } + if dispatchAllowed { + c.tickChoices = choices c.restoreOffers(now, samples) c.freezeFuseBudget(now, samples) - defer func() { c.tickFuseCaps = nil; c.tickFusePhases = 0; c.tickDirectiveActive = false }() } for _, observed := range observations { - c.tickOne(ctx, now, observed.config, observed.sample, dispatchAllowed, observed.wasPlugged, observed.wasDelivering) + c.tickOne(ctx, now, observed.config, observed.sample, choices[observed.config.ID], dispatchAllowed, observed.wasPlugged, observed.wasDelivering) + } +} + +// tickChoice is one loadpoint's surplus and phase decision for a tick. The +// fuse cap is measured for its phases, and the command names them. +type tickChoice struct { + surplusOn bool // live surplus replaces the plan + surplusAdds bool // live surplus may raise a scheduled charge + steps []float64 // surplus steps, picked once + phaseMode string // the command's phase_mode; "" is the driver's 3p +} + +// chooseTick advances the battery-SoC unlock and the surplus phase dwell +// once, before the shared fuse budget freezes. +func (c *Controller) chooseTick(now time.Time, cfg Config, sample EVSample, dispatchAllowed bool) tickChoice { + sched, _ := c.manager.GetSchedule(cfg.ID) + choice := tickChoice{surplusOn: c.surplusActive(cfg, sched), surplusAdds: c.surplusAddsToPlan(cfg, sched)} + if !sample.Connected || !dispatchAllowed || !c.driverCanDispatch(cfg.DriverName) { + return choice // No command this tick. + } + if hold, held := c.GetManualHold(cfg.ID, now); held { + choice.phaseMode = holdPhaseMode(hold, cfg, c.siteFuse()) + } else { + if choice.surplusOn || choice.surplusAdds { + choice.steps = c.pickSurplusSteps(now, cfg) + } + // Easee treats an unset phase_mode as 3p and then ignores a 1Φ step. + // A schedule still overrides the surplus 1Φ lock, or a deadline + // charge stays near 3.7 kW on a cloudy day. + // A solar-only loadpoint charges from surplus whatever its goal, so + // its schedule does not make the charge a planned one. + choice.phaseMode = resolvePhaseMode(cfg.PhaseMode, threePhaseCapable(cfg, c.siteFuse()), sched.HasTarget() && !cfg.SurplusOnly, + c.surplusLockedTo1P(cfg.ID), choice.surplusOn, c.dwellSelectedPhaseMode(cfg.ID)) + } + if c.siteFuse().Phases() == 1 { + choice.phaseMode = "1p" // As applyInstallationLimits sends it. + } + return choice +} + +// holdPhaseMode: a pinned hold or loadpoint mode wins. Otherwise Charge now +// runs as planned charging does: three phases on a charger that can, one on +// a charger that cannot. A hold below the 3Φ minimum goes out as auto, so a +// sticky Start on a 1380 W surplus slot does not die at 3Φ's 6 A. A +// client's 3p on a charger that cannot run three phases is not a pin: the +// app sends the phases the API reported, and a 3p order there is 0 W. The +// loadpoint's own mode decides. +func holdPhaseMode(hold ManualHold, cfg Config, site SiteFuse) string { + mode := hold.PhaseMode + if mode == "" || (mode == "3p" && !threePhaseCapable(cfg, site)) { + mode = cfg.PhaseMode + } + switch { + case mode != "" && mode != "auto": + return mode + case !threePhaseCapable(cfg, site): + return "1p" + case hold.PowerW < phaseMinChargeW(cfg, "3p", site): + return "auto" } + return "3p" } func (c *Controller) observeTick(now time.Time, cfg Config, sample EVSample, dispatchAllowed bool) { @@ -1246,6 +1335,7 @@ func (c *Controller) observeTick(now time.Time, cfg Config, sample EVSample, dis wasSurplus := cfg.SurplusOnly || (!sched.HasTarget() && armed) c.manager.SetSurplusWithheld(cfg.ID, wasSurplus && enteringSurplusPaused) c.manager.ObserveSample(cfg.ID, sample) + c.observePhases(cfg, sample, now) c.restoreManualHoldForSession(cfg.ID) c.evaluateBatteryBoost(cfg.ID, now, sample.Connected, dispatchAllowed && !sample.PowerUnavailable) } @@ -1305,24 +1395,17 @@ func (c *Controller) standDownBeforeStorage(ctx context.Context, now time.Time, } } -func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, sample EVSample, dispatchAllowed, wasPlugged, wasDelivering bool) { +func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, sample EVSample, choice tickChoice, dispatchAllowed, wasPlugged, wasDelivering bool) { if sample.ConnectionUnknown { delete(c.resumeOffers, lpCfg.ID) return } - // Resolve the schedule once per tick — used for the bat-SoC unlock - // (surplusActive / surplusAddsToPlan) and the phase decision below. - // Zero value when no schedule is set, which makes evalBatSoCArm a - // no-op. - var sched Schedule - if c.manager != nil { - sched, _ = c.manager.GetSchedule(lpCfg.ID) - } // surplusOn: surplus REPLACES the plan (surplus-only semantics). // surplusAdds: surplus is ADDED on top of a scheduled plan. Never // both true. - surplusOn := c.surplusActive(lpCfg, sched) - surplusAdds := c.surplusAddsToPlan(lpCfg, sched) + surplusOn, surplusAdds := choice.surplusOn, choice.surplusAdds + // The fuse cap was measured for this phase mode. + phaseMode := choice.phaseMode enteringSurplusPaused, _ := c.getSurplusPause(lpCfg.ID) selfWithheld := surplusOn && enteringSurplusPaused // Detect the disconnected→connected edge (state.PluggedIn flips @@ -1400,18 +1483,11 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s // guard a manual hold can never override. (surplusOn still drives // the phase-mode fallback further down.) cmd["power_w"] = holdW - // Phase mode: explicit hold > explicit LP config > surplus - // default ("auto") > driver default. Same surplus-active fallback - // as the non-hold branch so a sticky Start hold on a 1380 W - // surplus slot actually delivers instead of dying at the Easee - // driver's unset → "3p" interpretation. - switch { - case hold.PhaseMode != "": - cmd["phase_mode"] = hold.PhaseMode - case lpCfg.PhaseMode != "": - cmd["phase_mode"] = lpCfg.PhaseMode - case surplusOn: - cmd["phase_mode"] = "auto" + // A hold changed after the budget froze names its own phases; the + // phase check below makes it wait for the next tick's budget. + phaseMode = holdPhaseMode(hold, lpCfg, c.siteFuse()) + if phaseMode != "" { + cmd["phase_mode"] = phaseMode } switch { case hold.PhaseSplitW > 0: @@ -1445,7 +1521,8 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s cmd["site_phases"] = site.Phases() } } else { - cmdW, planReady, _ := c.computeCommandForBudget(now, lpCfg, sample.PowerW, false) + // Snap to steps legal on the phases this command names. + cmdW, planReady, _ := c.computeCommandForBudget(now, c.fuseConfig(lpCfg, phaseMode), sample.PowerW, false) cmdReason = "plan" if c.plan != nil { if d, ok := c.currentDirective(now); ok && d.LoadpointFallback[lpCfg.ID] { @@ -1473,7 +1550,7 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s if wantW <= 0 { wantW = lpCfg.MaxChargeW } - cmdW = c.computeSurplusCmd(now, lpCfg, wantW, sample.PowerW) + cmdW = c.computeSurplusCmd(now, lpCfg, wantW, choice.steps) if cmdW > 0 { cmdReason = "pv_surplus" } else { @@ -1482,7 +1559,7 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s } // Spare PV may raise a scheduled charge. It must not lower it. if surplusAdds { - surplusW := c.computeSurplusCmd(now, lpCfg, lpCfg.MaxChargeW, sample.PowerW) + surplusW := c.computeSurplusCmd(now, lpCfg, lpCfg.MaxChargeW, choice.steps) if surplusW > cmdW { cmdW = surplusW cmdReason = "pv_surplus" @@ -1509,16 +1586,6 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s cmdW = 0 } cmd["power_w"] = cmdW - // Easee treats an unset phase_mode as 3p and then ignores a 1Φ step. - // A schedule still overrides the surplus 1Φ lock, or a deadline - // charge stays near 3.7 kW on a cloudy day. - phaseMode := resolvePhaseMode( - lpCfg.PhaseMode, - sched.HasTarget(), - c.surplusLockedTo1P(lpCfg.ID), - surplusOn, - c.dwellSelectedPhaseMode(lpCfg.ID), - ) if phaseMode != "" { cmd["phase_mode"] = phaseMode } @@ -1545,11 +1612,12 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s // A manual diagnostic may lower a site limit, but never replace the // installation's voltage, phase count or fuse with a larger offer. c.applyInstallationLimits(cmd) + applyChargerPhaseLimit(cmd, lpCfg, c.siteFuse()) if applyCurrentCeiling(cmd) { cmdReason = "charger_limit" } if wantW, ok := cmd["power_w"].(float64); ok { - phaseMode, _ := cmd["phase_mode"].(string) + phaseMode, _ = cmd["phase_mode"].(string) c.recordFuseDemand(lpCfg.DriverName, wantW, phaseMode) // The reserve directive requires a known physical ceiling. Capture // its intent above, but never turn an unknown cap into permission. @@ -1558,22 +1626,28 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s wantW, cmdReason = 0, "fuse_limit" } } - limitedW, reason := c.applyFuseClampAndCooldown(now, lpCfg, wantW) + limitedW, reason := c.applyFuseClamp(lpCfg, wantW, phaseMode) cmd["power_w"] = limitedW if reason != "" { cmdReason = reason } } // A new API choice can change phase after the shared budget was frozen. - // Do not spend a 3p share on one phase; the next tick uses the new choice. - if w, ok := cmd["power_w"].(float64); ok && w > 0 && c.tickFusePhases > 1 { - if mode, _ := cmd["phase_mode"].(string); mode != "" && mode != "3p" { + // Do not spend that budget on other phases; the next tick uses the choice. + if w, ok := cmd["power_w"].(float64); ok && w > 0 && c.tickFuseCaps != nil { + if mode, _ := cmd["phase_mode"].(string); fusePhase(mode) != fusePhase(choice.phaseMode) { cmd["power_w"], cmdReason = float64(0), "fuse_limit" } } if applyCurrentCeiling(cmd) { cmdReason = "fuse_limit" } + // cap_amps_per_phase is how far this order may rise on its phases. A + // driver that briefly offers more than the order, as easee_cloud does to + // start a car at 6 A, must stay within it. Absent: Core knows no cap. + if capA, known := c.capAmpsPerPhase(lpCfg, cmd); known { + cmd["cap_amps_per_phase"] = capA + } // Tell the manager what was ordered, after every clamp has spoken. // The interruption latch reads this to keep a pause the box chose — // plan slot, Stop hold, surplus clamp — from ever reading as a @@ -1583,8 +1657,17 @@ func (c *Controller) tickOne(ctx context.Context, now time.Time, lpCfg Config, s if w, ok := cmd["power_w"].(float64); ok { offerW, haveOffer = w, true c.manager.setCommandedForManual(lpCfg.ID, w, cmdReason, manualCommandUpdatedAt) - phase := c.recordOfferPeak(now, lpCfg.ID, w, cmd) - c.saveOffer(now, lpCfg, sample, w, phase, cmdReason) + mode, _ := cmd["phase_mode"].(string) + split, _ := cmd["phase_split_w"].(float64) + c.manager.setCommandedPhases(lpCfg.ID, c.commandPhases(mode, w, split)) + c.saveOffer(now, lpCfg, sample, w, fusePhase(mode), cmdReason) + } + // A one-phase car takes a third of a three-phase order: command evidence + // compares its draw with that. + if st, known := c.manager.State(lpCfg.ID); known && haveOffer && st.CarPhases == 1 && c.siteFuse().Phases() == 3 { + if mode, _ := cmd["phase_mode"].(string); fusePhase(mode) == "3p" { + cmd["expected_w"] = offerW / 3 + } } payload, err := json.Marshal(cmd) if err != nil { @@ -1843,9 +1926,12 @@ func (c *Controller) shouldKickWallboxForResume(now time.Time, lpID string, surp // decides pause and resume. The magnitude tracks the instant value, or // a falling cloud imports before the average catches up. 0 pauses. // +// steps comes from pickSurplusSteps, called once per tick before the +// shared fuse budget freezes. +// // TODO: siteSurplusForEVW is one site budget. Two surplus loadpoints // each take all of it and can import. -func (c *Controller) computeSurplusCmd(now time.Time, lpCfg Config, wantW, currentEvW float64) float64 { +func (c *Controller) computeSurplusCmd(now time.Time, lpCfg Config, wantW float64, steps []float64) float64 { if c == nil { return wantW } @@ -1872,21 +1958,16 @@ func (c *Controller) computeSurplusCmd(now time.Time, lpCfg Config, wantW, curre // `−pvW − loadW`, i.e. PV-magnitude minus house load — invariant // under whatever the home battery is currently doing with the // surplus. The EV's own draw is part of that closure already, so - // we don't add currentEvW here. + // we don't add it here. instant := surplusW if instant < 0 { instant = 0 } avg := c.recordSurplus(lpCfg.ID, instant) - // Pick the step set: 3Φ-only by default, but fall back to all - // allowed steps (which lets the driver hand the wallbox a 1Φ- - // eligible amperage) when the forecast says we won't see enough - // surplus to sustain 3Φ for the rest of the day. The lock is - // sticky: once we've gone 1Φ for the session we stay 1Φ to - // avoid cycling the contactor across the phase-mode boundary - // each time clouds shift. - steps := c.pickSurplusSteps(now, lpCfg) + if steps == nil { + steps = c.pickSurplusSteps(now, lpCfg) // A hold ended after chooseTick. + } minStep := smallestNonZero(steps) // Compatibility variables for the rest of the function — the @@ -2175,37 +2256,28 @@ afterNearTerm: // surplusLockedTo1P reports whether the surplus_only 1Φ lock is // currently active for the given loadpoint. Read-only accessor. -// resolvePhaseMode picks the phase_mode sent to the driver. -// A schedule wins, or a deadline charge is stuck on the 1Φ lock. -// Then the day lock, then the 30 min dwell, then the operator pin. -func resolvePhaseMode(operatorMode string, scheduleActive, surplusLocked1P, surplusOn bool, dwell string) string { +// resolvePhaseMode picks the phase_mode sent to the driver. An operator pin +// wins; a 3Φ-only charger never gets the surplus 1Φ lock. Unset and "auto" +// mean the same. A charger that cannot reach 6 A on three phases runs on +// one. Otherwise planned charging runs on three: one phase never helps the +// fuse, it puts all the current on one, caps the car near 3.7 kW, and an +// Easee pauses to switch and may stay there. Only surplus charging without +// a schedule may use one phase: the day lock, then the 30 min dwell, then +// auto. +func resolvePhaseMode(operatorMode string, threePhase, scheduleActive, surplusLocked1P, surplusOn bool, dwell string) string { switch { - case scheduleActive: - if operatorMode == "" { - return "auto" - } + case operatorMode != "" && operatorMode != "auto": return operatorMode + case !threePhase: + return "1p" + case scheduleActive || !surplusOn: + return "3p" case surplusLocked1P: - // A 3Φ-only charger (operator pinned "3p") physically cannot - // trickle at 1Φ, so the surplus 1Φ lock must never override it — - // defends against a stale lock left from a previous "auto"/"1p" - // config. pickSurplusSteps also refuses to set the lock for a - // "3p" loadpoint, so this is belt-and-braces. - if operatorMode == "3p" { - return "3p" - } return "1p" - case surplusOn && (operatorMode == "" || operatorMode == "auto"): - if dwell != "" { - return dwell - } - if operatorMode == "" { - return "auto" - } - return operatorMode - default: - return operatorMode + case dwell != "": + return dwell } + return "auto" } func (c *Controller) surplusLockedTo1P(id string) bool { @@ -2435,8 +2507,19 @@ func (c *Controller) computeCommandForBudget(now time.Time, lpCfg Config, curren } remainingWh := budgetWh - alreadyWh wantW := EnergyBudgetToPowerW(remainingWh, remainingS) + // A one-phase car takes a three-phase order's current per phase on its + // one phase. The plan counts what it takes; the order names that current + // on all three, within the charger's maximum. + onePhaseCar := false + if st, known := c.manager.State(lpCfg.ID); known && st.CarPhases == 1 && c.siteFuse().Phases() == 3 && + fusePhase(c.tickChoices[lpCfg.ID].phaseMode) == "3p" { + onePhaseCar, wantW = true, wantW*3 + } maxW := lpCfg.MaxChargeW if ceiling, ok := d.LoadpointMaxPowerW[lpCfg.ID]; ok { + if onePhaseCar { + ceiling *= 3 + } maxW = min(maxW, ceiling) if maxW <= 0 || remainingWh <= 1e-6 || remainingS <= 0 { return 0, true, false diff --git a/go/internal/loadpoint/controller_bat_soc_unlock_test.go b/go/internal/loadpoint/controller_bat_soc_unlock_test.go index 6c9290983..927026595 100644 --- a/go/internal/loadpoint/controller_bat_soc_unlock_test.go +++ b/go/internal/loadpoint/controller_bat_soc_unlock_test.go @@ -289,10 +289,10 @@ func TestTickScheduleUnlockAddsSurplusOverEmptyPlan(t *testing.T) { if sent.power != 4140 || w != 4140 || r != "pv_surplus" { t.Fatalf("want 4140 W / pv_surplus; sent %.0f W, recorded (%.0f W, %q)", sent.power, w, r) } - // A scheduled charge keeps the schedule's phase behaviour: "auto" for - // an unset operator mode, never the surplus 1Φ lock. - if sent.phaseMode != "auto" { - t.Errorf("phase_mode = %q, want auto while a schedule is active", sent.phaseMode) + // A scheduled charge keeps the schedule's phase behaviour: three phases + // for an unset operator mode, never the surplus 1Φ lock. + if sent.phaseMode != "3p" { + t.Errorf("phase_mode = %q, want 3p while a schedule is active", sent.phaseMode) } } diff --git a/go/internal/loadpoint/controller_cap_amps_test.go b/go/internal/loadpoint/controller_cap_amps_test.go new file mode 100644 index 000000000..de8d7baf9 --- /dev/null +++ b/go/internal/loadpoint/controller_cap_amps_test.go @@ -0,0 +1,156 @@ +package loadpoint + +import ( + "context" + "math" + "testing" + "time" +) + +// cap_amps_per_phase tells a driver how far an order may rise on its own +// phases. easee_cloud starts a 6 A order at 7 A only when it says 7 A fits; +// max_amps_per_phase is the fuse rating and says nothing about the room. +func TestCommandCarriesCapAmpsPerPhase(t *testing.T) { + for _, tc := range []struct { + name string + mode string + planW float64 + totalW float64 // control.EVFuseBudget + roomA float64 // control.EVFusePhaseRoomA; NaN: no phase currents + hold *ManualHold + lastW float64 // last tick's 3p order, which roomA already counts + wantW float64 + wantMaxA float64 + wantCapA float64 + wantNoCap bool + }{ + // The room on the busiest phase takes a 16 A plan to 6 A. + {name: "room limits the order to 6 A", mode: "3p", planW: 11040, totalW: 17250, roomA: 6.4, + wantW: 4140, wantMaxA: 25, wantCapA: 6.4}, + {name: "room for exactly 7 A", mode: "3p", planW: 4140, totalW: 17250, roomA: 7, + wantW: 4140, wantMaxA: 25, wantCapA: 7}, + // A 6 A order on its way is in the room already; it counts once. + {name: "last 6 A order not shown yet", mode: "3p", planW: 4140, totalW: 4416, roomA: .4, lastW: 4140, + wantW: 4140, wantMaxA: 25, wantCapA: 6.4}, + // The plan wants 6 A; the cap is the room, not the plan. + {name: "room ample", mode: "3p", planW: 4140, totalW: 17250, roomA: 15, + wantW: 4140, wantMaxA: 25, wantCapA: 15}, + {name: "room past the charger's maximum", mode: "3p", planW: 4140, totalW: 17250, roomA: 20, + wantW: 4140, wantMaxA: 25, wantCapA: 16}, + // The hour's import limit or the fuse in total import, in watts. + {name: "total budget limits", mode: "3p", planW: 4140, totalW: 4500, roomA: 15, + wantW: 4140, wantMaxA: 25, wantCapA: 6.52}, + {name: "no phase currents", mode: "3p", planW: 4140, totalW: 5175, roomA: math.NaN(), + wantW: 4140, wantMaxA: 25, wantCapA: 7.5}, + // A hold's current limit is the order's max_amps_per_phase; its + // power is what the owner wants, as a plan's is. + {name: "manual hold with a current limit", mode: "3p", totalW: 17250, roomA: 15, + hold: &ManualHold{PowerW: 4140, PhaseMode: "3p", MaxAmpsPerPhase: 6.5, Persistent: true}, + wantW: 4140, wantMaxA: 6.5, wantCapA: 6.5}, + {name: "manual hold", mode: "3p", totalW: 17250, roomA: 15, + hold: &ManualHold{PowerW: 4140, PhaseMode: "3p", Persistent: true}, + wantW: 4140, wantMaxA: 25, wantCapA: 15}, + // One phase: watts over one phase's voltage. + {name: "one phase, total budget", mode: "1p", planW: 3680, totalW: 1500, roomA: 20, + wantW: 1380, wantMaxA: 25, wantCapA: 6.52}, + {name: "one phase, room", mode: "1p", planW: 1380, totalW: 17250, roomA: 7.5, + wantW: 1380, wantMaxA: 25, wantCapA: 7.5}, + // A 0 W order names no current to raise. + {name: "no room", mode: "3p", planW: 4140, totalW: 17250, roomA: 5, + wantW: 0, wantMaxA: 25, wantNoCap: true}, + } { + t.Run(tc.name, func(t *testing.T) { + now := time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 11040, + AllowedStepsW: threePhaseSteps, PhaseMode: tc.mode} + d := &Directive{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), + LoadpointEnergyWh: map[string]float64{cfg.ID: tc.planW / 4}, LoadpointMaxPowerW: map[string]float64{cfg.ID: tc.planW}} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, d, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.SetFuseEVMax(func() (float64, bool) { return tc.totalW, true }) + c.SetFusePhaseRoomA(func() (float64, bool) { return tc.roomA, !math.IsNaN(tc.roomA) }) + if tc.hold != nil { + c.SetManualHold(cfg.ID, *tc.hold) + } + if tc.lastW > 0 { + c.manager.SetCommanded(cfg.ID, tc.lastW, "plan") + c.manager.setCommandedPhases(cfg.ID, 3) + } + c.Tick(context.Background(), now) + got, ok := lastSetCurrent(sender.calls) + if !ok || math.Abs(got.power-tc.wantW) > .01 || got.maxAmpsPerPhase != tc.wantMaxA { + t.Fatalf("order %+v, want %.0f W with max_amps_per_phase %.1f", got, tc.wantW, tc.wantMaxA) + } + if tc.wantNoCap { + if got.capAmpsPerPhase != nil { + t.Fatalf("cap_amps_per_phase = %.2f, want none", *got.capAmpsPerPhase) + } + return + } + if got.capAmpsPerPhase == nil { + t.Fatal("cap_amps_per_phase missing") + } + capA := *got.capAmpsPerPhase + if math.Abs(capA-tc.wantCapA) > 1e-9 { + t.Fatalf("cap_amps_per_phase = %v, want %v", capA, tc.wantCapA) + } + if capA > got.maxAmpsPerPhase { + t.Fatalf("cap_amps_per_phase %.2f above max_amps_per_phase %.2f", capA, got.maxAmpsPerPhase) + } + }) + } +} + +// Without a fuse budget Core knows no cap, and an older Core sends none: +// a driver must read a missing field as "no room above the order". +func TestCommandOmitsCapAmpsWithoutFuseBudget(t *testing.T) { + now := time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 11040, + AllowedStepsW: threePhaseSteps, PhaseMode: "3p"} + d := &Directive{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), + LoadpointEnergyWh: map[string]float64{cfg.ID: 4140 / 4}, LoadpointMaxPowerW: map[string]float64{cfg.ID: 4140}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, d, map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}}, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.Tick(context.Background(), now) + got, ok := lastSetCurrent(sender.calls) + if !ok || got.power != 4140 { + t.Fatalf("order %+v, want 4140 W", got) + } + if got.capAmpsPerPhase != nil { + t.Fatalf("cap_amps_per_phase = %.2f without a fuse budget", *got.capAmpsPerPhase) + } +} + +// Two chargers never get the same room. Each keeps its own share, and the +// room neither took goes to the first in the split's order only. +func TestCapAmpsPerPhaseSplitsRoomBetweenChargers(t *testing.T) { + now := time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC) + a := Config{ID: "a", DriverName: "easee-a", MinChargeW: 1380, MaxChargeW: 11040, + AllowedStepsW: threePhaseSteps, PhaseMode: "3p"} + b := a + b.ID, b.DriverName = "b", "easee-b" + d := &Directive{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), + LoadpointEnergyWh: map[string]float64{a.ID: 4140 / 4, b.ID: 4140 / 4}, + LoadpointMaxPowerW: map[string]float64{a.ID: 4140, b.ID: 4140}} + samples := map[string]EVSample{a.DriverName: {Connected: true, RequestActive: true}, b.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{a, b}, d, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.SetFuseEVMax(func() (float64, bool) { return 17250, true }) + c.SetFusePhaseRoomA(func() (float64, bool) { return 13, true }) + c.Tick(context.Background(), now) + caps := map[string]float64{} + for _, sent := range sender.calls { + if sent.action != "ev_set_current" || sent.power != 4140 || sent.capAmpsPerPhase == nil { + t.Fatalf("order %+v, want 4140 W with a cap", sent) + } + caps[sent.driver] = *sent.capAmpsPerPhase + } + // 13 A of room: 6 A each, and the 1 A left goes to a. + if caps[a.DriverName] != 7 || caps[b.DriverName] != 6 { + t.Fatalf("caps %v, want a 7 A and b 6 A", caps) + } +} diff --git a/go/internal/loadpoint/controller_commanded_reason_test.go b/go/internal/loadpoint/controller_commanded_reason_test.go index 57084ea28..035597cd2 100644 --- a/go/internal/loadpoint/controller_commanded_reason_test.go +++ b/go/internal/loadpoint/controller_commanded_reason_test.go @@ -101,6 +101,27 @@ func TestCommandedReasonFuseCapZero(t *testing.T) { } } +// A cap the hour's import ceiling sets carries its own name, never the +// fuse's, also when it overrides a hold. +func TestCommandedReasonImportLimit(t *testing.T) { + base := time.Date(2026, 8, 30, 12, 0, 0, 0, time.UTC) + for _, capW := range []float64{0, 6210} { + sender := &fakeSender{} + cfg := reasonLoadpoint() + samples := map[string]EVSample{cfg.DriverName: {Connected: true, PowerW: 0, RequestActive: true}} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetManualHold(cfg.ID, ManualHold{PowerW: 11040, PhaseMode: "3p", Persistent: true}) + c.SetFuseEVMax(func() (float64, bool) { return capW, true }) + c.SetFuseLimitReason(func() string { return "import_limit" }) + + c.Tick(context.Background(), base) + + if w, r := commandedReason(t, c, cfg.ID); w != capW || r != "import_limit" { + t.Errorf("cap %.0f W from the hour's ceiling: got (%.0f, %q), want (%.0f, import_limit)", capW, w, r, capW) + } + } +} + func TestCommandedReasonFuseLimitOverridesManualHold(t *testing.T) { base := time.Date(2026, 8, 30, 12, 0, 0, 0, time.UTC) sender := &fakeSender{} diff --git a/go/internal/loadpoint/controller_fuse_budget.go b/go/internal/loadpoint/controller_fuse_budget.go index 93ef1185e..938db18e0 100644 --- a/go/internal/loadpoint/controller_fuse_budget.go +++ b/go/internal/loadpoint/controller_fuse_budget.go @@ -6,52 +6,51 @@ import ( "time" ) +// fuseRoom is what one charger may use this tick, whatever it wants. What no +// charger took goes to the first charger with a share, so two chargers are +// never given the same room. capAmpsPerPhase reports it to the driver. +type fuseRoom struct { + w float64 // share of the EV cap, in watts + phaseA float64 // room on the busiest phase above its last command; with phaseKnown + phaseKnown bool // the site meter reports phase currents +} + type fuseRequest struct { cfg Config deadline time.Time wantW float64 - baseW float64 goal bool } -// freezeFuseBudget divides one cap for the whole tick, before any command. -// A later callback or goal update cannot give a second car the first car's -// already offered share. Metered reductions earn new room only next tick. +// freezeFuseBudget splits the EV fuse cap between cars for the whole tick, +// before any command, in priority order. A later callback or goal update +// cannot give a second car the first car's share. The cap is the only fuse +// limit here: control.EVFuseBudget reads the site meter for the phases that +// FuseConfigs names. func (c *Controller) freezeFuseBudget(now time.Time, samples map[string]EVSample) { - c.tickFuseCaps = nil - c.tickFusePhases = 0 + c.tickFuseCaps, c.tickFuseRooms, c.tickFuseReason = nil, nil, "" if c.fuseEVMax == nil { - clear(c.fuseResumeTicks) return } total, ok := c.fuseEVMax() if !ok || total < 0 { - clear(c.fuseResumeTicks) return } + c.tickFuseReason = c.capReason() if !finite(total) { total = 0 } d, _ := c.currentDirective(now) var requests []fuseRequest - var baseTotal float64 for _, cfg := range c.manager.Configs() { sample := samples[cfg.DriverName] st, _ := c.manager.State(cfg.ID) - st.PluggedIn = sample.Connected && !sample.ConnectionUnknown - request := fuseRequest{cfg: c.fuseConfig(now, cfg), deadline: st.TargetTime} - // The meter may still show a car's highest recent offer on the same - // phases. Up to that is not an increase; the car's own readings, - // late or lone, never count. - if peak := c.offerPeaks[cfg.ID]; st.PluggedIn && peak.phase == c.phaseModeForFuse(now, request.cfg) { - request.baseW = peak.w - } - baseTotal += request.baseW - if st.PluggedIn && c.driverCanDispatch(cfg.DriverName) { + request := fuseRequest{cfg: c.fuseConfig(cfg, c.tickChoices[cfg.ID].phaseMode), deadline: st.TargetTime} + if sample.Connected && !sample.ConnectionUnknown && c.driverCanDispatch(cfg.DriverName) { if hold, held := c.GetManualHold(cfg.ID, now); held { request.wantW = clampManualPower(cfg, hold, c.siteFuse()) } else { - request.wantW, _, _ = c.computeCommandForBudget(now, cfg, sample.PowerW, false) + request.wantW, _, _ = c.computeCommandForBudget(now, request.cfg, sample.PowerW, false) } request.goal = request.wantW > 0 && !cfg.SurplusOnly if _, held := c.GetManualHold(cfg.ID, now); !held { @@ -81,129 +80,173 @@ func (c *Controller) freezeFuseBudget(now time.Time, samples map[string]EVSample } return a.cfg.ID < b.cfg.ID }) - c.tickFuseCaps = make(map[string]float64, len(requests)) - remaining, increase := total, math.Max(0, total-baseTotal) - if phaseIncrease, known := c.sharedPhaseIncreaseW(now, requests); known { - increase = math.Max(0, math.Min(increase, phaseIncrease)) + roomA, perPhase := 0.0, false + if c.fusePhaseRoomA != nil { + roomA, perPhase = c.fusePhaseRoomA() + perPhase = perPhase && finite(roomA) } + c.tickFuseCaps = make(map[string]float64, len(requests)) + c.tickFuseRooms = make(map[string]fuseRoom, len(requests)) + remaining := total for _, request := range requests { - target := floorChargeW(math.Min(request.wantW, remaining), request.cfg.MinChargeW, request.cfg.MaxChargeW, request.cfg.AllowedStepsW) - remaining -= target - // Another car's curtailment is spent only once it has settled. - want := math.Min(target, request.baseW+increase) + want := math.Min(request.wantW, remaining) + mode := c.tickChoices[request.cfg.ID].phaseMode + split := c.commandSplitW(now, request.cfg) + if perPhase { + want = math.Min(want, c.phaseRoomCapW(request.cfg.ID, mode, want, split, roomA)) + } cap := floorChargeW(want, request.cfg.MinChargeW, request.cfg.MaxChargeW, request.cfg.AllowedStepsW) - cap = c.recoverFuseCap(request.cfg.ID, request.wantW, cap) c.tickFuseCaps[request.cfg.ID] = cap - increase -= math.Max(0, cap-request.baseW) + remaining -= cap + room := fuseRoom{w: cap, phaseKnown: perPhase} + if perPhase { + // A cut frees room only once the meter shows it. + room.phaseA = math.Max(0, c.phaseRiseA(request.cfg.ID, mode, cap, split)) + roomA -= room.phaseA + } + c.tickFuseRooms[request.cfg.ID] = room + } + for _, request := range requests { + room := c.tickFuseRooms[request.cfg.ID] + if room.w <= 0 { + continue + } + room.w += math.Max(0, remaining) + room.phaseA += math.Max(0, roomA) + c.tickFuseRooms[request.cfg.ID] = room + break } } -type offerPeak struct { - w float64 - phase string - at time.Time +// capAmpsPerPhase is the most current per phase Core allows this charger on +// the phases its positive order names: its fuse room after the split between +// chargers, the import limit, max_amps_per_phase and the charger's maximum. +// It is not limited by what the plan or a hold wants, so it may exceed the +// order. Unknown without a frozen budget or a voltage. +func (c *Controller) capAmpsPerPhase(cfg Config, cmd map[string]any) (float64, bool) { + room, ok := c.tickFuseRooms[cfg.ID] + w, _ := cmd["power_w"].(float64) + v, _ := cmd["voltage"].(float64) + if !ok || !finite(room.w) || !finite(w) || w <= 0 || !finite(v) || v <= 0 { + return 0, false + } + mode, _ := cmd["phase_mode"].(string) + split, _ := cmd["phase_split_w"].(float64) + phases := c.commandPhases(mode, w, split) + capA := room.w / (v * float64(phases)) + if room.phaseKnown { + capA = math.Min(capA, c.creditA(cfg.ID, phases)+room.phaseA) + } + if cfg.MaxChargeW > 0 { + capA = math.Min(capA, cfg.MaxChargeW/(v*float64(phases))) + } + if !finite(capA) { + return 0, false + } + // Two decimals, rounded down; the nudge absorbs float error at a step. + capA = math.Floor(math.Max(0, capA)*100+1e-6) / 100 + if limit, known := cmd["max_amps_per_phase"].(float64); known && finite(limit) && limit > 0 { + capA = math.Min(capA, limit) + } + return capA, true } -// recordOfferPeak keeps the highest offer the meter may still show. A new -// phase mode starts over: the same watts on other phases are other currents. -func (c *Controller) recordOfferPeak(now time.Time, id string, w float64, cmd map[string]any) string { - phase, _ := cmd["phase_mode"].(string) - if phase == "" { - phase = "3p" // The driver's locked default, as in phaseModeForFuse. - } - if c.offerPeaks == nil { - c.offerPeaks = map[string]offerPeak{} +// commandPhases is where a command of w lands: 1 or 3 phases, as the +// driver resolves its phase_mode and split. A 1-phase site has one. +func (c *Controller) commandPhases(mode string, w, split float64) int { + if c.siteFuse().Phases() == 1 { + return 1 } - if peak := c.offerPeaks[id]; peak.phase != phase || w >= peak.w || now.Sub(peak.at) >= CommandSettle { - c.offerPeaks[id] = offerPeak{w: w, phase: phase, at: now} + if split <= 0 { + split = defaultPhaseSplitW } - return phase + return PhaseFor(mode, w, split) } -func (c *Controller) fuseConfig(now time.Time, cfg Config) Config { - if hold, held := c.GetManualHold(cfg.ID, now); held { - cfg.MinChargeW = manualMinChargeW(cfg, hold, c.siteFuse()) +// commandSplitW is the 1p/3p split this tick's command names. +func (c *Controller) commandSplitW(now time.Time, cfg Config) float64 { + if hold, held := c.GetManualHold(cfg.ID, now); held && hold.PhaseSplitW > 0 { + return hold.PhaseSplitW } - return cfg + return cfg.PhaseSplitW } -func (c *Controller) recoverFuseCap(id string, wantW, capW float64) float64 { - if wantW <= 0 { - delete(c.fuseResumeTicks, id) - return capW - } - if capW <= 0 { - if c.fuseResumeTicks == nil { - c.fuseResumeTicks = map[string]int{} - } - c.fuseResumeTicks[id] = 0 +// phaseRiseA is how much a command of w adds to the busiest phase over the +// loadpoint's last command. A single-phase command may land on any phase, +// so the last command's current comes off only where both use the same +// phases or the last one used all three. +func (c *Controller) phaseRiseA(id, mode string, w, split float64) float64 { + site := c.siteFuse() + if w <= 0 || site.Voltage <= 0 { return 0 } - if ready, paused := c.fuseResumeTicks[id]; paused { - ready++ - if ready < fuseResumeSafeTicks { - c.fuseResumeTicks[id] = ready - return 0 - } - delete(c.fuseResumeTicks, id) - } - return capW + phases := c.commandPhases(mode, w, split) + return w/(site.Voltage*float64(phases)) - c.creditA(id, phases) } -// A new single-phase offer can land on the busiest phase, so an increase -// needs measured room there. Cutting an overage is the shared budget's job. -func (c *Controller) sharedPhaseIncreaseW(now time.Time, requests []fuseRequest) (float64, bool) { - if c.perPhaseMeterAmps == nil { - return 0, false +func (c *Controller) creditA(id string, phases int) float64 { + st, _ := c.manager.State(id) + site := c.siteFuse() + if !st.CommandedKnown || st.CommandedW <= 0 || site.Voltage <= 0 || (st.CommandedPhases != 3 && st.CommandedPhases != phases) { + return 0 } - l1, l2, l3, ok := c.perPhaseMeterAmps() + return st.CommandedW / (site.Voltage * float64(min(st.CommandedPhases, site.Phases()))) +} + +// phaseRoomCapW is the most a command may offer when it can add at most +// roomA to the busiest phase. An auto command below the split lands on one +// phase. Room limits rises only: keeping the last command is never a cut, +// since the meter or the pending count already holds it, and only a +// measured overage cuts (EVFuseBudget). The same watts on other phases are +// a rise on some of them, so the last command is kept only when the order +// it allows lands on the same phases. +func (c *Controller) phaseRoomCapW(id, mode string, wantW, split, roomA float64) float64 { site := c.siteFuse() - if !ok || !finite(site.MaxAmps) || !finite(site.Voltage) || site.MaxAmps <= 0 || site.Voltage <= 0 { - return 0, false + if site.Voltage <= 0 { + return wantW } - if !finite(l1) || !finite(l2) || !finite(l3) { - return 0, true + roomA = math.Max(0, roomA) + phases := c.commandPhases(mode, wantW, split) + capW := (c.creditA(id, phases) + roomA) * site.Voltage * float64(phases) + if phases == 3 && c.commandPhases(mode, capW, split) == 1 { + capW = math.Min(capW, (c.creditA(id, 1)+roomA)*site.Voltage) } - phaseCount := site.Phases() - for _, request := range requests { - if c.phaseModeForFuse(now, request.cfg) != "3p" { - phaseCount = 1 - } + if st, ok := c.manager.State(id); ok && st.CommandedKnown && st.CommandedPhases > 0 && st.CommandedW > capW && + c.commandPhases(mode, math.Min(wantW, st.CommandedW), split) == st.CommandedPhases { + capW = st.CommandedW } - c.tickFusePhases = phaseCount - worst := math.Max(math.Abs(l1), math.Max(math.Abs(l2), math.Abs(l3))) - return (site.MaxAmps - fusePhaseMarginA - worst) * site.Voltage * float64(phaseCount), true + return capW } -// Empty is the driver's locked-3p default unless the command branch selects -// auto or the surplus day lock selects 1p. Do not advance surplus hysteresis -// here: an enabled unlock may arm later in this tick, so retain its 1p bound. -func (c *Controller) phaseModeForFuse(now time.Time, cfg Config) string { - sched, _ := c.manager.GetSchedule(cfg.ID) - surplus := cfg.SurplusOnly - if !sched.HasTarget() && sched.SurplusUnlockBatSoC > 0 && c.batSoC != nil && - !(c.tickDirectiveActive && c.tickDirectiveOK && c.tickDirective.LoadpointFallback[cfg.ID]) { - surplus = true - } - mode := cfg.PhaseMode - if hold, held := c.GetManualHold(cfg.ID, now); held { - if hold.PhaseMode != "" { - mode = hold.PhaseMode +// FuseConfigs returns the loadpoints with the phase mode each command names +// this tick. control.EVFuseBudget measures room on those phases, so an auto +// or 1p offer never gets three phases of room. Outside a tick, and for a +// loadpoint that gets no command, the configured mode stands. +func (c *Controller) FuseConfigs() []Config { + configs := c.manager.Configs() + for i := range configs { + if mode := c.tickChoices[configs[i].ID].phaseMode; mode != "" { + configs[i].PhaseMode = mode } - if mode == "" && surplus { - mode = "auto" - } - } else { - // A surplus dwell can change while the offer is computed. Only a - // fixed 3p mode earns three phases of new headroom in this guard. - mode = resolvePhaseMode(mode, sched.HasTarget(), c.surplusLockedTo1P(cfg.ID), surplus, "") } + return configs +} + +// fusePhase names the phases a command uses. Empty is the driver's locked 3p. +func fusePhase(mode string) string { if mode == "" { return "3p" } return mode } +// fuseConfig snaps a fuse cap to the smallest legal step on the phases the +// command names. A 3p label on a 1p-sized offer is below 6 A and never charges. +func (c *Controller) fuseConfig(cfg Config, phaseMode string) Config { + cfg.MinChargeW = phaseMinChargeW(cfg, phaseMode, c.siteFuse()) + return cfg +} + func (c *Controller) currentDirective(now time.Time) (Directive, bool) { if c.tickDirectiveActive { return c.tickDirective, c.tickDirectiveOK diff --git a/go/internal/loadpoint/controller_fuse_phase_test.go b/go/internal/loadpoint/controller_fuse_phase_test.go index 1df0ce47a..f5ad4fd59 100644 --- a/go/internal/loadpoint/controller_fuse_phase_test.go +++ b/go/internal/loadpoint/controller_fuse_phase_test.go @@ -13,10 +13,15 @@ func TestSharedFuseGuardUsesCommandPhase(t *testing.T) { manual, schedule, surplus bool wantW, worstA, commandW float64 }{ - {name: "legacy_plan", wantW: 13110, worstA: 5, commandW: 13110, demandMode: "3p"}, - {name: "legacy_manual", manual: true, wantW: 13110, worstA: 5, commandW: 13110, demandMode: "3p"}, - {name: "legacy_manual_over_schedule", manual: true, schedule: true, wantW: 13110, worstA: 5, commandW: 13110, demandMode: "3p"}, - {name: "scheduled_auto", schedule: true, wantW: 2760, worstA: 20, commandMode: "auto", demandMode: "auto"}, + // An unset mode is auto: three phases, named in the order. + {name: "legacy_plan", wantW: 13110, worstA: 5, commandW: 13110, commandMode: "3p", demandMode: "3p"}, + {name: "legacy_manual", manual: true, wantW: 13110, worstA: 5, commandW: 13110, commandMode: "3p", demandMode: "3p"}, + {name: "legacy_manual_over_schedule", manual: true, schedule: true, wantW: 13110, worstA: 5, commandW: 13110, commandMode: "3p", demandMode: "3p"}, + {name: "scheduled_1p", configMode: "1p", schedule: true, wantW: 2760, worstA: 20, commandMode: "1p", demandMode: "1p"}, + // Planned charging on auto runs on three phases: the 3Φ minimum does + // not fit 4 A of room, and no one-phase order takes its place. + {name: "scheduled_auto_runs_3p", configMode: "auto", schedule: true, wantW: 4140, worstA: 20, commandMode: "3p", demandMode: "3p"}, + {name: "scheduled_auto_room", configMode: "auto", schedule: true, wantW: 13110, worstA: 5, commandW: 13110, commandMode: "3p", demandMode: "3p"}, {name: "surplus_manual_auto", manual: true, surplus: true, wantW: 2760, worstA: 20, commandMode: "auto", demandMode: "auto"}, {name: "manual_1p_over_3p", configMode: "3p", manual: true, holdMode: "1p", wantW: 2760, worstA: 20, commandMode: "1p", demandMode: "1p"}, {name: "manual_3p_over_auto", configMode: "auto", manual: true, holdMode: "3p", wantW: 13110, worstA: 5, commandW: 13110, commandMode: "3p", demandMode: "3p"}, @@ -31,8 +36,7 @@ func TestSharedFuseGuardUsesCommandPhase(t *testing.T) { sender := &fakeSender{} c := newTestController(t, []Config{cfg}, d, samples, sender) c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return tc.worstA, tc.worstA, tc.worstA, true }) - c.SetFuseEVMax(func() (float64, bool) { return 20000, true }) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return tc.worstA })) if tc.schedule { c.manager.SetSchedule(cfg.ID, Schedule{SoC: .8, TimeOfDayMinUTC: 13 * 60, Recurring: true}) } @@ -60,15 +64,14 @@ func TestAutomaticManualReleasePrecedesSharedPhaseBudget(t *testing.T) { t.Run(release, func(t *testing.T) { now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) cfg := Config{ID: "garage", DriverName: "charger", MinChargeW: 1380, MaxChargeW: 13110, - AllowedStepsW: []float64{0, 1380, 2760, 4140, 13110}, VehicleCapacityWh: 60000, PluginSoC: .8} + AllowedStepsW: []float64{0, 1380, 2760, 4140, 13110}, VehicleCapacityWh: 60000, PluginSoC: .8, PhaseMode: "1p"} d := &Directive{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), LoadpointEnergyWh: map[string]float64{cfg.ID: 690}, LoadpointMaxPowerW: map[string]float64{cfg.ID: 2760}} samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: release != "idle"}} sender := &fakeSender{} c := newTestController(t, []Config{cfg}, d, samples, sender) c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return 20, 20, 20, true }) - c.SetFuseEVMax(func() (float64, bool) { return 2070, true }) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 20 })) c.manager.SetSchedule(cfg.ID, Schedule{SoC: .9, TimeOfDayMinUTC: 13 * 60, Recurring: true}) hold := ManualHold{PowerW: 4140, PhaseMode: "3p", Persistent: true} if release == "soc" { @@ -86,7 +89,7 @@ func TestAutomaticManualReleasePrecedesSharedPhaseBudget(t *testing.T) { t.Fatal("completed manual hold was not released") } command, known := lastSetCurrent(sender.calls) - if !known || command.phaseMode != "auto" || command.power != 0 { + if !known || command.phaseMode != "1p" || command.power != 0 { t.Fatalf("automatic order reused the earlier manual 3p budget: %+v", command) } }) @@ -98,10 +101,11 @@ func TestManualPhaseChangeAfterSharedBudgetWaitsForNextTick(t *testing.T) { name, beforeMode, afterMode string beforeW, afterW, worstA, commandW float64 }{ - {"3p_to_1p", "3p", "1p", 4140, 1380, 20, 0}, - {"3p_to_auto", "3p", "auto", 4140, 1380, 20, 0}, - {"new_stop", "3p", "3p", 4140, 0, 20, 0}, - {"1p_to_3p", "1p", "3p", 1380, 4140, 17, 1380}, + {"3p_to_1p", "3p", "1p", 4140, 1380, 17, 0}, + {"3p_to_auto", "3p", "auto", 4140, 1380, 17, 0}, + {"new_stop", "3p", "3p", 4140, 0, 17, 0}, + // 1,380 W on three phases is 2 A each: no legal 3p step. + {"1p_to_3p", "1p", "3p", 1380, 4140, 17, 0}, } { t.Run(tc.name, func(t *testing.T) { now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) @@ -114,9 +118,7 @@ func TestManualPhaseChangeAfterSharedBudgetWaitsForNextTick(t *testing.T) { sender := &fakeSender{} c := newTestController(t, []Config{a, b}, d, samples, sender) c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - worstA := tc.worstA - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return worstA, worstA, worstA, true }) - c.SetFuseEVMax(func() (float64, bool) { return 2070, true }) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return tc.worstA })) c.SetManualHold(a.ID, ManualHold{PhaseMode: "3p", Persistent: true}) c.SetManualHold(b.ID, ManualHold{PowerW: tc.beforeW, PhaseMode: tc.beforeMode, Persistent: true}) c.send = func(ctx context.Context, driver string, payload []byte) error { @@ -137,7 +139,6 @@ func TestManualPhaseChangeAfterSharedBudgetWaitsForNextTick(t *testing.T) { } } if tc.beforeMode == "3p" && tc.afterW > 0 { - worstA = 17 c.Tick(context.Background(), now.Add(5*time.Second)) command, known = lastSetCurrent(sender.calls) if !known || command.power != 1380 || command.phaseMode != tc.afterMode { @@ -147,29 +148,3 @@ func TestManualPhaseChangeAfterSharedBudgetWaitsForNextTick(t *testing.T) { }) } } - -// #1674, 05:49: the car drew 10,242 W at 15 A when one 0 W reading -// arrived. Keeping the 10,350 W offer is not an increase; the phase gate -// must not treat it as one and stop the charge at 2,449.5 W of room. -func TestLoneZeroReadingKeepsOfferThroughPhaseGate(t *testing.T) { - now := time.Date(2026, 10, 17, 5, 49, 0, 0, time.UTC) - cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 4140, MaxChargeW: 10350, PhaseMode: "3p", - AllowedStepsW: []float64{0, 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350}} - samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true, PowerW: 10242}} - sender := &fakeSender{} - c := newTestController(t, []Config{cfg}, nil, samples, sender) - c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - l1 := 5.0 // The car starts while L1 has room for it. - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return l1, 2.2, 1.0, true }) - c.SetFuseEVMax(func() (float64, bool) { return 16905, true }) - c.SetManualHold(cfg.ID, ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - c.TickWithDispatch(context.Background(), now, true) - l1 = 20.45 - for i, w := range []float64{10242, 0, 10242} { - samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: w} - c.TickWithDispatch(context.Background(), now.Add(time.Duration(5*(i+1))*time.Second), true) - if command, known := lastSetCurrent(sender.calls); !known || command.power != 10350 { - t.Fatalf("reading %.0f W changed the offer: %+v", w, command) - } - } -} diff --git a/go/internal/loadpoint/controller_fuse_recovery_test.go b/go/internal/loadpoint/controller_fuse_recovery_test.go index 3a9b4fdbb..2cec068c8 100644 --- a/go/internal/loadpoint/controller_fuse_recovery_test.go +++ b/go/internal/loadpoint/controller_fuse_recovery_test.go @@ -2,7 +2,6 @@ package loadpoint import ( "context" - "math" "testing" "time" ) @@ -14,58 +13,6 @@ func fuseRecoveryLoadpoint() Config { return cfg } -func TestFuseBudgetRecoversAfterTwoSafeTicks(t *testing.T) { - for _, manual := range []bool{false, true} { - for _, initial := range []struct { - name string - capW float64 - }{{"below_minimum", 3560}, {"zero", 0}, {"invalid", math.NaN()}} { - name := "scheduled/" + initial.name - if manual { - name = "manual/" + initial.name - } - t.Run(name, func(t *testing.T) { - now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) - cfg := fuseRecoveryLoadpoint() - d := &Directive{SlotStart: now, SlotEnd: now.Add(15 * time.Minute), LoadpointEnergyWh: map[string]float64{cfg.ID: 2587.5}, LoadpointMaxPowerW: map[string]float64{cfg.ID: 10350}} - samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true, PowerW: 10000}} - sender := &fakeSender{} - c := newTestController(t, []Config{cfg}, d, samples, sender) - c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - if manual { - c.SetManualHold(cfg.ID, ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - } - capW := initial.capW - c.SetFuseEVMax(func() (float64, bool) { return capW, true }) - check := func(at time.Time, allowed bool, want float64) { - t.Helper() - c.TickWithDispatch(context.Background(), at, allowed) - set, ok := lastSetCurrent(sender.calls) - if !ok || set.power != want { - t.Fatalf("at %v: command=%+v, want %.0f W", at.Sub(now), set, want) - } - } - check(now, true, 0) - capW = 13560 - check(now.Add(5*time.Second), true, 0) - check(now.Add(10*time.Second), true, 10350) - // A later loss of headroom must stop immediately and require - // two safe dispatch ticks again, never a five-minute lock. - capW = 3000 - check(now.Add(15*time.Second), true, 0) - capW = 13560 - check(now.Add(20*time.Second), true, 0) - check(now.Add(25*time.Second), false, 0) - check(now.Add(30*time.Second), true, 0) - check(now.Add(35*time.Second), true, 10350) - // A saved Stop remains a Stop after the fuse budget recovers. - c.SetManualHold(cfg.ID, ManualHold{Persistent: true}) - check(now.Add(40*time.Second), true, 0) - }) - } - } -} - func TestFirstConnectionIsObservedBeforeFuseBudget(t *testing.T) { now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) cfg := fuseRecoveryLoadpoint() @@ -90,33 +37,6 @@ func TestFirstConnectionIsObservedBeforeFuseBudget(t *testing.T) { } } -func TestSharedPhaseBudgetDoesNotIntegrateSameOverload(t *testing.T) { - now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) - cfg := fuseRecoveryLoadpoint() - samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true, PowerW: 8280}} - sender := &fakeSender{} - c := newTestController(t, []Config{cfg}, nil, samples, sender) - c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetManualHold(cfg.ID, ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - // Our own 12 A offer is established before the overload; a reading - // from the car is never what the guard credits. - worstA, otherA, budgetW := 12.0, 12.0, 8280.0 - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return worstA, otherA, otherA, true }) - c.SetFuseEVMax(func() (float64, bool) { return budgetW, true }) - c.Tick(context.Background(), now.Add(-5*time.Second)) - worstA, otherA, budgetW = 27, 20, 6210 - for tick := 0; tick < 12; tick++ { - c.Tick(context.Background(), now.Add(time.Duration(tick)*5*time.Second)) - set, ok := lastSetCurrent(sender.calls) - // Offered EV=12 A, worst site phase=27 A. A 9 A EV offer - // leaves that phase at 24 A, below the 25 A fuse. Re-reading - // the same sample cannot justify a second accumulated trim. - if !ok || set.power != 6210 { - t.Fatalf("tick %d: command=%+v, want stable legal 6210 W", tick, set) - } - } -} - func TestManualSinglePhaseUsesItsLegalMinimumInSharedFuseBudget(t *testing.T) { for _, phaseMode := range []string{"1p", "auto"} { t.Run(phaseMode, func(t *testing.T) { @@ -138,30 +58,6 @@ func TestManualSinglePhaseUsesItsLegalMinimumInSharedFuseBudget(t *testing.T) { } } -func TestNewConnectionDoesNotInheritFuseRecovery(t *testing.T) { - now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) - cfg := fuseRecoveryLoadpoint() - sender := &fakeSender{} - samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} - c := newTestController(t, []Config{cfg}, nil, samples, sender) - c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetManualHold(cfg.ID, ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - capW := 0.0 - c.SetFuseEVMax(func() (float64, bool) { return capW, true }) - c.Tick(context.Background(), now) - capW = 13560 - c.Tick(context.Background(), now.Add(5*time.Second)) - samples[cfg.DriverName] = EVSample{Connected: false} - c.Tick(context.Background(), now.Add(10*time.Second)) - samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true} - c.SetManualHold(cfg.ID, ManualHold{PowerW: 10350, PhaseMode: "3p", Persistent: true}) - c.Tick(context.Background(), now.Add(15*time.Second)) - set, ok := lastSetCurrent(sender.calls) - if !ok || set.power != 10350 { - t.Fatalf("new session inherited the previous session's recovery wait: %+v", set) - } -} - func TestManualPhaseOverrideKeepsSixAmpMinimum(t *testing.T) { now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC) cfg := fuseRecoveryLoadpoint() @@ -187,11 +83,10 @@ func TestUnspecifiedSurplusPhaseKeepsSinglePhaseFuseBound(t *testing.T) { sender := &fakeSender{} c := newTestController(t, []Config{cfg}, nil, samples, sender) c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return 20, 20, 20, true }) // A hold without a phase override inherits surplus mode's auto choice. // At2760 W that may be a12 A offer on the already20 A phase. c.SetManualHold(cfg.ID, ManualHold{PowerW: 2760, Persistent: true}) - c.SetFuseEVMax(func() (float64, bool) { return 2760, true }) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 20 })) c.Tick(context.Background(), now) set, ok := lastSetCurrent(sender.calls) if !ok || set.power != 0 { diff --git a/go/internal/loadpoint/controller_fusephase_test.go b/go/internal/loadpoint/controller_fusephase_test.go deleted file mode 100644 index b20e34e7d..000000000 --- a/go/internal/loadpoint/controller_fusephase_test.go +++ /dev/null @@ -1,21 +0,0 @@ -package loadpoint - -import ( - "testing" - "time" -) - -// The phase increase gate must read hot-reloaded fuse params, not the -// startup-only c.site: a worst phase with room at fuse=16 has none at 11. -func TestSiteFuseHotReloadAppliesToPhaseIncreaseGate(t *testing.T) { - c := &Controller{} - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return 13, 5, 5, true }) - c.SetSiteFuse(SiteFuse{MaxAmps: 16, Voltage: 230, PhaseCnt: 3}) - if got, known := c.sharedPhaseIncreaseW(time.Now(), nil); !known || got != 1380 { - t.Fatalf("fuse 16, worst 13 A: increase=%v known=%v, want 1380 W", got, known) - } - c.SetSiteFuse(SiteFuse{MaxAmps: 11, Voltage: 230, PhaseCnt: 3}) - if got, known := c.sharedPhaseIncreaseW(time.Now(), nil); !known || got >= 0 { - t.Fatalf("hot-reloaded fuse 11, worst 13 A: increase=%v known=%v, want none", got, known) - } -} diff --git a/go/internal/loadpoint/controller_hold_restart_test.go b/go/internal/loadpoint/controller_hold_restart_test.go new file mode 100644 index 000000000..b2035d25a --- /dev/null +++ b/go/internal/loadpoint/controller_hold_restart_test.go @@ -0,0 +1,120 @@ +package loadpoint + +import ( + "context" + "testing" + "time" +) + +// bootController is one Core process over a state.db that outlives it. +func bootController(t *testing.T, store *sessionMemory, samples map[string]EVSample) (*Controller, *fakeSender) { + t.Helper() + m := NewManager() + m.Load([]Config{holdLoadpoint()}) + m.SetSessionStore(store) + sender := &fakeSender{} + plan := PlanFunc(func(time.Time) (Directive, bool) { return Directive{}, false }) + tel := TelemetryFunc(func(driver string) (EVSample, bool) { s, ok := samples[driver]; return s, ok }) + c := NewController(m, plan, tel, sender.Send) + c.SetSiteFuse(SiteFuse{MaxAmps: 16, Voltage: 230, PhaseCnt: 3}) + c.SetManualHoldSaver(func(id string, h ManualHold, cleared bool) { + if err := m.PersistManualHold(id, h, cleared); err != nil { + t.Error(err) + } + }) + return c, sender +} + +func tickWants(t *testing.T, c *Controller, sender *fakeSender, now time.Time, power float64, unconfirmed bool) { + t.Helper() + c.Tick(context.Background(), now) + if last, ok := lastSetCurrent(sender.calls); !ok || last.power != power { + t.Fatalf("commands=%+v, want latest power %v", sender.calls, power) + } + if st, _ := c.manager.State("garage"); st.ManualRestoreUnconfirmed != unconfirmed { + t.Fatalf("manual_restore_unconfirmed = %v, want %v", st.ManualRestoreUnconfirmed, unconfirmed) + } +} + +// Charge now ends on Stop, unplug or a full car. A restart is none of them, +// even when the charger reports its identity before its session. +func TestChargeNowWaitsForSessionAfterRestart(t *testing.T) { + for _, reconnect := range []bool{false, true} { + store := &sessionMemory{data: map[string]string{}} + sample := EVSample{Connected: true, RequestActive: true, ConnectionGeneration: 1, DeviceID: "easee:A", SessionID: "session-1"} + samples := map[string]EVSample{"easee": sample} + c, sender := bootController(t, store, samples) + now := time.Now() + c.SetManualHold("garage", ManualHold{PowerW: 4140, Persistent: true}) + tickWants(t, c, sender, now, 4140, false) + + c, sender = bootController(t, store, samples) + sample.SessionID = "" + samples["easee"] = sample + tickWants(t, c, sender, now.Add(20*time.Second), 0, true) + if reconnect { + // A new socket is not a new car; only the session can tell. + sample.ConnectionGeneration = 2 + samples["easee"] = sample + tickWants(t, c, sender, now.Add(21*time.Second), 0, true) + } + sample.SessionID = "session-1" + samples["easee"] = sample + tickWants(t, c, sender, now.Add(22*time.Second), 4140, false) + } +} + +// Most chargers, OCPP included, never report a session ID. The same +// hardware still plugged in resumes at once; an unplug still ends it. +func TestChargeNowWithoutSessionIDResumesAfterRestart(t *testing.T) { + store := &sessionMemory{data: map[string]string{}} + sample := EVSample{Connected: true, RequestActive: true, ConnectionGeneration: 1, DeviceID: "rigco:OCPPSIM02"} + samples := map[string]EVSample{"easee": sample} + c, sender := bootController(t, store, samples) + now := time.Now() + c.SetManualHold("garage", ManualHold{PowerW: 4140, Persistent: true}) + tickWants(t, c, sender, now, 4140, false) + + // After a restart the charger connects before it reports its cable. + samples["easee"] = EVSample{ConnectionUnknown: true, ConnectionGeneration: 1} + c, sender = bootController(t, store, samples) + c.Tick(context.Background(), now.Add(20*time.Second)) + samples["easee"] = sample + tickWants(t, c, sender, now.Add(21*time.Second), 4140, false) + + sample.Connected = false + samples["easee"] = sample + c.Tick(context.Background(), now.Add(22*time.Second)) + sample.Connected = true + samples["easee"] = sample + tickWants(t, c, sender, now.Add(23*time.Second), 0, false) + if h, ok := c.GetManualHold("garage", now); ok { + t.Fatalf("the next car inherited Charge now: %+v", h) + } + c, sender = bootController(t, store, samples) + tickWants(t, c, sender, now.Add(30*time.Second), 0, false) + if h, ok := c.GetManualHold("garage", now); ok { + t.Fatalf("Charge now returned after unplug and restart: %+v", h) + } +} + +// A different session after a restart means the car changed while Core +// was down. It must not inherit the earlier Charge now. +func TestChargeNowDoesNotFollowAnotherSessionAfterRestart(t *testing.T) { + store := &sessionMemory{data: map[string]string{}} + sample := EVSample{Connected: true, RequestActive: true, DeviceID: "easee:A", SessionID: "session-1"} + samples := map[string]EVSample{"easee": sample} + c, sender := bootController(t, store, samples) + now := time.Now() + c.SetManualHold("garage", ManualHold{PowerW: 4140, Persistent: true}) + tickWants(t, c, sender, now, 4140, false) + + c, sender = bootController(t, store, samples) + sample.SessionID = "" + samples["easee"] = sample + tickWants(t, c, sender, now.Add(20*time.Second), 0, true) + sample.SessionID = "session-2" + samples["easee"] = sample + tickWants(t, c, sender, now.Add(21*time.Second), 0, true) + tickWants(t, c, sender, now.Add(22*time.Second), 0, true) +} diff --git a/go/internal/loadpoint/controller_hold_restore.go b/go/internal/loadpoint/controller_hold_restore.go index 04ac994c5..45299c329 100644 --- a/go/internal/loadpoint/controller_hold_restore.go +++ b/go/internal/loadpoint/controller_hold_restore.go @@ -61,8 +61,11 @@ func (c *Controller) restoreManualHoldForSession(id string) { } // A Start before the first hardware reading binds here. Once bound, a // different hardware/session/config generation must not inherit its power. + // A restore still waiting for proof has no power to inherit; the saved + // record is checked against the next identity instead of overwritten. c.manualBindings[id] = current - if changed { + restored, seen := c.manualRestored[id] + if pending := seen && !restored; changed && !pending { if hold, held := c.holds[id]; held { if hold.PowerW > 0 { c.holds[id] = ManualHold{Persistent: true} diff --git a/go/internal/loadpoint/controller_manual_hold_test.go b/go/internal/loadpoint/controller_manual_hold_test.go index c706e04f5..bc57adfcc 100644 --- a/go/internal/loadpoint/controller_manual_hold_test.go +++ b/go/internal/loadpoint/controller_manual_hold_test.go @@ -110,12 +110,13 @@ func TestManualHoldExpiresFallsBackToPlan(t *testing.T) { cmd := sender.calls[0] // Should have followed the plan, not the hold. Plan allocates // 6 kWh / 1 h = 6000 W, snapped to nearest step 6210 W (3Φ-flavored - // budget). And phase_mode comes from lpCfg ("auto"), not hold ("1p"). + // budget). And phase_mode comes from lpCfg ("auto", which planned + // charging sends as "3p"), not hold ("1p"). if cmd.power != 6210 { t.Errorf("power = %.0f, want 6210 (plan path), hold should have expired", cmd.power) } - if cmd.phaseMode != "auto" { - t.Errorf("phase_mode = %q, want \"auto\" (lpCfg path), hold should have expired", cmd.phaseMode) + if cmd.phaseMode != "3p" { + t.Errorf("phase_mode = %q, want \"3p\" (lpCfg path), hold should have expired", cmd.phaseMode) } // Expired hold should have been lazily evicted on the read. if _, active := c.GetManualHold(cfg.ID, now); active { diff --git a/go/internal/loadpoint/controller_offer_restore.go b/go/internal/loadpoint/controller_offer_restore.go index f44a0de30..1361da27d 100644 --- a/go/internal/loadpoint/controller_offer_restore.go +++ b/go/internal/loadpoint/controller_offer_restore.go @@ -89,7 +89,7 @@ func (c *Controller) restoreOffers(now time.Time, samples map[string]EVSample) { continue } c.manager.setCommandedForManual(cfg.ID, saved.PowerW, saved.Reason, time.Time{}) - c.recordOfferPeak(now, cfg.ID, saved.PowerW, map[string]any{"phase_mode": saved.PhaseMode}) + c.manager.setCommandedPhases(cfg.ID, c.commandPhases(saved.PhaseMode, saved.PowerW, cfg.PhaseSplitW)) if c.savedOffers == nil { c.savedOffers = map[string]savedOffer{} } diff --git a/go/internal/loadpoint/controller_phase_choice_test.go b/go/internal/loadpoint/controller_phase_choice_test.go new file mode 100644 index 000000000..804f24036 --- /dev/null +++ b/go/internal/loadpoint/controller_phase_choice_test.go @@ -0,0 +1,294 @@ +package loadpoint + +import ( + "context" + "math" + "testing" + "time" +) + +// threePhaseSteps are Easee's whole-amp offers at 230 V: 6–16 A on one +// phase, then 6–16 A on three. +var threePhaseSteps = []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680, + 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040} + +// meteredEVCap stands in for control.EVFuseBudget: the plugged cars' last +// offers plus the room the meter shows below the fuse less 1 A, on the +// phases FuseConfigs names for this tick. +func meteredEVCap(c *Controller, worstA func() float64) func() (float64, bool) { + return func() (float64, bool) { + site := c.siteFuse() + phases, offersW := float64(site.Phases()), 0.0 + for _, cfg := range c.FuseConfigs() { + if st, ok := c.manager.State(cfg.ID); ok && st.PluggedIn { + if cfg.PhaseMode != "" && cfg.PhaseMode != "3p" { + phases = 1 + } + offersW += math.Max(0, st.CommandedW) + } + } + return math.Max(0, offersW+(site.MaxAmps-1-worstA())*site.Voltage*phases), true + } +} + +// R2 of the 21 Oct 2026 review: a saved battery-SoC unlock that is not +// armed made the loadpoint's own fuse layer expect "auto" while Charge now +// sent the driver's 3p. The last offer counted as 0 and new room as one +// phase, so the charge ran 4,830 / 0 / 0 W with fuse_limit on a 25 A site. +func TestChargeNowHoldsWithUnarmedBatteryUnlock(t *testing.T) { + now := time.Date(2026, 1, 10, 19, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 4140, MaxChargeW: 11040, + AllowedStepsW: []float64{0, 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040}} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.manager.SetSchedule(cfg.ID, Schedule{SurplusUnlockBatSoC: 0.9}) + c.SetBatSoCProvider(func() (float64, bool) { return 0.5, true }) + c.SetSiteSurplusForEV(func() (float64, bool) { return -2000, true }) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + evW := 0.0 // The meter shows 3 A of house load plus the car's last offer. + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 3 + evW/690 })) + c.SetManualHold(cfg.ID, ManualHold{PowerW: 11040, Persistent: true, StartedAt: now, UpdatedAt: now}) + for tick := 0; tick < 20; tick++ { + c.Tick(context.Background(), now) + command, ok := lastSetCurrent(sender.calls) + st, _ := c.manager.State(cfg.ID) + if !ok || command.power < cfg.MaxChargeW-690 || st.CommandedReason == "fuse_limit" { + t.Fatalf("tick %d: offer %+v, reason %q; want about %.0f W", tick, command, st.CommandedReason, cfg.MaxChargeW) + } + if c.IsBatSoCArmed(cfg.ID) { + t.Fatal("the unlock armed at 50% SoC") + } + evW = command.power + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: evW} + now = now.Add(5 * time.Second) + } +} + +// On a one-phase site the command goes out as 1p. The loadpoint's own fuse +// layer expected the empty mode's 3p, so the last offer counted as 0 and +// Charge now ran 4,830 / 0 / 0 W. +func TestChargeNowHoldsOnOnePhaseSite(t *testing.T) { + now := time.Date(2026, 1, 10, 19, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 7360, + AllowedStepsW: []float64{0, 1380, 2300, 3220, 4140, 4830, 5520, 6440, 7360}} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 1}) + evW := 0.0 // 3 A of house load plus the car's last offer. + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 3 + evW/230 })) + c.SetManualHold(cfg.ID, ManualHold{PowerW: 4830, Persistent: true, StartedAt: now, UpdatedAt: now}) + for tick := 0; tick < 20; tick++ { + c.Tick(context.Background(), now) + command, ok := lastSetCurrent(sender.calls) + if st, _ := c.manager.State(cfg.ID); !ok || command.power != 4830 || command.phaseMode != "1p" { + t.Fatalf("tick %d: offer %+v, reason %q; want 4830 W on 1p", tick, command, st.CommandedReason) + } + evW = command.power + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: evW} + now = now.Add(5 * time.Second) + } +} + +// R1 of the 21 Oct 2026 review: with an unset or auto phase mode the +// surplus dwell sent 3p while the fuse room was measured on one phase. +// 9 kW of surplus gave 2,300–3,450 W labelled 3p, under 6 A, so the car +// did not charge. The cap must be measured for the dwell's 3p. +func TestSurplusOnlyAutoPhaseOffersLegalThreePhaseSteps(t *testing.T) { + for _, phaseMode := range []string{"", "auto"} { + t.Run("mode_"+phaseMode, func(t *testing.T) { + now := time.Date(2026, 6, 1, 12, 0, 0, 0, time.UTC) + cfg := surplusTestConfig() + cfg.DriverName = "easee" + cfg.PhaseMode = phaseMode + cfg.AllowedStepsW = threePhaseSteps + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + const surplusW = 9000 + c.SetSiteSurplusForEV(func() (float64, bool) { return surplusW, true }) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + evW := 0.0 // PV export on every phase, less the car's draw. + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return math.Abs(evW-surplusW) / 690 })) + for tick := 0; tick < 20; tick++ { + c.Tick(context.Background(), now) + command, ok := lastSetCurrent(sender.calls) + st, _ := c.manager.State(cfg.ID) + if !ok || command.phaseMode != "3p" || command.power/(3*230) < 6 || command.power > surplusW { + t.Fatalf("tick %d: offer %+v, reason %q; want a 3p step of at least 6 A within the surplus", tick, command, st.CommandedReason) + } + // The first offer may stop short while the meter still shows + // the full export; from then on it follows the surplus. + if tick > 0 && command.power < surplusW-690 { + t.Fatalf("tick %d: offer %+v, reason %q; want within 1 A of %d W", tick, command, st.CommandedReason, surplusW) + } + evW = command.power + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: evW} + now = now.Add(5 * time.Second) + } + }) + } +} + +// A one-phase charger left on an empty phase mode names no phases. Its own +// 6 A minimum stands; a three-phase floor of 4,140 W is above its 3,680 W +// maximum, and it would never charge. +func TestOnePhaseChargerWithEmptyModeCharges(t *testing.T) { + now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "charger", MinChargeW: 1380, MaxChargeW: 3680, + AllowedStepsW: []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680}} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + evW := 0.0 // 3 A of house load plus the car on one phase. + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 3 + evW/230 })) + c.SetManualHold(cfg.ID, ManualHold{PowerW: 3680, Persistent: true}) + for tick := 0; tick < 12; tick++ { + c.Tick(context.Background(), now) + command, ok := lastSetCurrent(sender.calls) + if st, _ := c.manager.State(cfg.ID); !ok || command.power != 3680 { + t.Fatalf("tick %d: offer %+v, reason %q; want 3680 W", tick, command, st.CommandedReason) + } + evW = command.power + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: evW} + now = now.Add(5 * time.Second) + } +} + +// A fuse cap snaps to steps the commanded phases can carry. Four amps of +// room per phase is 2,760 W, but on three phases that is under 6 A. +func TestThreePhaseFuseCapNeverSnapsBelowSixAmps(t *testing.T) { + now := time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 11040, + AllowedStepsW: threePhaseSteps, PhaseMode: "3p"} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 20 })) + c.SetManualHold(cfg.ID, ManualHold{PowerW: 11040, Persistent: true}) + c.Tick(context.Background(), now) + command, ok := lastSetCurrent(sender.calls) + if st, _ := c.manager.State(cfg.ID); !ok || command.power != 0 || st.CommandedReason != "fuse_limit" { + t.Fatalf("offer %+v, reason %q; want 0 W fuse_limit", command, st.CommandedReason) + } +} + +// onePhaseSteps are whole amps from 6 to 16 A on one phase at 230 V. +var onePhaseSteps = []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680} + +// commandAmps is the current per phase a driver draws for the command, as +// the OCPP driver works it out: the watts over the phases the command +// names, never above max_amps_per_phase. +func commandAmps(cmd sentCommand) float64 { + if cmd.power <= 0 || cmd.voltage <= 0 { + return 0 + } + a := cmd.power / (cmd.voltage * float64(PhaseFor(cmd.phaseMode, cmd.power, defaultPhaseSplitW))) + if cmd.maxAmpsPerPhase > 0 { + a = math.Min(a, cmd.maxAmpsPerPhase) + } + return a +} + +// Simulatorprov 21 Oct 2026, case s5s: surplus only, 9.3 kW of surplus and +// a plan that shows no PV. The day's 1p lock hands back every step, so 1p +// went out at 5,750 W with the 25 A fuse as its only limit: 25 A on one +// phase of a 16 A charger. A one-phase charger left on an empty mode gets +// the same lock and must keep its full 16 A. +func TestSurplusOnePhaseOfferStaysWithinTheCharger(t *testing.T) { + for _, tc := range []struct { + name string + cfg Config + }{ + {"16 A three-phase charger, auto", Config{MinChargeW: 4140, MaxChargeW: 11040, PhaseMode: "auto", AllowedStepsW: threePhaseSteps}}, + {"16 A one-phase charger, empty mode", Config{MinChargeW: 1380, MaxChargeW: 3680, AllowedStepsW: onePhaseSteps}}, + } { + t.Run(tc.name, func(t *testing.T) { + now := time.Date(2026, 10, 21, 12, 0, 0, 0, time.Local) + cfg := tc.cfg + cfg.ID, cfg.DriverName, cfg.SurplusOnly = "garage", "evse", true + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.SetSiteSurplusForEV(func() (float64, bool) { return 9310, true }) + c.SetNearTermPeakSurplusW(func(time.Duration) (float64, bool) { return 0, true }) + c.SetPeakRemainingSurplusW(func() (float64, bool) { return 0, true }) + for tick := 0; tick < 10; tick++ { + c.Tick(context.Background(), now) + cmd, _ := lastSetCurrent(sender.calls) + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: cmd.power} + now = now.Add(5 * time.Second) + } + onePhase := 0 + for _, cmd := range sender.calls { + if cmd.action != "ev_set_current" { + continue + } + if a := commandAmps(cmd); a > 16+1e-9 { + t.Fatalf("offer %+v is %.1f A per phase to a 16 A charger", cmd, a) + } + if cmd.phaseMode == "1p" { + onePhase++ + if cmd.maxAmpsPerPhase > 16+1e-9 { + t.Fatalf("1p offer %+v allows %.1f A per phase", cmd, cmd.maxAmpsPerPhase) + } + } + } + last, ok := lastSetCurrent(sender.calls) + if onePhase == 0 || !ok || last.phaseMode != "1p" || last.power != 3680 { + t.Fatalf("%d 1p offers, last %+v; want the 1p lock to charge at 3,680 W (16 A)", onePhase, last) + } + }) + } +} + +// Charge now on one phase gets the charger's own current on that phase, +// and the fuse decides the rest. A 16 A charger that can run three phases +// gets 16 A, not the 25 A fuse. A 32 A charger, whether it switches phases +// or is a one-phase charger, keeps its 32 A when the fuse allows it. +func TestChargeNowOnOnePhaseKeepsTheChargersCurrent(t *testing.T) { + for _, tc := range []struct { + name string + cfg Config + fuseA float64 + holdW float64 + wantW float64 + wantA float64 + reason string + }{ + {"16 A three-phase charger, auto", Config{MinChargeW: 4140, MaxChargeW: 11040, PhaseMode: "auto", AllowedStepsW: threePhaseSteps}, 25, 7360, 3680, 16, "charger_limit"}, + {"16 A three-phase charger, empty mode", Config{MinChargeW: 4140, MaxChargeW: 11040, AllowedStepsW: threePhaseSteps}, 25, 7360, 3680, 16, "charger_limit"}, + {"32 A three-phase charger, auto", Config{MinChargeW: 1380, MaxChargeW: 22080, PhaseMode: "auto"}, 40, 7360, 7360, 32, "manual_hold"}, + {"32 A one-phase charger", Config{MinChargeW: 1380, MaxChargeW: 7360, PhaseMode: "1p"}, 40, 7360, 7360, 32, "manual_hold"}, + {"16 A one-phase charger, empty mode", Config{MinChargeW: 1380, MaxChargeW: 3680, AllowedStepsW: onePhaseSteps}, 25, 3680, 3680, 16, "manual_hold"}, + } { + t.Run(tc.name, func(t *testing.T) { + now := time.Date(2026, 10, 21, 19, 0, 0, 0, time.UTC) + cfg := tc.cfg + cfg.ID, cfg.DriverName = "garage", "evse" + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + sender := &fakeSender{} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: tc.fuseA, Voltage: 230, PhaseCnt: 3}) + evW := 0.0 // 3 A of house load plus the car on one phase. + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 3 + evW/230 })) + c.SetManualHold(cfg.ID, ManualHold{PowerW: tc.holdW, PhaseMode: "1p", Persistent: true}) + for tick := 0; tick < 12; tick++ { + c.Tick(context.Background(), now) + cmd, ok := lastSetCurrent(sender.calls) + st, _ := c.manager.State(cfg.ID) + if !ok || cmd.phaseMode != "1p" || cmd.power != tc.wantW || math.Abs(commandAmps(cmd)-tc.wantA) > 1e-9 || st.CommandedReason != tc.reason { + t.Fatalf("tick %d: offer %+v (%.1f A), reason %q; want %.0f W on 1p, %.0f A, %q", + tick, cmd, commandAmps(cmd), st.CommandedReason, tc.wantW, tc.wantA, tc.reason) + } + evW = cmd.power + samples[cfg.DriverName] = EVSample{Connected: true, RequestActive: true, PowerW: evW} + now = now.Add(5 * time.Second) + } + }) + } +} diff --git a/go/internal/loadpoint/controller_phase_handoff_test.go b/go/internal/loadpoint/controller_phase_handoff_test.go index 94714db04..249d74c75 100644 --- a/go/internal/loadpoint/controller_phase_handoff_test.go +++ b/go/internal/loadpoint/controller_phase_handoff_test.go @@ -54,22 +54,30 @@ func runPhaseTick(t *testing.T, cfg Config, site SiteFuse, wantWh float64) sentC return sender.calls[0] } +// Planned charging on an auto loadpoint runs on three phases. A plan of +// 4,000 W, below 6 A a phase, goes out as the 3Φ minimum, not as one phase. func TestCommandCarriesPhaseModePreference(t *testing.T) { cfg := phaseLoadpoint("auto", 0, 0) cmd := runPhaseTick(t, cfg, SiteFuse{MaxAmps: 16, Voltage: 230, PhaseCnt: 3}, 4000) - if cmd.phaseMode != "auto" { - t.Errorf("phase_mode in cmd = %q, want \"auto\"", cmd.phaseMode) + if cmd.phaseMode != "3p" || cmd.power != 4140 { + t.Errorf("cmd = %.0f W phase_mode %q, want 4140 W \"3p\"", cmd.power, cmd.phaseMode) + } + cfg = phaseLoadpoint("1p", 0, 0) + if cmd := runPhaseTick(t, cfg, SiteFuse{MaxAmps: 16, Voltage: 230, PhaseCnt: 3}, 3000); cmd.phaseMode != "1p" { + t.Errorf("a 1p pin sent phase_mode %q", cmd.phaseMode) } } -func TestEmptyPhaseModeOmittedFromCommand(t *testing.T) { - // Empty PhaseMode (legacy default) MUST NOT land in the cmd — - // drivers that don't read this field should see exactly the - // pre-phase-switching shape and behave as locked-3Φ. +// An unset PhaseMode means auto: a charger that can reach 6 A on three +// phases runs on three, and says so, so the order is raised to the 3Φ +// minimum instead of going out as a 4,000 W "unset" that Easee runs as +// 5.8 A on three phases and stops. Drivers that read no phase_mode (OCPP +// takes "3p" as the site's phases) see the same order as before. +func TestEmptyPhaseModeIsThreePhase(t *testing.T) { cfg := phaseLoadpoint("", 0, 0) cmd := runPhaseTick(t, cfg, SiteFuse{MaxAmps: 16, Voltage: 230, PhaseCnt: 3}, 4000) - if cmd.phaseMode != "" { - t.Errorf("phase_mode = %q, want empty (omitted from cmd)", cmd.phaseMode) + if cmd.phaseMode != "3p" || cmd.power != 4140 { + t.Errorf("cmd = %.0f W phase_mode %q, want 4140 W \"3p\"", cmd.power, cmd.phaseMode) } } diff --git a/go/internal/loadpoint/controller_phase_mode_test.go b/go/internal/loadpoint/controller_phase_mode_test.go index 577c8deaf..e75a499eb 100644 --- a/go/internal/loadpoint/controller_phase_mode_test.go +++ b/go/internal/loadpoint/controller_phase_mode_test.go @@ -2,46 +2,58 @@ package loadpoint import "testing" -// TestResolvePhaseMode locks the phase_mode precedence — in particular the -// operator directive (2026-05-30) that an active charge schedule overrides the -// surplus 1Φ forecast lock so a deadline-driven charge can pull 3Φ grid power. +// TestResolvePhaseMode locks the phase_mode precedence: an operator pin wins, +// unset means auto, a charger that cannot reach 6 A on three phases runs on +// one, planned charging runs on three (Fredrik, 22 Oct 2026: one phase never +// helps the fuse and caps the car near 3.7 kW), and only surplus charging +// without a schedule may use one phase. The 2026-05-30 directive stands: a +// schedule overrides the surplus 1Φ forecast lock. func TestResolvePhaseMode(t *testing.T) { cases := []struct { name string operatorMode string + oneOnly bool // the charger cannot reach 6 A on three phases scheduleActive bool locked1P bool surplusOn bool dwell string want string }{ - // The fix: a schedule beats the 1Φ forecast lock. - {"schedule overrides 1p lock (auto)", "auto", true, true, true, "1p", "auto"}, - {"schedule overrides 1p lock (unset)", "", true, true, true, "1p", "auto"}, + // A schedule beats the 1Φ forecast lock: auto and unset are three + // phases. + {"schedule overrides 1p lock (auto)", "auto", false, true, true, true, "1p", "3p"}, + {"schedule overrides 1p lock (unset)", "", false, true, true, true, "1p", "3p"}, // An explicit operator pin is still honoured under a schedule. - {"schedule honours explicit 3p", "3p", true, true, false, "", "3p"}, - {"schedule honours explicit 1p", "1p", true, true, false, "", "1p"}, + {"schedule honours explicit 3p", "3p", false, true, true, false, "", "3p"}, + {"schedule honours explicit 1p", "1p", false, true, true, false, "", "1p"}, // No schedule: the surplus 1Φ lock applies as before. - {"1p lock applies without schedule", "auto", false, true, true, "3p", "1p"}, + {"1p lock applies without schedule", "auto", false, false, true, true, "3p", "1p"}, // A 3Φ-only charger must never be forced to 1Φ by the surplus // lock — it physically cannot trickle single-phase (e.g. CTEK). - {"3p charger overrides 1p lock", "3p", false, true, true, "1p", "3p"}, + {"3p charger overrides 1p lock", "3p", false, false, true, true, "1p", "3p"}, // No schedule, surplus-active, auto → dwell verdict. - {"dwell verdict when surplus auto", "auto", false, false, true, "3p", "3p"}, - {"auto default when no dwell", "auto", false, false, true, "", "auto"}, - {"unset → auto under surplus", "", false, false, true, "", "auto"}, - // No schedule, no surplus → operator's mode verbatim. - {"operator verbatim, no surplus", "auto", false, false, false, "", "auto"}, - {"operator explicit 1p verbatim", "1p", false, false, false, "", "1p"}, - {"operator unset verbatim", "", false, false, false, "", ""}, + {"dwell verdict when surplus auto", "auto", false, false, false, true, "3p", "3p"}, + {"auto default when no dwell", "auto", false, false, false, true, "", "auto"}, + {"unset → auto under surplus", "", false, false, false, true, "", "auto"}, + // No schedule, no surplus: auto and unset are three phases, a pin is + // verbatim. + {"auto is 3p without surplus", "auto", false, false, false, false, "", "3p"}, + {"unset is 3p without surplus", "", false, false, false, false, "", "3p"}, + {"operator explicit 1p verbatim", "1p", false, false, false, false, "", "1p"}, + // The day lock and the dwell belong to surplus charging alone. + {"1p lock ignored without surplus", "auto", false, false, true, false, "1p", "3p"}, + // A one-phase-only charger runs on one phase, whatever drives it. + {"one-phase charger, schedule", "auto", true, true, false, false, "", "1p"}, + {"one-phase charger, unset", "", true, false, false, false, "", "1p"}, + {"one-phase charger pinned 3p", "3p", true, false, false, false, "", "3p"}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - got := resolvePhaseMode(tc.operatorMode, tc.scheduleActive, tc.locked1P, tc.surplusOn, tc.dwell) + got := resolvePhaseMode(tc.operatorMode, !tc.oneOnly, tc.scheduleActive, tc.locked1P, tc.surplusOn, tc.dwell) if got != tc.want { - t.Errorf("resolvePhaseMode(%q, sched=%v, locked1P=%v, surplus=%v, dwell=%q) = %q, want %q", - tc.operatorMode, tc.scheduleActive, tc.locked1P, tc.surplusOn, tc.dwell, got, tc.want) + t.Errorf("resolvePhaseMode(%q, one-phase=%v, sched=%v, locked1P=%v, surplus=%v, dwell=%q) = %q, want %q", + tc.operatorMode, tc.oneOnly, tc.scheduleActive, tc.locked1P, tc.surplusOn, tc.dwell, got, tc.want) } }) } diff --git a/go/internal/loadpoint/controller_shared_fuse_test.go b/go/internal/loadpoint/controller_shared_fuse_test.go index 7117e0179..3fcd495c7 100644 --- a/go/internal/loadpoint/controller_shared_fuse_test.go +++ b/go/internal/loadpoint/controller_shared_fuse_test.go @@ -104,8 +104,7 @@ func TestOnePhaseCarsShareWorstPhaseBudget(t *testing.T) { sender := &fakeSender{} c := newTestController(t, configs, d, map[string]EVSample{"charger-a": {Connected: true, RequestActive: true}, "charger-b": {Connected: true, RequestActive: true}}, sender) c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) - c.SetPerPhaseMeterAmps(func() (float64, float64, float64, bool) { return 1000.0 / 230, 1000.0 / 230, 1000.0 / 230, true }) - c.SetFuseEVMax(func() (float64, bool) { return 14250, true }) + c.SetFuseEVMax(meteredEVCap(c, func() float64 { return 1000.0 / 230 })) c.Tick(context.Background(), now) total := 0.0 commands := map[string]float64{} diff --git a/go/internal/loadpoint/controller_surplus_test.go b/go/internal/loadpoint/controller_surplus_test.go index 39e2223b2..cdaf6f303 100644 --- a/go/internal/loadpoint/controller_surplus_test.go +++ b/go/internal/loadpoint/controller_surplus_test.go @@ -49,7 +49,7 @@ func TestSurplusCmd_PauseResumeHysteresis(t *testing.T) { surplus = 3000 var got float64 for i := 0; i < 4; i++ { - got = c.computeSurplusCmd(now, cfg, 6900, 0) + got = c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) } if got != 0 { t.Fatalf("after 4 below-min ticks expected paused (got=%v)", got) @@ -64,7 +64,7 @@ func TestSurplusCmd_PauseResumeHysteresis(t *testing.T) { now = now.Add(60 * time.Second) // past surplusMinPauseHold (35s) surplus = 4140 // exactly minStep3, no margin for i := 0; i < 4; i++ { - got = c.computeSurplusCmd(now, cfg, 6900, 0) + got = c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) now = now.Add(5 * time.Second) } if got != 0 { @@ -77,7 +77,7 @@ func TestSurplusCmd_PauseResumeHysteresis(t *testing.T) { surplus = 5000 resumed := false for i := 0; i < 4; i++ { - got = c.computeSurplusCmd(now, cfg, 6900, 0) + got = c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) now = now.Add(5 * time.Second) if got > 0 { resumed = true @@ -110,7 +110,7 @@ func TestSurplusCmd_UpStepGatedOnAverage(t *testing.T) { c.SetSiteSurplusForEV(func() (float64, bool) { return surplus, true }) step := func(s float64) float64 { surplus = s - got := c.computeSurplusCmd(now, cfg, 11040, 0) + got := c.computeSurplusCmd(now, cfg, 11040, c.pickSurplusSteps(now, cfg)) now = now.Add(5 * time.Second) return got } @@ -168,7 +168,7 @@ func TestSurplusCmd_MinPauseHold(t *testing.T) { // Force pause via 4 below-min ticks. surplus = 1000 for i := 0; i < 4; i++ { - c.computeSurplusCmd(now, cfg, 6900, 0) + c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) } if !c.surplusPausedFor(cfg.ID) { t.Fatalf("setup: expected paused") @@ -179,7 +179,7 @@ func TestSurplusCmd_MinPauseHold(t *testing.T) { now = now.Add(10 * time.Second) surplus = 9000 for i := 0; i < 4; i++ { - got := c.computeSurplusCmd(now, cfg, 6900, 0) + got := c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) if got != 0 { t.Fatalf("within %v of pause edge expected still paused, got=%v", surplusMinPauseHold, got) @@ -191,7 +191,7 @@ func TestSurplusCmd_MinPauseHold(t *testing.T) { now = now.Add(35 * time.Second) resumed := false for i := 0; i < 4; i++ { - got := c.computeSurplusCmd(now, cfg, 6900, 0) + got := c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) now = now.Add(5 * time.Second) if got > 0 { resumed = true @@ -222,9 +222,9 @@ func TestSurplusCmd_StepSnap(t *testing.T) { surplus = 8000 for i := 0; i < 4; i++ { // prime the rolling window so the pause hysteresis stays open - c.computeSurplusCmd(now, cfg, 6900, 0) + c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) } - if got := c.computeSurplusCmd(now, cfg, 6900, 0); got != 6900 { + if got := c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)); got != 6900 { t.Fatalf("with 8000 W surplus + 6900 W wantW expected snap to 6900, got=%v", got) } @@ -234,9 +234,9 @@ func TestSurplusCmd_StepSnap(t *testing.T) { // higher would leak grid import. surplus = 5500 for i := 0; i < 4; i++ { - c.computeSurplusCmd(now, cfg, 6900, 0) + c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)) } - if got := c.computeSurplusCmd(now, cfg, 6900, 0); got != 4140 { + if got := c.computeSurplusCmd(now, cfg, 6900, c.pickSurplusSteps(now, cfg)); got != 4140 { t.Fatalf("with 5500 W surplus expected snap down to 4140, got=%v", got) } } @@ -249,7 +249,7 @@ func TestSurplusCmd_NoReader_ReturnsWantW(t *testing.T) { c := NewController(NewManager(), nil, nil, nil) cfg := surplusTestConfig() now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC) - if got := c.computeSurplusCmd(now, cfg, 4140, 0); got != 4140 { + if got := c.computeSurplusCmd(now, cfg, 4140, c.pickSurplusSteps(now, cfg)); got != 4140 { t.Fatalf("no-reader fallback should pass wantW through, got=%v", got) } } @@ -263,7 +263,7 @@ func TestSurplusCmd_StaleReader_Returns0(t *testing.T) { cfg := surplusTestConfig() c.SetSiteSurplusForEV(func() (float64, bool) { return 0, false }) now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC) - if got := c.computeSurplusCmd(now, cfg, 4140, 0); got != 0 { + if got := c.computeSurplusCmd(now, cfg, 4140, c.pickSurplusSteps(now, cfg)); got != 0 { t.Fatalf("stale reader expected pause-to-0, got=%v", got) } } diff --git a/go/internal/loadpoint/controller_test.go b/go/internal/loadpoint/controller_test.go index c1c74932d..574478f0e 100644 --- a/go/internal/loadpoint/controller_test.go +++ b/go/internal/loadpoint/controller_test.go @@ -25,6 +25,7 @@ type sentCommand struct { voltage float64 maxAmpsPerPhase float64 sitePhases int + capAmpsPerPhase *float64 } func lastSetCurrent(calls []sentCommand) (sentCommand, bool) { @@ -48,14 +49,15 @@ func countAction(calls []sentCommand, action string) int { func (f *fakeSender) Send(ctx context.Context, driver string, payload []byte) error { var d struct { - Action string `json:"action"` - PowerW float64 `json:"power_w"` - PhaseMode string `json:"phase_mode"` - PhaseSplitW float64 `json:"phase_split_w"` - MinPhaseHoldS int `json:"min_phase_hold_s"` - Voltage float64 `json:"voltage"` - MaxAmpsPerPhase float64 `json:"max_amps_per_phase"` - SitePhases int `json:"site_phases"` + Action string `json:"action"` + PowerW float64 `json:"power_w"` + PhaseMode string `json:"phase_mode"` + PhaseSplitW float64 `json:"phase_split_w"` + MinPhaseHoldS int `json:"min_phase_hold_s"` + Voltage float64 `json:"voltage"` + MaxAmpsPerPhase float64 `json:"max_amps_per_phase"` + SitePhases int `json:"site_phases"` + CapAmpsPerPhase *float64 `json:"cap_amps_per_phase"` } if err := json.Unmarshal(payload, &d); err != nil { return err @@ -65,7 +67,7 @@ func (f *fakeSender) Send(ctx context.Context, driver string, payload []byte) er phaseMode: d.PhaseMode, phaseSplitW: d.PhaseSplitW, minPhaseHoldS: d.MinPhaseHoldS, voltage: d.Voltage, maxAmpsPerPhase: d.MaxAmpsPerPhase, - sitePhases: d.SitePhases, + sitePhases: d.SitePhases, capAmpsPerPhase: d.CapAmpsPerPhase, }) return f.err } diff --git a/go/internal/loadpoint/deadline_need.go b/go/internal/loadpoint/deadline_need.go index 73f88e0eb..034287090 100644 --- a/go/internal/loadpoint/deadline_need.go +++ b/go/internal/loadpoint/deadline_need.go @@ -43,22 +43,30 @@ func DeadlineNeed(st State) (energyWh, powerW float64, ok bool) { // lower of the charger's and the car's maximum, and the charger's steps up to // it. When the charger's lowest step is above the car's maximum it keeps that // one step: the car takes what it can, and a plan with no step would never -// charge. +// charge. The minimum is never below the lowest step: PlanningSteps leaves +// out levels the charger's phases cannot run. func LimitPower(st State, steps []float64) (minW, maxW float64, out []float64) { minW, maxW = st.MinChargeW, st.MaxChargeW + lowest := 0.0 + for _, w := range steps { + if w > 0 && (lowest == 0 || w < lowest) { + lowest = w + } + } + minW = math.Max(minW, lowest) if st.CarMaxW <= 0 || (maxW > 0 && st.CarMaxW >= maxW) { + if maxW > 0 && minW > maxW { + minW = maxW + } return minW, maxW, steps } maxW = st.CarMaxW - lowest, kept := 0.0, false + kept := false for _, w := range steps { if w <= maxW { out = append(out, w) kept = kept || w > 0 } - if w > 0 && (lowest == 0 || w < lowest) { - lowest = w - } } if !kept && lowest > 0 { out = append(out, lowest) diff --git a/go/internal/loadpoint/loadpoint.go b/go/internal/loadpoint/loadpoint.go index d27993533..7bcb105fa 100644 --- a/go/internal/loadpoint/loadpoint.go +++ b/go/internal/loadpoint/loadpoint.go @@ -57,10 +57,10 @@ type Config struct { // PhaseMode selects how the controller picks between 1Φ and 3Φ // delivery each tick. "3p" (default) and "1p" lock the install to - // one mode and filter AllowedStepsW accordingly. "auto" lets the - // controller switch based on the MPC budget, with a hysteresis - // timer (MinPhaseHoldS) preventing flap. Empty == "3p" for - // backward compatibility with pre-switching configs. + // one mode and filter AllowedStepsW accordingly. "auto" charges on + // three phases; only surplus charging below the 3Φ minimum switches + // to one, with a hysteresis timer (MinPhaseHoldS) preventing flap. + // Empty == "3p" for backward compatibility with pre-switching configs. PhaseMode string `yaml:"phase_mode,omitempty" json:"phase_mode,omitempty"` // PhaseSplitW is the wantW threshold below which "auto" picks 1Φ. @@ -203,6 +203,13 @@ type State struct { // PlanningTarget). CarMaxW float64 `json:"-"` CarChargeLimit float64 `json:"-"` + // CarPhases is 1 when this session's car draws on one phase of a + // three-phase order, seen on the site meter, and 0 when unknown; a car + // profile's phases may set it later. CarPhase is that phase, 1-3. The + // plan counts the car's one-phase levels (OnePhaseCarSteps) and the + // order names the current per phase it should take. + CarPhases int `json:"-"` + CarPhase int `json:"-"` // MinChargeW / MaxChargeW / AllowedStepsW are repeated here so the // UI has everything for rendering in one fetch. @@ -260,12 +267,15 @@ type State struct { CommandedSinceMs int64 `json:"commanded_since_ms,omitempty"` // Internal identity of the manual choice used to compute the order. ManualCommandUpdatedAt time.Time `json:"-"` + // CommandedPhases is 1 or 3: the phases the last command lands on, from + // its phase_mode and split. Zero before the first command. + CommandedPhases int `json:"-"` // CommandedReason names the dispatch branch that decided CommandedW: // "plan", "no_plan_budget", "pv_surplus", "pv_surplus_pause", - // "fuse_limit", "fuse_cooldown", "site_meter_stale", "manual_hold", - // "wake_kick". Empty until the first dispatch tick. The UI renders - // the specific cause instead of a generic "paused by the box". + // "fuse_limit", "import_limit", "site_meter_stale", "manual_hold", + // "wake_kick". Empty until the first dispatch tick. The UI renders the specific cause + // instead of a generic "paused by the box". CommandedReason string `json:"commanded_reason,omitempty"` // GridDeferred is true while MPC has deferred grid-funded planning @@ -405,6 +415,9 @@ type loadpointRuntime struct { goalRiskNotified bool vehicleAnchored bool chargedSinceStart bool + // carPhases is this session's evidence of a one-phase car + // (one_phase_car.go). + carPhases carPhases Config pluggedIn bool @@ -507,6 +520,7 @@ type loadpointRuntime struct { commandedW float64 commandedKnown bool commandedReason string + commandedPhases int // 1 or 3: where the last command lands; 0 before it // commandedSince is when the current (commandedW, commandedReason) pair // was first ordered. The manual status counts elapsed time from it. commandedSince time.Time @@ -561,9 +575,9 @@ func (m *Manager) SetCommandedW(id string, w float64) { // dispatch branch chose that value — the fact operators were left to // reverse-engineer from amp readings (#1009). Reasons are short stable // tokens: "plan", "no_plan_budget", "pv_surplus", "pv_surplus_pause", -// "fuse_limit", "fuse_cooldown", "site_meter_stale", "manual_hold", -// "wake_kick". Empty keeps whatever was recorded before (used by the -// legacy SetCommandedW wrapper). No-op for an unknown id. +// "fuse_limit", "import_limit", "site_meter_stale", "manual_hold", +// "wake_kick". Empty keeps whatever was recorded before (used by the legacy SetCommandedW wrapper). +// No-op for an unknown id. func (m *Manager) SetCommanded(id string, w float64, reason string) { m.setCommandedForManual(id, w, reason, time.Time{}) } @@ -586,6 +600,15 @@ func (m *Manager) setCommandedForManual(id string, w float64, reason string, upd } } +// setCommandedPhases records the phases the last command lands on. +func (m *Manager) setCommandedPhases(id string, phases int) { + m.mu.Lock() + defer m.mu.Unlock() + if lp, ok := m.byID[id]; ok { + lp.commandedPhases = phases + } +} + // Load replaces the configured set. Idempotent: existing state is // carried across when the ID is kept; removed IDs are dropped. func (m *Manager) Load(cfgs []Config) { @@ -834,11 +857,13 @@ func (m *Manager) observe(id string, pluggedIn bool, powerW, deliveredWh float64 lp.deliveredHistory = nil lp.notRequestingSince = time.Time{} lp.chargingDeclined = false + lp.carPhases = carPhases{} } if !pluggedIn { // Plug-out: drop any pending completion timer / latch. lp.notRequestingSince = time.Time{} lp.chargingDeclined = false + lp.carPhases = carPhases{} if lp.capacityFromCar { // The car that measured its own capacity left with its // session — the next one may be different. @@ -1354,6 +1379,11 @@ func (lp *loadpointRuntime) snapshot() State { CommandedW: lp.commandedW, CommandedReason: lp.commandedReason, CommandedKnown: lp.commandedKnown, + CommandedPhases: lp.commandedPhases, + CarPhase: lp.carPhases.phase, + } + if lp.carPhases.phase > 0 { + st.CarPhases = 1 } if lp.VehicleCapacityWh <= 0 { st.VehicleCapacityWh = 60000 diff --git a/go/internal/loadpoint/manual_hold_state.go b/go/internal/loadpoint/manual_hold_state.go index 36cd4d482..71562f9cc 100644 --- a/go/internal/loadpoint/manual_hold_state.go +++ b/go/internal/loadpoint/manual_hold_state.go @@ -153,10 +153,12 @@ func (m *Manager) setManualPersistenceError(id string, err error) { } } -// RestoreManualHold returns pending until hardware identity is known. A -// verified same-session positive hold or a hardware-bound pause is restored. -// Unknown/changed sessions and legacy records return a persistent zero-W hold -// plus unconfirmed: failed restoration must not start automatic charging. +// RestoreManualHold returns pending until the record's proof is known: the +// hardware identity, and the session ID when the record has one. A record +// without a session ID is checked by hardware alone; most chargers never +// report one, and an observed unplug still ends the hold. Changed hardware or +// sessions and legacy records return a persistent zero-W hold plus +// unconfirmed: failed restoration must not start automatic charging. func (m *Manager) RestoreManualHold(id string) (ManualHold, string) { m.sessionMu.Lock() defer m.sessionMu.Unlock() @@ -227,9 +229,14 @@ func (m *Manager) RestoreManualHold(id string) (ManualHold, string) { !record.Hold.ExpiresAt.IsZero() || !finite(record.Hold.PowerW) || record.Hold.PowerW < 0 { return ManualHold{Persistent: true}, "unconfirmed" } - if record.Hold.PowerW == 0 || (sessionID != "" && sessionID == record.SessionID) { + if record.Hold.PowerW == 0 || record.SessionID == "" || sessionID == record.SessionID { return record.Hold, "restored" } + if sessionID == "" { + // A charger can report its session after its identity. Deciding now + // would turn Charge now into a pause on every restart. + return ManualHold{Persistent: true}, "pending" + } return ManualHold{Persistent: true}, "unconfirmed" } diff --git a/go/internal/loadpoint/manual_hold_state_test.go b/go/internal/loadpoint/manual_hold_state_test.go index eadeb728a..f042bc0f8 100644 --- a/go/internal/loadpoint/manual_hold_state_test.go +++ b/go/internal/loadpoint/manual_hold_state_test.go @@ -141,7 +141,7 @@ func TestManualHoldRestoreRequiresHardwareAndActiveSession(t *testing.T) { }{ {"easee:ABC", "session-1", "restored", 4140}, {"easee:ABC", "session-2", "unconfirmed", 0}, - {"easee:ABC", "", "unconfirmed", 0}, + {"easee:ABC", "", "pending", 0}, {"easee:OTHER", "session-1", "unconfirmed", 0}, } { m := sessionManager(store, "garage", "charger") diff --git a/go/internal/loadpoint/manual_limits.go b/go/internal/loadpoint/manual_limits.go index c1392c12a..c0f12a08b 100644 --- a/go/internal/loadpoint/manual_limits.go +++ b/go/internal/loadpoint/manual_limits.go @@ -40,19 +40,34 @@ func clampManualPower(cfg Config, hold ManualHold, site SiteFuse) float64 { } func manualMinChargeW(cfg Config, hold ManualHold, site SiteFuse) float64 { - floor := cfg.MinChargeW mode := hold.PhaseMode if mode == "" { mode = cfg.PhaseMode } - if (mode == "1p" || mode == "auto") && site.Phases() == 3 { - voltage := site.Voltage - if !finite(voltage) || voltage <= 0 { - voltage = 230 - } - floor = math.Max(floor/3, 6*voltage) + if mode == "1p" || mode == "auto" { + return phaseMinChargeW(cfg, mode, site) + } + return cfg.MinChargeW +} + +// phaseMinChargeW is the smallest legal offer on the phases a command names: +// 6 A on one phase for 1p or auto, 6 A on each phase for 3p. An empty mode +// names no phases; the charger decides, and its configured minimum stands. +func phaseMinChargeW(cfg Config, mode string, site SiteFuse) float64 { + if site.Phases() != 3 { + return cfg.MinChargeW + } + voltage := site.Voltage + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + switch mode { + case "1p", "auto": + return math.Max(cfg.MinChargeW/3, 6*voltage) + case "3p": + return math.Max(cfg.MinChargeW, 18*voltage) } - return floor + return cfg.MinChargeW } func (c *Controller) applyInstallationLimits(cmd map[string]any) { @@ -74,6 +89,45 @@ func (c *Controller) applyInstallationLimits(cmd map[string]any) { } } +// applyChargerPhaseLimit keeps a 1p order to a charger that can run three +// phases within the current that charger takes on one phase; otherwise a +// 25 A fuse lets 25 A onto one phase of a 16 A charger. It only lowers the +// order's limit. A 1p charger's maximum already caps its one phase, and on +// a one-phase site the maximum is the one-phase rating. +func applyChargerPhaseLimit(cmd map[string]any, cfg Config, site SiteFuse) { + if mode, _ := cmd["phase_mode"].(string); mode != "1p" || cfg.PhaseMode == "1p" || site.Phases() != 3 { + return + } + a := chargerPhaseAmps(cfg, site.Voltage) + if a < 6 { + return // Not a rating any charger has; leave the order as it is. + } + limit, _ := cmd["max_amps_per_phase"].(float64) + if !finite(limit) || limit <= 0 || limit > a { + cmd["max_amps_per_phase"] = a + } +} + +// chargerPhaseAmps is the most current the charger is known to take on one +// phase: its maximum spread over three phases, or the split below which +// auto puts it on one phase, whichever is larger. The maximum alone gives a +// one-phase charger left on an empty mode a third of its current (3,680 W +// is 5.3 A, so it never charges); the split alone gives a 32 A charger 16 A. +func chargerPhaseAmps(cfg Config, voltage float64) float64 { + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + split := cfg.PhaseSplitW + if !finite(split) || split <= 0 { + split = defaultPhaseSplitW + } + maxW := cfg.MaxChargeW + if !finite(maxW) || maxW < 0 { + maxW = 0 + } + return math.Max(maxW/(3*voltage), split/voltage) +} + // The watt order must reflect the final current ceiling too. In particular, // drivers often interpret max_amps_per_phase=0 as an absent override; send an // explicit zero-power order when the fuse leaves less than the 6 A minimum. diff --git a/go/internal/loadpoint/manual_limits_test.go b/go/internal/loadpoint/manual_limits_test.go index 3d7b5c7f2..429cdd258 100644 --- a/go/internal/loadpoint/manual_limits_test.go +++ b/go/internal/loadpoint/manual_limits_test.go @@ -86,7 +86,7 @@ func TestFusePowerBudgetNeverRoundsUp(t *testing.T) { cfg.AllowedStepsW = tc.steps c := newTestController(t, []Config{cfg}, nil, map[string]EVSample{}, &fakeSender{}) c.SetFuseEVMax(func() (float64, bool) { return tc.cap, true }) - got, _ := c.applyFuseClampAndCooldown(time.Now(), cfg, tc.want) + got, _ := c.applyFuseClamp(cfg, tc.want, "") if got != tc.expected || got > tc.cap || got > tc.want { t.Fatalf("want %v cap %v: sent %v, expected %v", tc.want, tc.cap, got, tc.expected) } diff --git a/go/internal/loadpoint/manual_status.go b/go/internal/loadpoint/manual_status.go index 427160876..c008c5c8c 100644 --- a/go/internal/loadpoint/manual_status.go +++ b/go/internal/loadpoint/manual_status.go @@ -40,7 +40,7 @@ type ManualStatus struct { // current, in its words (reason_no_current_label). Empty when it has none. ChargerReason string `json:"charger_reason,omitempty"` // LimitReason names the clamp behind ManualLimited: "fuse_limit", - // "fuse_cooldown" or "site_meter_stale". + // "import_limit", "fuse_cooldown" or "site_meter_stale". LimitReason string `json:"limit_reason,omitempty"` } @@ -108,7 +108,7 @@ const ( // before the hold was installed and says nothing about it. func holdClampReason(reason string) bool { switch reason { - case "fuse_limit", "fuse_cooldown", "site_meter_stale", "charger_limit": + case "fuse_limit", "import_limit", "fuse_cooldown", "site_meter_stale", "charger_limit": return true } return false diff --git a/go/internal/loadpoint/manual_unbound_state_test.go b/go/internal/loadpoint/manual_unbound_state_test.go index da51da1f1..34148b4b8 100644 --- a/go/internal/loadpoint/manual_unbound_state_test.go +++ b/go/internal/loadpoint/manual_unbound_state_test.go @@ -18,11 +18,16 @@ func TestManualRestrictionSurvivesRestartWithoutHardwareIdentity(t *testing.T) { restarted.ObserveSession("garage", true, 0, 1000, true, device, "") c := NewController(restarted, nil, nil, nil) c.restoreManualHoldForSession("garage") - if h, ok := c.GetManualHold("garage", time.Now()); !ok || !h.Persistent || h.PowerW != 0 { + // Only hardware identity can carry power across a restart. + want := 0.0 + if device == "easee:ABC" { + want = watts + } + if h, ok := c.GetManualHold("garage", time.Now()); !ok || !h.Persistent || h.PowerW != want { t.Fatalf("%q/%v: restart lost restriction: %+v %v", device, watts, h, ok) } st, _ := restarted.State("garage") - if st.ManualRestoreUnconfirmed != (watts > 0) { + if st.ManualRestoreUnconfirmed != (watts != want) { t.Fatalf("%q/%v: confirmation=%v", device, watts, st.ManualRestoreUnconfirmed) } if err := restarted.PersistManualHold("garage", ManualHold{}, true); err != nil { diff --git a/go/internal/loadpoint/one_phase_car.go b/go/internal/loadpoint/one_phase_car.go new file mode 100644 index 000000000..ca2c1d816 --- /dev/null +++ b/go/internal/loadpoint/one_phase_car.go @@ -0,0 +1,279 @@ +package loadpoint + +import ( + "math" + "time" +) + +// A one-phase car, as many plug-in hybrids are, draws on one phase of a +// three-phase charger: the order's current per phase, on that phase alone. +// The charger stays on three phases. The fuse already counts a three-phase +// order's current on every phase, which covers whichever phase the car +// uses. What changes is what the car gets: the plan counts one phase, and +// the order names the current per phase the car should take. +// +// The site meter shows it. When FTW's three-phase order rises by at least +// onePhaseStepA a phase, the meter's change from just before is watched over +// the charger's response time. One phase rising by about the order's change +// while the other two stay put is a one-phase car; all three rising is not. +// A house load does not follow FTW's orders, and the charger's own reading +// must have changed by what the phases show, so a kettle cannot mark a car +// and a heat pump cannot clear one. A two-phase car moves two phases and is +// never marked. With no reading from before the car drew, as after a Core +// restart, the charger's own draw is the reference: one phase carrying all +// of it and the other two less than a third, which no three- or two-phase +// car can show. + +const ( + onePhaseStepA = 2.0 // the smallest order rise that is a test + onePhaseHold = 30 * time.Second // a verdict holds this long on the meter + // onePhaseSettle is the least response time a test waits, and + // onePhaseUnknownSettle the response time of a charger that declares + // none, as the fuse limit counts it. + onePhaseSettle = time.Minute + onePhaseUnknownSettle = 3 * time.Minute +) + +// SitePhases is the site meter's current on each phase and when it read +// them. +type SitePhases struct { + A [3]float64 + At time.Time +} + +// phaseTest is one order rise being watched on the meter. +type phaseTest struct { + at, until time.Time + baseline [3]float64 // the meter just before the order took effect + baseDrawA float64 // the charger's draw then, per one phase's voltage + fromA float64 // the order per phase before the rise + verdict int // 1-3: one phase, that one; -1: three phases; 0: none + since time.Time // when the meter began to show the verdict +} + +// carPhases is one session's evidence. phase is 1-3 once the car is seen on +// that phase alone. +type carPhases struct { + orderA float64 // per phase, the order the next rise is measured from + // rising is set once the order has gone above orderA, and before is the + // meter read before that first raise went out: a fast charger has + // applied it by the next reading, and a fuse ramp climbs 1 A at a time. + rising bool + before [3]float64 + beforeDrawA float64 + // last and lastDrawA are the meter and the charger's draw read before + // this tick's command; seen is false until the session has one. + last [3]float64 + lastDrawA float64 + seen bool + test *phaseTest + steady int // the one phase the charger's whole draw shows on, 0 none + since time.Time + phase int +} + +// phaseVerdict reads one meter reading against the baseline of a rise of +// riseA a phase. drawA is the change in the charger's own power since the +// baseline, over one phase's voltage. +func phaseVerdict(amps, baseline [3]float64, riseA, drawA float64) int { + var moved [3]float64 + k, big := 0, math.Inf(-1) + for i := range amps { + moved[i] = amps[i] - baseline[i] + if moved[i] > big { + k, big = i, moved[i] + } + } + // The car's phase rises by the order's rise at most, and by at least + // half of it once the charger has applied it. + if !finite(big) || big < math.Max(onePhaseStepA, riseA/2) || big > riseA+onePhaseStepA { + return 0 + } + quiet, rose := 0, 0 + for i, m := range moved { + if i == k { + continue + } + if math.Abs(m) <= math.Max(1, big/4) { + quiet++ + } + if m >= big/2 { + rose++ + } + } + sum := moved[0] + moved[1] + moved[2] + switch { + case !finite(drawA): + return 0 + case rose == 2 && drawA >= 0.6*sum && drawA <= 1.5*sum: + return -1 + case quiet == 2 && drawA >= 0.6*big && drawA <= 1.5*big: + return k + 1 + } + return 0 +} + +// steadyVerdict is the check with no baseline: the phase (1-3) that carries +// all of the charger's draw while the other two carry less than a third of +// it, or 0. drawA is the charger's draw over one phase's voltage. +func steadyVerdict(amps [3]float64, drawA float64) int { + if !finite(drawA) || drawA < 6 { + return 0 + } + phase, quiet := 0, 0 + for i, a := range amps { + switch { + case a >= drawA-1: + phase = i + 1 + case a < drawA/3-1: + quiet++ + } + } + if quiet == 2 { + return phase + } + return 0 +} + +// SetSitePhases wires the site meter's phase currents for the one-phase car +// check. nil turns the check off. +func (c *Controller) SetSitePhases(f func() (SitePhases, bool)) { + if c != nil { + c.sitePhases = f + } +} + +// observePhases runs this tick's one-phase car check on a fresh meter +// reading. The response time is the charger's declared power age, at least +// onePhaseSettle, or onePhaseUnknownSettle when it declares none. +func (c *Controller) observePhases(cfg Config, sample EVSample, now time.Time) { + site := c.siteFuse() + if c.sitePhases == nil || !sample.Connected || site.Phases() != 3 { + return + } + meter, ok := c.sitePhases() + if !ok || meter.At.After(now.Add(time.Second)) || now.Sub(meter.At) > 10*time.Second { + return + } + voltage := site.Voltage + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + drawA := math.NaN() + if !sample.PowerUnavailable && finite(sample.PowerW) { + drawA = sample.PowerW / voltage + } + settle := onePhaseUnknownSettle + if sample.PowerMaxAge > 0 { + settle = max(onePhaseSettle, sample.PowerWindow()) + } + c.manager.observePhases(cfg.ID, meter, drawA, voltage, settle, now) +} + +// observePhases starts a test when the session's three-phase order rises +// by onePhaseStepA a phase, and judges it on the meter until the charger's +// response time and onePhaseHold have passed. The order may rise further +// meanwhile, as a plan catching up does; the test then expects that rise. +// A test whose order falls back is dropped. A one-phase verdict marks the +// session's car; a three-phase one clears the mark. +func (m *Manager) observePhases(id string, meter SitePhases, drawA, voltage float64, settle time.Duration, now time.Time) { + m.mu.Lock() + defer m.mu.Unlock() + lp, ok := m.byID[id] + if !ok || !lp.pluggedIn { + return + } + s := &lp.carPhases + previous, previousDrawA, seen := s.last, s.lastDrawA, s.seen + s.last, s.lastDrawA, s.seen = meter.A, drawA, true + if !lp.commandedKnown || lp.commandedPhases != 3 || !finite(lp.commandedW) { + s.test, s.rising, s.steady = nil, false, 0 + return + } + orderA := math.Max(0, lp.commandedW) / (3 * voltage) + if t := s.test; t != nil && (orderA-t.fromA < onePhaseStepA || now.After(t.until)) { + s.test = nil + } + if s.test == nil { + switch { + case orderA <= s.orderA: + s.orderA, s.rising = orderA, false // A cut is the base of the next rise. + case !seen: + s.orderA = orderA // The order was out before the first reading. + case !s.rising: + s.rising, s.before, s.beforeDrawA = true, previous, previousDrawA + } + if !s.rising || orderA-s.orderA < onePhaseStepA { + m.observeSteadyPhase(s, meter.A, drawA, now) + return + } + s.test = &phaseTest{at: now, until: now.Add(settle + onePhaseHold), baseline: s.before, + baseDrawA: s.beforeDrawA, fromA: s.orderA} + s.orderA, s.rising, s.steady = orderA, false, 0 + } + t := s.test + s.orderA = max(s.orderA, orderA) + if meter.At.Before(t.at) { + return + } + if v := phaseVerdict(meter.A, t.baseline, orderA-t.fromA, drawA-t.baseDrawA); v != t.verdict { + t.verdict, t.since = v, now + } + if t.verdict == 0 || now.Sub(t.since) < onePhaseHold { + return + } + s.phase = max(t.verdict, 0) + s.test = nil +} + +// observeSteadyPhase marks an unmarked car whose charger's whole draw has +// shown on one phase alone for onePhaseHold. Caller holds m.mu. +func (m *Manager) observeSteadyPhase(s *carPhases, amps [3]float64, drawA float64, now time.Time) { + if s.phase != 0 { + return + } + if v := steadyVerdict(amps, drawA); v != s.steady { + s.steady, s.since = v, now + } + if s.steady != 0 && now.Sub(s.since) >= onePhaseHold { + s.phase = s.steady + } +} + +// chargerPhaseMaxA is the most current a charger offers on each phase of a +// three-phase order: its maximum over three phases, within the site fuse. +func chargerPhaseMaxA(cfg Config, site SiteFuse) float64 { + voltage := site.Voltage + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + maxA := 32.0 + if finite(cfg.MaxChargeW) && cfg.MaxChargeW > 0 { + maxA = math.Floor(cfg.MaxChargeW/(3*voltage) + 1e-9) + } + if finite(site.MaxAmps) && site.MaxAmps > 0 { + maxA = math.Min(maxA, math.Floor(site.MaxAmps)) + } + return maxA +} + +// OnePhaseCarSteps are the levels a one-phase car takes from a three-phase +// charger: 6 A up to the charger's current per phase, on one phase. +func OnePhaseCarSteps(cfg Config, site SiteFuse) []float64 { + voltage := site.Voltage + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + steps := []float64{0} + for a := 6.0; a <= chargerPhaseMaxA(cfg, site); a++ { + steps = append(steps, a*voltage) + } + return steps +} + +// OnePhaseCarMaxW is the most a one-phase car takes from the charger: its +// current per phase on one phase. +func OnePhaseCarMaxW(cfg Config, site SiteFuse) float64 { + steps := OnePhaseCarSteps(cfg, site) + return steps[len(steps)-1] +} diff --git a/go/internal/loadpoint/one_phase_car_test.go b/go/internal/loadpoint/one_phase_car_test.go new file mode 100644 index 000000000..35a268884 --- /dev/null +++ b/go/internal/loadpoint/one_phase_car_test.go @@ -0,0 +1,350 @@ +package loadpoint + +import ( + "context" + "encoding/json" + "math" + "slices" + "testing" + "time" +) + +func TestPhaseVerdict(t *testing.T) { + base := [3]float64{1.6, 1.7, 2.3} + plus := func(a, b, c float64) [3]float64 { return [3]float64{base[0] + a, base[1] + b, base[2] + c} } + for _, c := range []struct { + name string + amps [3]float64 + drawA float64 + want int + }{ + {"one-phase car on L1", plus(7, 0, 0), 7, 1}, + {"one-phase car on L3", plus(0, 0.3, 7), 7, 3}, + {"car limited below the order", plus(5, 0, 0), 5, 1}, + {"three-phase car", plus(7, 7, 7), 21, -1}, + {"two-phase car", plus(7, 7, 0), 14, 0}, + {"kettle while the car draws nothing", plus(9, 0, 0), 0, 0}, + {"car and kettle on one phase", plus(16, 0, 0), 7, 0}, + {"not applied yet", plus(0, 0, 0), 0, 0}, + } { + if got := phaseVerdict(c.amps, base, 7, c.drawA); got != c.want { + t.Errorf("%s: verdict %d, want %d", c.name, got, c.want) + } + } +} + +// The meter marks the session's car: a three-phase order rises by 7 A a +// phase, and 40 s later only L1 rises by 7 A while the charger reads 1.6 kW. +// A three-phase verdict clears it, and the next car starts unknown. +func TestOnePhaseCarMarkedFromTheMeter(t *testing.T) { + m := NewManager() + m.Load([]Config{{ID: "garage", DriverName: "easee", MinChargeW: 1600, MaxChargeW: 11100}}) + m.Observe("garage", true, 0, 0, true) + start := time.Date(2026, 10, 22, 3, 0, 0, 0, time.UTC) + base := [3]float64{1.6, 1.7, 2.3} + order := func(w float64) { m.SetCommanded("garage", w, "plan"); m.setCommandedPhases("garage", 3) } + run := func(from, to time.Duration, amps func(time.Duration) [3]float64, drawW float64) { + for d := from; d <= to; d += 5 * time.Second { + at := start.Add(d) + m.observePhases("garage", SitePhases{A: amps(d), At: at}, drawW/230, 230, time.Minute, at) + } + } + state := func() State { st, _ := m.State("garage"); return st } + still := func(time.Duration) [3]float64 { return base } + + order(0) + run(0, 10*time.Second, still, 0) + order(4830) + onL1 := func(d time.Duration) [3]float64 { + if d < 55*time.Second { + return base + } + return [3]float64{base[0] + 7, base[1], base[2]} + } + run(15*time.Second, 80*time.Second, onL1, 1610) + if st := state(); st.CarPhases != 0 { + t.Fatal("marked before the meter showed the rise for 30 s") + } + run(85*time.Second, 90*time.Second, onL1, 1610) + if st := state(); st.CarPhases != 1 || st.CarPhase != 1 { + t.Fatalf("after 35 s on L1 alone: car phases %d on L%d", st.CarPhases, st.CarPhase) + } + // A rise that shows on all three phases, with the charger's draw rising + // by their sum, is a three-phase car: here one marked by mistake. + threePhase := func(a float64) func(time.Duration) [3]float64 { + return func(time.Duration) [3]float64 { return [3]float64{base[0] + a, base[1] + a, base[2] + a} } + } + run(95*time.Second, 120*time.Second, threePhase(7), 3*1610) + order(11040) + allThree := func(d time.Duration) [3]float64 { + if d < 160*time.Second { + return threePhase(7)(d) + } + return threePhase(16)(d) + } + run(125*time.Second, 200*time.Second, allThree, 11040) + if st := state(); st.CarPhases != 0 { + t.Fatalf("a rise on all three phases left the mark on L%d", st.CarPhase) + } + order(4830) + run(205*time.Second, 210*time.Second, still, 0) + order(11040) + run(215*time.Second, 300*time.Second, onL1, 1610) + m.Observe("garage", false, 0, 0, true) + m.Observe("garage", true, 0, 0, true) + if st := state(); st.CarPhases != 0 { + t.Fatal("the next car would inherit the mark") + } +} + +// Reviewer's P4: a one-phase car on the field table. Without the mark the +// planner got [0, 4800] and the car drew 1.6 kW of a 4.8 kW order. Marked, +// the planner gets the car's one-phase levels up to 3.7 kW. +func TestOnePhaseCarPlansItsOwnLevels(t *testing.T) { + site := SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3} + cfg := Config{MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: "auto", AllowedStepsW: easeeFieldSteps} + st := State{MinChargeW: 1600, MaxChargeW: 11100, CarMaxW: 3700} + _, maxW, steps := LimitPower(st, OnePhaseCarSteps(cfg, site)) + if maxW < 3680 || maxW > 3700 || steps[len(steps)-1] != 3680 || !slices.Contains(steps, 1380) { + t.Fatalf("one-phase car plan: max %.0f W, steps %v", maxW, steps) + } + if got := OnePhaseCarMaxW(Config{MaxChargeW: 11100}, SiteFuse{MaxAmps: 10, Voltage: 230, PhaseCnt: 3}); got != 2300 { + t.Fatalf("a 10 A fuse holds a one-phase car to %.0f W, want 2300 W", got) + } +} + +// Marked, a plan slot at the car's 3.7 kW goes out as a three-phase order +// of 16 A a phase, and the command says the car should draw a third of it. +func TestOnePhaseCarOrderNamesItsCurrent(t *testing.T) { + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: "auto", AllowedStepsW: easeeFieldSteps} + slot := time.Now().Add(-time.Second) + dir := &Directive{SlotStart: slot, SlotEnd: slot.Add(time.Hour), LoadpointEnergyWh: map[string]float64{cfg.ID: 3680}} + var payload map[string]any + send := func(_ context.Context, _ string, p []byte) error { return json.Unmarshal(p, &payload) } + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + c := NewController(NewManager(), PlanFunc(func(time.Time) (Directive, bool) { return *dir, true }), + TelemetryFunc(func(d string) (EVSample, bool) { s, ok := samples[d]; return s, ok }), send) + c.manager.Load([]Config{cfg}) + c.SetSiteFuse(SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3}) + c.manager.Observe(cfg.ID, true, 0, 0, true) + c.manager.SetSchedule(cfg.ID, Schedule{SoC: .8}) + c.manager.byID[cfg.ID].carPhases.phase = 1 + c.Tick(context.Background(), time.Now()) + power, _ := payload["power_w"].(float64) + expected, _ := payload["expected_w"].(float64) + if payload["phase_mode"] != "3p" || power != 11100 || math.Abs(expected-3700) > 1 { + t.Fatalf("sent %v", payload) + } +} + +// phaseRig feeds a session's meter and charger readings every 5 s. +type phaseRig struct { + m *Manager + start time.Time +} + +func newPhaseRig() *phaseRig { + m := NewManager() + m.Load([]Config{{ID: "garage", DriverName: "easee", MinChargeW: 4140, MaxChargeW: 11040}}) + m.Observe("garage", true, 0, 0, true) + return &phaseRig{m: m, start: time.Date(2026, 10, 22, 3, 0, 0, 0, time.UTC)} +} + +func (r *phaseRig) order(w float64) { + r.m.SetCommanded("garage", w, "plan") + r.m.setCommandedPhases("garage", 3) +} + +func (r *phaseRig) run(from, to time.Duration, amps func(time.Duration) [3]float64, drawW func(time.Duration) float64) { + for d := from; d <= to; d += 5 * time.Second { + at := r.start.Add(d) + r.m.observePhases("garage", SitePhases{A: amps(d), At: at}, drawW(d)/230, 230, time.Minute, at) + } +} + +func (r *phaseRig) carPhases() int { st, _ := r.m.State("garage"); return st.CarPhases } + +// Review M1: the charger's draw is compared as a change. A first test spoilt +// by 2 A on L2 leaves a car already drawing 6 A on L1; its next 2 A rise, +// on L1 alone with the charger's draw up by 2 A, marks it. +func TestOnePhaseCarMarkedOnALaterRise(t *testing.T) { + if v := phaseVerdict([3]float64{12, 2, 2}, [3]float64{8, 2, 2}, 4, 4); v != 1 { + t.Fatalf("a 4 A rise on L1 with the car up by 4 A: verdict %d", v) + } + r := newPhaseRig() + still := func(a [3]float64) func(time.Duration) [3]float64 { return func(time.Duration) [3]float64 { return a } } + draw := func(w float64) func(time.Duration) float64 { return func(time.Duration) float64 { return w } } + r.order(0) + r.run(0, 10*time.Second, still([3]float64{2, 2, 2}), draw(0)) + r.order(4140) + r.run(15*time.Second, 35*time.Second, still([3]float64{2, 2, 2}), draw(0)) + r.run(40*time.Second, 200*time.Second, still([3]float64{8, 4, 2}), draw(1380)) + if r.carPhases() != 0 { + t.Fatal("marked through 2 A on L2") + } + r.order(8 * 690) + r.run(205*time.Second, 230*time.Second, still([3]float64{8, 4, 2}), draw(1380)) + r.run(235*time.Second, 300*time.Second, still([3]float64{10, 4, 2}), draw(1840)) + if r.carPhases() != 1 { + t.Fatal("a 2 A rise on L1 alone, followed by the charger, did not mark the car") + } +} + +// Review M2: a heat pump starting at 2 A a phase during a 2 A rise moves all +// three phases, but the charger's draw rose by 2 A, not 8: the mark stays. +func TestHeatPumpDoesNotClearAOnePhaseCar(t *testing.T) { + r := newPhaseRig() + r.m.byID["garage"].carPhases = carPhases{orderA: 6, phase: 1} + r.order(6 * 690) + base := [3]float64{8, 2, 2} + r.run(0, 10*time.Second, func(time.Duration) [3]float64 { return base }, func(time.Duration) float64 { return 1380 }) + r.order(8 * 690) + r.run(15*time.Second, 200*time.Second, func(d time.Duration) [3]float64 { + a := base + if d >= 20*time.Second { + a = [3]float64{a[0] + 2, a[1] + 2, a[2] + 2} + } + if d >= 60*time.Second { + a[0] += 2 + } + return a + }, func(d time.Duration) float64 { + if d >= 60*time.Second { + return 8 * 230 + } + return 6 * 230 + }) + if r.carPhases() != 1 { + t.Fatal("a heat pump starting during a rise cleared the one-phase car") + } +} + +// Review M3: after a Core restart the order is out and the car draws before +// the first reading. With no baseline, a one-phase car still shows its whole +// draw on one phase and less than a third on the others; a three-phase car +// never does. +func TestOnePhaseCarMarkedAfterARestart(t *testing.T) { + for _, c := range []struct { + name string + amps [3]float64 + drawW float64 + want int + }{ + {"one-phase car at 16 A on L1, house 3/1/1 A", [3]float64{19, 1, 1}, 3680, 1}, + {"three-phase car at 16 A, house 3/1/1 A", [3]float64{19, 17, 17}, 11040, 0}, + {"two-phase car at 16 A, house 3/1/1 A", [3]float64{19, 17, 1}, 7360, 0}, + } { + r := newPhaseRig() + r.order(11040) + r.run(0, 60*time.Second, func(time.Duration) [3]float64 { return c.amps }, func(time.Duration) float64 { return c.drawW }) + if got := r.carPhases(); got != c.want { + t.Errorf("%s: car phases %d, want %d", c.name, got, c.want) + } + } +} + +// Device-simulator rig s14 on 2e847104: a three-phase charger with the +// settings page's minimum, 4,140 W. Once the car was marked one-phase the +// planner got its one-phase levels with that minimum, dropped every level +// below it and planned nothing. The car's levels are the charger's +// three-phase steps divided by three, up to its one-phase maximum. +func TestOnePhaseCarPlanningLimits(t *testing.T) { + site := SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3} + planned := func(minW, maxW float64, steps []float64) (n int) { + for _, w := range steps { + if w > 0 && w >= minW && w <= maxW { + n++ + } + } + return n + } + for _, c := range []struct { + name string + cfg Config + carMaxW float64 + wantMin, wantMax float64 + wantPlannedLevels int + }{ + {"settings page's charger", Config{MinChargeW: 4140, MaxChargeW: 11000, PhaseMode: "auto"}, 0, 1380, 3450, 10}, + {"three-phase table", Config{MinChargeW: 4140, MaxChargeW: 11040, PhaseMode: "3p", AllowedStepsW: []float64{0, 4140, 6900, 11040}}, 0, 1380, 3680, 3}, + {"field table", Config{MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: "auto", AllowedStepsW: easeeFieldSteps}, 0, 1600, 10400.0 / 3, 9}, + {"minimum 4,800 W", Config{MinChargeW: 4800, MaxChargeW: 11000, PhaseMode: "auto"}, 0, 1610, 3450, 9}, + {"car's own maximum", Config{MinChargeW: 4140, MaxChargeW: 11000, PhaseMode: "auto"}, 3000, 1380, 3000, 8}, + } { + st := State{MinChargeW: c.cfg.MinChargeW, MaxChargeW: c.cfg.MaxChargeW, AllowedStepsW: c.cfg.AllowedStepsW, CarMaxW: c.carMaxW, CarPhases: 1} + minW, maxW, steps := PlanningLimits(st, c.cfg, site) + if math.Abs(minW-c.wantMin) > .5 || math.Abs(maxW-c.wantMax) > .5 || planned(minW, maxW, steps) != c.wantPlannedLevels { + t.Errorf("%s: min %.0f max %.0f, %d levels the planner keeps (%v); want %.0f, %.0f, %d", + c.name, minW, maxW, planned(minW, maxW, steps), steps, c.wantMin, c.wantMax, c.wantPlannedLevels) + } + } + // An unmarked car keeps the charger's limits. + cfg := Config{MinChargeW: 4140, MaxChargeW: 11000, PhaseMode: "auto"} + if minW, maxW, _ := PlanningLimits(State{MinChargeW: 4140, MaxChargeW: 11000}, cfg, site); minW != 4140 || maxW != 11000 { + t.Fatalf("unmarked car: %.0f-%.0f W", minW, maxW) + } +} + +// Review of #1806: every level the planner keeps for a one-phase car is a +// third of an order the charger's table has. A plan slot at that level goes +// out as that order, within the charger and the fuse, and the car draws +// what the plan counts. A third of the charger's minimum was not: on the +// field table 1,380 W became 4,140 W, which the table lacks, and 0 W went +// out; coarse tables drew up to 30 % less than the plan. +func TestOnePhaseCarLevelsAreOrders(t *testing.T) { + for _, ch := range []struct { + name string + minW, maxW float64 + steps []float64 + }{ + {"settings page", 4140, 11000, nil}, + {"field table", 1600, 11100, easeeFieldSteps}, + {"minimum 4,800 W", 4800, 11000, nil}, + {"coarse table", 4800, 11000, []float64{0, 4800, 7200, 11000}}, + {"three-phase table", 4140, 11040, []float64{0, 4140, 6900, 11040}}, + } { + for _, fuseA := range []float64{16, 25} { + site := SiteFuse{MaxAmps: fuseA, Voltage: 230, PhaseCnt: 3} + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: ch.minW, MaxChargeW: ch.maxW, PhaseMode: "auto", AllowedStepsW: ch.steps} + st := State{MinChargeW: ch.minW, MaxChargeW: ch.maxW, AllowedStepsW: ch.steps, CarPhases: 1} + minW, maxW, steps := PlanningLimits(st, cfg, site) + kept := 0 + for _, level := range steps { + if level <= 0 || level < minW || level > maxW { + continue // The planner drops it (mpc normalizedSteps). + } + kept++ + power, expected := onePhaseCarOrder(cfg, site, level) + if power <= 0 || power > ch.maxW || power/690 > fuseA+1e-6 || math.Abs(expected-level) > level/100 { + t.Errorf("%s, %g A: a plan slot at %.0f W went out as %.0f W, the car drawing %.0f W", + ch.name, fuseA, level, power, expected) + } + } + if kept == 0 { + t.Errorf("%s, %g A: the planner keeps no level of %v (%.0f-%.0f W)", ch.name, fuseA, steps, minW, maxW) + } + } + } +} + +// onePhaseCarOrder is the order a marked one-phase car's plan slot at +// levelW goes out as, with the plan's peak, and the draw it names. +func onePhaseCarOrder(cfg Config, site SiteFuse, levelW float64) (powerW, expectedW float64) { + slot := time.Now().Add(-time.Second) + dir := &Directive{SlotStart: slot, SlotEnd: slot.Add(15 * time.Minute), + LoadpointEnergyWh: map[string]float64{cfg.ID: levelW / 4}, LoadpointMaxPowerW: map[string]float64{cfg.ID: levelW}} + var payload map[string]any + send := func(_ context.Context, _ string, p []byte) error { return json.Unmarshal(p, &payload) } + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + c := NewController(NewManager(), PlanFunc(func(time.Time) (Directive, bool) { return *dir, true }), + TelemetryFunc(func(d string) (EVSample, bool) { s, ok := samples[d]; return s, ok }), send) + c.manager.Load([]Config{cfg}) + c.SetSiteFuse(site) + c.manager.Observe(cfg.ID, true, 0, 0, true) + c.manager.SetSchedule(cfg.ID, Schedule{SoC: .8}) + c.manager.byID[cfg.ID].carPhases.phase = 1 + c.Tick(context.Background(), time.Now()) + powerW, _ = payload["power_w"].(float64) + expectedW, _ = payload["expected_w"].(float64) + return powerW, expectedW +} diff --git a/go/internal/loadpoint/planned_three_phase_test.go b/go/internal/loadpoint/planned_three_phase_test.go new file mode 100644 index 000000000..83efdbc55 --- /dev/null +++ b/go/internal/loadpoint/planned_three_phase_test.go @@ -0,0 +1,189 @@ +package loadpoint + +import ( + "context" + "slices" + "testing" + "time" +) + +// The Easee step table from the 22 Oct 2026 field report: one-phase levels +// up to 3,800 W, three-phase levels from 4,800 W. +var easeeFieldSteps = []float64{0, 1600, 1900, 2100, 2400, 2600, 2800, 3100, 3300, 3500, 3800, + 4800, 5500, 6200, 6900, 7600, 8300, 9000, 9700, 10400, 11100} + +// The planner spread that night at 3.1-3.5 kW, which only one phase can +// deliver. It now gets three-phase levels only, unless the loadpoint is +// pinned to one phase or its charger cannot reach 6 A on three. +func TestPlanningStepsKeepThreePhaseLevels(t *testing.T) { + site := SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3} + for _, mode := range []string{"auto", "", "3p"} { + got := PlanningSteps(Config{MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: mode, AllowedStepsW: easeeFieldSteps}, site) + if !slices.Equal(got, []float64{0, 4800, 5500, 6200, 6900, 7600, 8300, 9000, 9700, 10400, 11100}) { + t.Fatalf("%q: planner steps %v", mode, got) + } + _, maxW, _ := LimitPower(State{MinChargeW: 1600, MaxChargeW: 11100}, got) + if minW, _, _ := LimitPower(State{MinChargeW: 1600, MaxChargeW: 11100}, got); minW != 4800 || maxW != 11100 { + t.Fatalf("%q: planner minimum %.0f W, want the lowest step", mode, minW) + } + } + if got := PlanningSteps(Config{MinChargeW: 1600, MaxChargeW: 11100, PhaseMode: "1p", AllowedStepsW: easeeFieldSteps}, site); !slices.Equal(got, easeeFieldSteps) { + t.Fatalf("a 1p pin lost its one-phase steps: %v", got) + } + if got := PlanningSteps(Config{MinChargeW: 1380, MaxChargeW: 3680, AllowedStepsW: []float64{0, 1380, 3680}}, site); !slices.Equal(got, []float64{0, 1380, 3680}) { + t.Fatalf("a charger that cannot reach 6 A on three phases lost its steps: %v", got) + } + // A table with nothing at 6 A on three phases is a one-phase charger. + oneOnly := []float64{0, 1380, 2300, 3680} + if got := PlanningSteps(Config{MinChargeW: 1380, MaxChargeW: 5520, PhaseMode: "auto", AllowedStepsW: oneOnly}, site); !slices.Equal(got, oneOnly) { + t.Fatalf("one-phase-only table: %v, want it kept", got) + } + // So is a maximum below 6 A on three phases, with no table. + if got := PlanningSteps(Config{MinChargeW: 1380, MaxChargeW: 3680}, site); !slices.Equal(got, []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680}) { + t.Fatalf("16 A one-phase charger without a table: %v, want its one-phase ladder", got) + } +} + +// What goes out for a plan slot, by the charger's table and phase mode. +// A one-phase-only table runs on one phase (master sent auto and charged); +// unset is auto, raised to the three-phase minimum on a charger that can. +func TestPlannedOrderFollowsTheChargersPhases(t *testing.T) { + site := SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3} + for _, c := range []struct { + name, mode string + steps []float64 + maxW, wh float64 + wantW float64 + wantMode string + }{ + {"one-phase-only table, auto", "auto", []float64{0, 1380, 2300, 3680}, 5520, 4140, 3680, "1p"}, + {"one-phase-only table, unset", "", []float64{0, 1380, 2300, 3680}, 5520, 2300, 2300, "1p"}, + {"unset, low slot", "", []float64{0, 1380, 2300, 3680, 4140, 6900, 11000}, 11000, 2300, 4140, "3p"}, + {"auto, low slot", "auto", []float64{0, 1380, 2300, 3680, 4140, 6900, 11000}, 11000, 2300, 4140, "3p"}, + } { + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: c.maxW, PhaseMode: c.mode, AllowedStepsW: c.steps} + cmd := runPhaseTick(t, cfg, site, c.wh) + if cmd.power != c.wantW || cmd.phaseMode != c.wantMode { + t.Errorf("%s: plan %.0f Wh/h sent %.0f W %q, want %.0f W %q", c.name, c.wh, cmd.power, cmd.phaseMode, c.wantW, c.wantMode) + } + } +} + +// "Only spare solar" with a goal stays a solar charge: 2.5 kW of surplus on +// a low-sun day goes out on one phase, not as a three-phase order an Easee +// turns into 3.3 A and stops. +func TestSolarOnlyWithGoalKeepsOnePhaseSurplus(t *testing.T) { + base := time.Date(2026, 10, 22, 12, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 11000, PhaseMode: "auto", SurplusOnly: true, + AllowedStepsW: []float64{0, 1380, 1840, 2300, 2760, 3220, 3680, 4140, 6900, 11000}} + sender := &fakeSender{} + dir := &Directive{SlotStart: base.Add(-time.Second), SlotEnd: base.Add(15 * time.Minute), LoadpointEnergyWh: map[string]float64{cfg.ID: 0}} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + c := newTestController(t, []Config{cfg}, dir, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3}) + c.manager.SetSchedule(cfg.ID, Schedule{SoC: 0.8, TimeOfDayMinUTC: 7 * 60, Recurring: true}) + c.SetSiteSurplusForEV(func() (float64, bool) { return 2500, true }) + c.SetPeakRemainingSurplusW(func() (float64, bool) { return 3000, true }) + for i := range 3 { + c.Tick(context.Background(), base.Add(time.Duration(i)*5*time.Second)) + } + cmd, _ := lastSetCurrent(sender.calls) + if cmd.power <= 0 || cmd.power > 2500 || (cmd.phaseMode != "1p" && cmd.phaseMode != "auto") { + t.Fatalf("2.5 kW of surplus sent %.0f W %q, want a one-phase order", cmd.power, cmd.phaseMode) + } +} + +func TestChargeNowRunsOnThreePhases(t *testing.T) { + site := SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3} + for _, c := range []struct { + name, cfgMode, holdMode string + powerW, maxW float64 + want string + }{ + {"auto, full power", "auto", "", 11100, 11100, "3p"}, + {"auto, below the 3p minimum", "auto", "", 1380, 11100, "auto"}, + {"unset, low hold", "", "", 1380, 11100, "auto"}, + {"unset, full power", "", "", 11100, 11100, "3p"}, + {"hold pinned to 1p", "auto", "1p", 3680, 11100, "1p"}, + {"loadpoint pinned to 1p", "1p", "", 3680, 11100, "1p"}, + {"hold auto, full power", "", "auto", 11100, 11100, "3p"}, + {"one-phase charger, unset", "", "", 3680, 3680, "1p"}, + // A client's 3p is not a pin on a charger that cannot run it. + {"one-phase charger, client sends 3p", "", "3p", 3680, 3680, "1p"}, + {"one-phase charger on auto, client sends 3p", "auto", "3p", 3680, 3680, "1p"}, + {"one-phase charger pinned 3p", "3p", "3p", 3680, 3680, "3p"}, + } { + got := holdPhaseMode(ManualHold{PowerW: c.powerW, PhaseMode: c.holdMode}, Config{MinChargeW: 1380, MaxChargeW: c.maxW, PhaseMode: c.cfgMode}, site) + if got != c.want { + t.Errorf("%s: phase_mode %q, want %q", c.name, got, c.want) + } + } +} + +// Field report 22 Oct 2026: a one-phase order of 3,500 W under auto, with +// 2.4 A of room left on the car's own phase. The site cap rose one step, +// past the 3,680 W split, so the split read the order as three-phase, +// credited it nothing, and cut it to the room alone: 1,600 W, then less, +// then 0, with every phase under its limit. Keeping the current order is +// never a cut; only a measured overage cuts. +func TestSplitKeepsTheCurrentOrder(t *testing.T) { + now := time.Date(2026, 10, 22, 3, 47, 45, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "charger", MinChargeW: 1600, MaxChargeW: 11100, + AllowedStepsW: easeeFieldSteps, PhaseMode: "auto", SurplusOnly: true} + for _, c := range []struct { + name string + mode string + orderW float64 + orderPhases int + siteCapW float64 + roomA float64 + wantMin, wantMax float64 + }{ + {"one-phase order, cap past the split", "auto", 3500, 1, 3730, 2.4, 3500, 3500}, + {"one-phase order, no room", "auto", 3500, 1, 3730, 0, 3500, 3500}, + // The site cap is the measured limit; keeping never beats it. + {"one-phase order after a measured cut", "auto", 3500, 1, 2000, -3, 0, 2000}, + {"three-phase order after a measured cut", "3p", 9700, 3, 2000, -3, 0, 2000}, + // The same watts on other phases are a rise on some of them: a + // three-phase order that now resolves to one phase gets the room + // alone, and 5,000 W moved from one phase to three adds 7.2 A to + // two phases that carried none of it. + {"three-phase order moving to one phase", "auto", 4140, 3, 3000, 2.4, 0, (6 + 2.4) * 230}, + {"one-phase order moving to three phases", "3p", 5000, 1, 20000, 2, 0, 2 * 690}, + } { + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + ctrl := newTestController(t, []Config{cfg}, nil, samples, &fakeSender{}) + ctrl.SetSiteFuse(SiteFuse{MaxAmps: 20, Voltage: 230, PhaseCnt: 3}) + ctrl.SetFuseEVMax(func() (float64, bool) { return c.siteCapW, true }) + ctrl.SetFusePhaseRoomA(func() (float64, bool) { return c.roomA, true }) + ctrl.manager.Observe(cfg.ID, true, c.orderW, 0, true) + ctrl.manager.SetCommanded(cfg.ID, c.orderW, "fuse_limit") + ctrl.manager.setCommandedPhases(cfg.ID, c.orderPhases) + ctrl.tickChoices = map[string]tickChoice{cfg.ID: {surplusOn: true, phaseMode: c.mode}} + ctrl.freezeFuseBudget(now, samples) + if got := ctrl.tickFuseCaps[cfg.ID]; got < c.wantMin || got > c.wantMax { + t.Errorf("%s: cap %.0f W, want %.0f-%.0f W", c.name, got, c.wantMin, c.wantMax) + } + } +} + +// Field matrix, hold3p: an app sends Charge now as 3p for an auto +// loadpoint. On a 32 A one-phase charger whose table stops at 3,680 W, a +// 3p order has no step and went out as 0 W. Core runs it on one phase. +func TestChargeNowOnAOnePhaseChargerIgnoresAClients3p(t *testing.T) { + base := time.Date(2026, 10, 22, 20, 0, 0, 0, time.UTC) + cfg := Config{ID: "garage", DriverName: "easee", MinChargeW: 1380, MaxChargeW: 7360, PhaseMode: "auto", + AllowedStepsW: []float64{0, 1380, 2300, 3680}} + sender := &fakeSender{} + samples := map[string]EVSample{cfg.DriverName: {Connected: true, RequestActive: true}} + c := newTestController(t, []Config{cfg}, nil, samples, sender) + c.SetSiteFuse(SiteFuse{MaxAmps: 25, Voltage: 230, PhaseCnt: 3}) + c.SetManualHold(cfg.ID, ManualHold{PowerW: cfg.MaxChargeW, PhaseMode: "3p", Persistent: true, StartedAt: base, UpdatedAt: base}) + for i := range 3 { + c.Tick(context.Background(), base.Add(time.Duration(i)*5*time.Second)) + } + cmd, ok := lastSetCurrent(sender.calls) + if !ok || cmd.power != 3680 || cmd.phaseMode != "1p" { + t.Fatalf("Charge now sent %.0f W %q, want 3680 W 1p", cmd.power, cmd.phaseMode) + } +} diff --git a/go/internal/loadpoint/planning_steps.go b/go/internal/loadpoint/planning_steps.go index b6271b18b..1e2614a2c 100644 --- a/go/internal/loadpoint/planning_steps.go +++ b/go/internal/loadpoint/planning_steps.go @@ -1,16 +1,19 @@ package loadpoint -import "math" +import ( + "math" + "slices" +) -// PlanningSteps exposes legal whole-amp offers for a scheduled charge when -// no explicit device steps were configured. Scheduled auto mode uses three -// phases, as resolvePhaseMode does. Explicit device steps keep their meaning. +// PlanningSteps exposes legal whole-amp offers for a scheduled charge. +// Scheduled charging runs on the car's phases, as resolvePhaseMode sends it: +// three on a charger that can reach 6 A on each, else one. A three-phase +// charger's explicit steps keep those at or above 6 A on its phases; one- +// phase levels in a shared table would let the plan spread a night at 3 kW +// that only one phase can deliver. Other explicit steps keep their meaning. func PlanningSteps(cfg Config, site SiteFuse) []float64 { - if len(cfg.AllowedStepsW) > 0 { - return append([]float64(nil), cfg.AllowedStepsW...) - } phases := site.Phases() - if cfg.PhaseMode == "1p" { + if !threePhaseCapable(cfg, site) { phases = 1 } voltage := site.Voltage @@ -18,6 +21,18 @@ func PlanningSteps(cfg Config, site SiteFuse) []float64 { voltage = 230 } step := voltage * float64(phases) + if len(cfg.AllowedStepsW) > 0 { + if phases == 1 { + return append([]float64(nil), cfg.AllowedStepsW...) + } + steps := []float64{0} + for _, w := range cfg.AllowedStepsW { + if w >= 6*step && (cfg.MaxChargeW <= 0 || w <= cfg.MaxChargeW) { + steps = append(steps, w) + } + } + return steps + } minimum := math.Max(6, math.Ceil(cfg.MinChargeW/step)) maximum := math.Floor(cfg.MaxChargeW / step) if site.MaxAmps > 0 { @@ -33,3 +48,59 @@ func PlanningSteps(cfg Config, site SiteFuse) []float64 { } return steps } + +// PlanningLimits is the power range and the steps the planner gets for a +// loadpoint's session, held to the car by LimitPower. A one-phase car on a +// three-phase charger takes a third of each three-phase order, so its +// levels are the charger's three-phase steps divided by three, up to its +// one-phase maximum: each is a third of an order the charger's table and +// minimum allow. The charger's own minimum would leave the planner no level. +func PlanningLimits(st State, cfg Config, site SiteFuse) (minW, maxW float64, steps []float64) { + planning := PlanningSteps(cfg, site) + if st.CarPhases == 1 && threePhaseCapable(cfg, site) { + topW := OnePhaseCarMaxW(cfg, site) + levels := []float64{0} + for _, w := range planning { + if w > 0 && w/3 <= topW+1e-9 { + levels = append(levels, w/3) + } + } + if len(levels) > 1 { + st.MinChargeW, st.MaxChargeW = slices.Min(levels[1:]), slices.Max(levels) + return LimitPower(st, levels) + } + } + return LimitPower(st, planning) +} + +// ThreePhaseCapable reports whether a charger can run three phases at 6 A +// each; see threePhaseCapable. +func ThreePhaseCapable(cfg Config, site SiteFuse) bool { return threePhaseCapable(cfg, site) } + +// threePhaseCapable reports whether a charger can run three phases at 6 A +// each: a three-phase site, no 1p pin, and a step table, or else a maximum, +// that reaches 6 A on every phase. A table with nothing that high is a +// one-phase charger whatever its mode says. A one-phase charger above 16 A +// with such a table needs phase_mode 1p. +func threePhaseCapable(cfg Config, site SiteFuse) bool { + if site.Phases() != 3 || cfg.PhaseMode == "1p" { + return false + } + voltage := site.Voltage + if !finite(voltage) || voltage <= 0 { + voltage = 230 + } + minimum := 18 * voltage + if finite(cfg.MaxChargeW) && cfg.MaxChargeW > 0 && cfg.MaxChargeW < minimum { + return false + } + if len(cfg.AllowedStepsW) == 0 { + return true + } + for _, w := range cfg.AllowedStepsW { + if w >= minimum { + return true + } + } + return false +} diff --git a/go/internal/ocpp/charging_needs.go b/go/internal/ocpp/charging_needs.go index 3a2a76dd7..3e0f2f69c 100644 --- a/go/internal/ocpp/charging_needs.go +++ b/go/internal/ocpp/charging_needs.go @@ -198,7 +198,7 @@ func (h *handlerV201) OnNotifyEVChargingNeeds(id string, req *smartcharging.Noti "departure", n.DepartureTime) h.noteChargingNeeds(id, n) - h.telSuccess(id) + h.telHealth(id) // from the car, so no proof of Available return smartcharging.NewNotifyEVChargingNeedsResponse(smartcharging.EVChargingNeedsStatusAccepted), nil } @@ -207,7 +207,7 @@ func (h *handlerV201) OnNotifyEVChargingNeeds(id string, req *smartcharging.Noti // acknowledged and dropped — accepting it costs nothing and refusing would // make the charger retry forever. func (h *handlerV201) OnNotifyEVChargingSchedule(id string, _ *smartcharging.NotifyEVChargingScheduleRequest) (*smartcharging.NotifyEVChargingScheduleResponse, error) { - h.telSuccess(id) + h.telHealth(id) return smartcharging.NewNotifyEVChargingScheduleResponse(types201.GenericStatusAccepted), nil } diff --git a/go/internal/ocpp/config.go b/go/internal/ocpp/config.go index c6708d174..9a6fa9fad 100644 --- a/go/internal/ocpp/config.go +++ b/go/internal/ocpp/config.go @@ -128,8 +128,11 @@ func (c *Config) Defaults() { if c.Path == "" { c.Path = "/" } + // Half the 60 s watchdog and history age limit: an idle charger often + // sends nothing else, so one late heartbeat would otherwise mark it + // offline and its Available 0 W stale. if c.HeartbeatIntervalS == 0 { - c.HeartbeatIntervalS = 60 + c.HeartbeatIntervalS = 30 } } diff --git a/go/internal/ocpp/config_test.go b/go/internal/ocpp/config_test.go new file mode 100644 index 000000000..05adf19c6 --- /dev/null +++ b/go/internal/ocpp/config_test.go @@ -0,0 +1,26 @@ +package ocpp + +import ( + "testing" + + "github.com/lorenzodonini/ocpp-go/ocpp1.6/core" + + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// Core asks chargers for a heartbeat at half the 60 s watchdog unless the +// owner set one; a saved value wins. +func TestHeartbeatDefaultIsHalfTheWatchdog(t *testing.T) { + for _, tc := range []struct{ set, want int }{{0, 30}, {60, 60}, {120, 120}} { + c := Config{HeartbeatIntervalS: tc.set} + c.Defaults() + if c.HeartbeatIntervalS != tc.want { + t.Fatalf("heartbeat_interval_s %d: got %d, want %d", tc.set, c.HeartbeatIntervalS, tc.want) + } + h := NewHandler(telemetry.NewStore(), c.HeartbeatIntervalS) + conf, err := h.OnBootNotification("charger", &core.BootNotificationRequest{ChargePointVendor: "Rig", ChargePointModel: "One"}) + if err != nil || conf.Interval != tc.want { + t.Fatalf("heartbeat_interval_s %d: boot asked for %d s (%v)", tc.set, conf.Interval, err) + } + } +} diff --git a/go/internal/ocpp/forecast_power_test.go b/go/internal/ocpp/forecast_power_test.go index c0f61cc34..53016e05d 100644 --- a/go/internal/ocpp/forecast_power_test.go +++ b/go/internal/ocpp/forecast_power_test.go @@ -9,8 +9,10 @@ import ( "github.com/lorenzodonini/ocpp-go/ocpp1.6/core" types16 "github.com/lorenzodonini/ocpp-go/ocpp1.6/types" + "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/authorization" "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/availability" "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/meter" + "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/provisioning" "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/transactions" types201 "github.com/lorenzodonini/ocpp-go/ocpp2.0.1/types" "github.com/srcfl/ftw/go/internal/telemetry" @@ -23,8 +25,17 @@ type forecastOCPPFixture struct { energy func() status func() available func() - stop func() - phases func(l1, l2, l3 float64, at time.Time) + // availableAt sends Available with the charger's time; zero sends none. + availableAt func(time.Time) + // unavailable and faulted leave power unknown; heartbeat is the charger's. + unavailable func() + faulted func() + heartbeat func() + authorize func() + boot func() + start func() + stop func() + phases func(l1, l2, l3 float64, at time.Time) } func newForecastOCPPFixture(t *testing.T, version string) forecastOCPPFixture { @@ -44,9 +55,25 @@ func newForecastOCPPFixture(t *testing.T, version string) forecastOCPPFixture { f.status = func() { h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusCharging}) } - f.available = func() { - h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusAvailable}) + f.availableAt = func(at time.Time) { + req := &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusAvailable} + if !at.IsZero() { + req.Timestamp = types16.NewDateTime(at) + } + h.OnStatusNotification("charger", req) + } + f.unavailable = func() { + h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusUnavailable}) + } + f.authorize = func() { h.OnAuthorize("charger", &core.AuthorizeRequest{IdTag: "card"}) } + f.boot = func() { + h.OnBootNotification("charger", &core.BootNotificationRequest{ChargePointVendor: "Rig", ChargePointModel: "One"}) + } + f.start = func() { h.OnStartTransaction("charger", &core.StartTransactionRequest{ConnectorId: 1, IdTag: "card"}) } + f.faulted = func() { + h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 1, Status: core.ChargePointStatusFaulted, ErrorCode: core.GroundFailure}) } + f.heartbeat = func() { h.OnHeartbeat("charger", &core.HeartbeatRequest{}) } f.stop = func() { h.OnStopTransaction("charger", &core.StopTransactionRequest{}) } f.phases = func(l1, l2, l3 float64, at time.Time) { h.OnMeterValues("charger", &core.MeterValuesRequest{ConnectorId: 1, MeterValue: []types16.MeterValue{{Timestamp: types16.NewDateTime(at), SampledValue: []types16.SampledValue{ @@ -66,9 +93,29 @@ func newForecastOCPPFixture(t *testing.T, version string) forecastOCPPFixture { f.status = func() { v.OnStatusNotification("charger", &availability.StatusNotificationRequest{EvseID: 1, ConnectorID: 1, ConnectorStatus: availability.ConnectorStatusOccupied}) } - f.available = func() { - v.OnStatusNotification("charger", &availability.StatusNotificationRequest{EvseID: 1, ConnectorID: 1, ConnectorStatus: availability.ConnectorStatusAvailable}) + f.availableAt = func(at time.Time) { + req := &availability.StatusNotificationRequest{EvseID: 1, ConnectorID: 1, ConnectorStatus: availability.ConnectorStatusAvailable} + if !at.IsZero() { + req.Timestamp = types201.NewDateTime(at) + } + v.OnStatusNotification("charger", req) + } + f.authorize = func() { + v.OnAuthorize("charger", &authorization.AuthorizeRequest{IdToken: types201.IdToken{IdToken: "card", Type: types201.IdTokenTypeISO14443}}) + } + f.boot = func() { + v.OnBootNotification("charger", &provisioning.BootNotificationRequest{ChargingStation: provisioning.ChargingStationType{VendorName: "Rig", Model: "One"}}) } + f.start = func() { + v.OnTransactionEvent("charger", &transactions.TransactionEventRequest{EventType: transactions.TransactionEventStarted}) + } + f.unavailable = func() { + v.OnStatusNotification("charger", &availability.StatusNotificationRequest{EvseID: 1, ConnectorID: 1, ConnectorStatus: availability.ConnectorStatusUnavailable}) + } + f.faulted = func() { + v.OnStatusNotification("charger", &availability.StatusNotificationRequest{EvseID: 1, ConnectorID: 1, ConnectorStatus: availability.ConnectorStatusFaulted}) + } + f.heartbeat = func() { v.OnHeartbeat("charger", &availability.HeartbeatRequest{}) } f.stop = func() { v.OnTransactionEvent("charger", &transactions.TransactionEventRequest{EventType: transactions.TransactionEventEnded}) } @@ -80,6 +127,8 @@ func newForecastOCPPFixture(t *testing.T, version string) forecastOCPPFixture { }}}}) } } + // Chargers stamp their status; 2.0.1 requires it. + f.available = func() { f.availableAt(time.Now()) } return f } func (f forecastOCPPFixture) reading() telemetry.ForecastReading { @@ -218,8 +267,8 @@ func TestOCPPLastPowerSharedAcceptRule(t *testing.T) { if raw := f.tel.Get("charger", telemetry.DerEV); raw == nil || raw.RawW != 0 { t.Fatalf("Available left lastPowerW set: %+v", raw) } - if r := f.reading(); r.Valid { - t.Fatalf("Available published a measured zero: %+v", r) + if r := f.reading(); !r.Valid || r.EVW != 0 { + t.Fatalf("Available is not a measured zero: %+v", r) } f2 := newForecastOCPPFixture(t, version) @@ -251,3 +300,251 @@ func TestMeterPowerWPrefersUnphasedThenSum(t *testing.T) { t.Fatal("negative/NaN samples accepted") } } + +// live is the balance every client shows: the site meter, PV and the charger. +func (f forecastOCPPFixture) live() telemetry.ForecastReading { + f.tel.Update("site", telemetry.DerMeter, 5000, nil, nil) + f.tel.RecordDriverSuccess("site") + f.tel.Update("pv", telemetry.DerPV, -1000, nil, nil) + f.tel.RecordDriverSuccess("pv") + return f.tel.LiveMeasurementNow("site", telemetry.ForecastOptions{ExpectedFlows: []telemetry.ForecastFlow{{Driver: "charger", DerType: telemetry.DerEV}}}) +} + +// age moves the published charger reading back, as if no message came for d. +func (f forecastOCPPFixture) age(d time.Duration) { + f.h.mu.Lock() + s := f.h.chargers["charger"] + s.forecastPower.MeasuredAtMS -= d.Milliseconds() + s.forecastPower.ReceivedAtMS -= d.Milliseconds() + f.h.mu.Unlock() + f.h.pushReading("charger", s) +} + +func (f forecastOCPPFixture) wantMeasuredZero(t *testing.T, what string) { + t.Helper() + if r := f.live(); r.LoadQuality != telemetry.LoadMeasured || r.EVW != 0 || r.LoadW != 6000 { + t.Fatalf("%s: live Home use %s (%s), EV %v W, load %v W", what, r.LoadQuality, r.LoadReason, r.EVW, r.LoadW) + } + if r := f.reading(); !r.Valid || !r.Known[telemetry.DerEV] || r.EVW != 0 { + t.Fatalf("%s: forecast balance %+v", what, r) + } + if raw := f.tel.Get("charger", telemetry.DerEV); raw == nil || raw.RawW != 0 { + t.Fatalf("%s: dispatch power %+v", what, raw) + } +} + +func TestOCPPAvailableIsAMeasuredZeroInBothVersions(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + t.Run(version, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.available() + f.wantMeasuredZero(t, "Available after boot") + + // No message for two minutes: the zero is as old as its evidence. + f.age(2 * time.Minute) + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured || !strings.HasSuffix(r.LoadReason, ":charger:ev") { + t.Fatalf("an old Available stayed current: %s (%s)", r.LoadQuality, r.LoadReason) + } + f.heartbeat() + f.wantMeasuredZero(t, "heartbeat") + + // A car arrives. Its draw is unknown until the meter reports. + f.status() + f.heartbeat() + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured { + t.Fatalf("the zero outlived Available: %s", r.LoadReason) + } + f.available() + f.wantMeasuredZero(t, "Available again") + + // A new socket says nothing about the connector until it reports. + f.h.OnDisconnect("charger") + f.h.OnConnect("charger") + f.heartbeat() + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured { + t.Fatalf("a reconnect kept the zero before any status: %s", r.LoadReason) + } + }) + } +} + +func TestOCPPAvailableAfterSessionIsAMeasuredZero(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + for _, order := range []string{"stop first", "Available first"} { + t.Run(version+" "+order, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.status() + f.power(3000, time.Now()) + if r := f.live(); r.LoadQuality != telemetry.LoadMeasured || r.EVW != 3000 || r.LoadW != 3000 { + t.Fatalf("charging with MeterValues: %s (%s), EV %v W, load %v W", r.LoadQuality, r.LoadReason, r.EVW, r.LoadW) + } + if r := f.reading(); !r.Valid || r.EVW != 3000 { + t.Fatalf("charging forecast balance: %+v", r) + } + lastCharge := time.Now() + time.Sleep(2 * time.Millisecond) + if order == "stop first" { + f.stop() + f.available() + } else { + f.available() + f.stop() + } + f.wantMeasuredZero(t, "after the session") + + // A meter sample from before the car left cannot bring its draw back. + f.power(3000, lastCharge.Add(time.Millisecond)) + f.wantMeasuredZero(t, "late session sample") + if version == "2.0.1" { + // A periodic update with no car keeps the zero until Ended. + v := &handlerV201{f.h} + v.OnTransactionEvent("charger", &transactions.TransactionEventRequest{EventType: transactions.TransactionEventUpdated, TransactionInfo: transactions.Transaction{ChargingState: transactions.ChargingStateIdle}}) + f.wantMeasuredZero(t, "idle transaction update") + } + f.heartbeat() + f.wantMeasuredZero(t, "heartbeat after the session") + }) + } + } +} + +func TestOCPPUnavailableAndFaultedStayUnknown(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + for name, status := range map[string]func(forecastOCPPFixture){ + "Unavailable": func(f forecastOCPPFixture) { f.unavailable() }, + "Faulted": func(f forecastOCPPFixture) { f.faulted() }, + } { + t.Run(version+" "+name, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.available() + status(f) + f.heartbeat() + r := f.live() + if r.LoadQuality != telemetry.LoadEstimated || !strings.HasPrefix(r.LoadReason, "unknown_power:charger:") { + t.Fatalf("%s became known power: %s (%s)", name, r.LoadQuality, r.LoadReason) + } + if r := f.reading(); r.Valid || !strings.HasPrefix(r.Reason, "unknown_power:charger:") { + t.Fatalf("%s became a measured balance: %+v", name, r) + } + }) + } + } +} + +// Connector 0 is the charge point. Its Available is not a cable status. +func TestOCPPChargePointAvailableIsNotACableStatus(t *testing.T) { + f := newForecastOCPPFixture(t, "1.6") + chargePointAvailable := func() { + f.h.OnStatusNotification("charger", &core.StatusNotificationRequest{ConnectorId: 0, Status: core.ChargePointStatusAvailable}) + } + chargePointAvailable() + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured { + t.Fatal("connector 0 Available became a measured zero") + } + if raw := f.tel.Get("charger", telemetry.DerEV); raw == nil || !strings.Contains(string(raw.Data), `"connection_unknown":true`) { + t.Fatalf("connector 0 Available reported the cable: %+v", raw) + } + + f.status() + f.power(3000, time.Now()) + chargePointAvailable() + if r := f.live(); r.LoadQuality != telemetry.LoadMeasured || r.EVW != 3000 { + t.Fatalf("connector 0 Available cleared a charging car: %s (%s), EV %v W", r.LoadQuality, r.LoadReason, r.EVW) + } + if v := f.h.Snapshot()["charger"]; !v.Connected || !v.Charging { + t.Fatalf("connector 0 Available unplugged the car: %+v", v) + } + + time.Sleep(2 * time.Millisecond) + f.available() + chargePointAvailable() + f.age(2 * time.Minute) + f.heartbeat() + f.wantMeasuredZero(t, "connector 0 after connector 1 Available") +} + +func TestOCPPAvailableConfirmsOnlyAZero(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + t.Run(version, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.available() + time.Sleep(2 * time.Millisecond) + f.power(3000, time.Now()) + f.age(2 * time.Minute) + f.heartbeat() + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured || !strings.HasSuffix(r.LoadReason, ":charger:ev") { + t.Fatalf("a heartbeat kept an old 3000 W current: %s (%s), EV %v W", r.LoadQuality, r.LoadReason, r.EVW) + } + }) + } +} + +func TestOCPPAvailableGateFollowsTheChargerClock(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + for name, stamp := range map[string]func(time.Time) time.Time{ + "status time": func(now time.Time) time.Time { return now }, + "no status time": func(time.Time) time.Time { return time.Time{} }, + } { + t.Run(version+" "+name, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.h.mu.Lock() + f.h.chargers["charger"].powerConnectedAt = time.Now().Add(-time.Hour) + f.h.mu.Unlock() + // The charger's clock is 30 s behind Core's. The car plugs in at once. + behind := time.Now().Add(-30 * time.Second) + f.availableAt(stamp(behind)) + f.status() + f.power(3000, behind.Add(time.Second)) + if r := f.live(); r.LoadQuality != telemetry.LoadMeasured || r.EVW != 3000 { + t.Fatalf("the car's first sample was rejected: %s (%s), EV %v W", r.LoadQuality, r.LoadReason, r.EVW) + } + }) + } + } +} + +func TestOCPPZeroEndsWhenTheCarArrives(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + for name, arrive := range map[string]func(forecastOCPPFixture){ + "status": func(f forecastOCPPFixture) { f.status() }, + "transaction": func(f forecastOCPPFixture) { f.start() }, + } { + t.Run(version+" "+name, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.available() + arrive(f) + r := f.live() + if r.LoadQuality == telemetry.LoadMeasured || !strings.HasPrefix(r.LoadReason, "unknown_power:charger:") { + t.Fatalf("the car arrived and 0 W stayed measured: %s (%s)", r.LoadQuality, r.LoadReason) + } + if r := f.reading(); r.Valid { + t.Fatalf("the car arrived and 0 W stayed a training balance: %+v", r) + } + time.Sleep(2 * time.Millisecond) + f.power(3000, time.Now()) + if r := f.live(); r.LoadQuality != telemetry.LoadMeasured || r.EVW != 3000 { + t.Fatalf("the car's sample after Available: %s (%s), EV %v W", r.LoadQuality, r.LoadReason, r.EVW) + } + }) + } + } +} + +func TestOCPPAnyMessageConfirmsAvailable(t *testing.T) { + for _, version := range []string{"1.6", "2.0.1"} { + t.Run(version, func(t *testing.T) { + f := newForecastOCPPFixture(t, version) + f.available() + f.age(2 * time.Minute) + f.authorize() + f.wantMeasuredZero(t, "Authorize") + + // A boot may follow a reset; the charger reports its statuses again. + f.age(2 * time.Minute) + f.boot() + if r := f.live(); r.LoadQuality == telemetry.LoadMeasured { + t.Fatalf("a boot confirmed Available: %s", r.LoadReason) + } + }) + } +} diff --git a/go/internal/ocpp/handlers.go b/go/internal/ocpp/handlers.go index c06dd4435..64b9ccd76 100644 --- a/go/internal/ocpp/handlers.go +++ b/go/internal/ocpp/handlers.go @@ -79,7 +79,16 @@ type chargerState struct { sessionMeterWh float64 lastPowerW float64 forecastPower telemetry.ForecastPowerSample - powerConnectedAt time.Time + // meterAtMS is the newest meter sample time accepted. A heartbeat moves + // forecastPower's time to Core's clock; this gate stays on the meter's. + meterAtMS int64 + // available: the connector reported Available on this socket. With no + // vehicle no current flows, so its power is a known 0 W. OCPP reports a + // status change when it happens, so any later message from the charger + // confirms the reading. State is per charger, not per connector: on a + // two-connector charger an Available on one claims 0 W for both. + available bool + powerConnectedAt time.Time // lastAmps is the most recent per-phase limit this charger accepted. // A resume with no rate of its own restores it. lastAmps float64 @@ -203,7 +212,16 @@ func (h *Handler) isApproved(id string) bool { // The tel* wrappers are the quarantine choke points: every telemetry write // for a charger goes through one of them, so a pending charger is dropped // here once rather than gated at each call site. +// telSuccess records a message from the charger. While the connector is +// Available, any message also confirms its 0 W. func (h *Handler) telSuccess(id string) { + h.telHealth(id) + h.confirmAvailable(id) +} + +// telHealth records a message without confirming the connector status: a +// new socket or a boot says nothing about it until the charger reports. +func (h *Handler) telHealth(id string) { if h.isApproved(id) { h.tel.RecordDriverSuccess(id) } @@ -413,7 +431,7 @@ func (h *Handler) OnConnect(id string) { h.scheduleIdentityProbeLocked(id, s) h.mu.Unlock() h.pushReading(id, s) - h.telSuccess(id) + h.telHealth(id) h.maybeProbeCapability(id) } @@ -464,7 +482,7 @@ func (h *Handler) OnBootNotification(id string, req *core.BootNotificationReques s.firmware = req.FirmwareVersion h.mu.Unlock() h.noteIdentity(id) - h.telSuccess(id) + h.telHealth(id) h.maybeProbeCapability(id) return core.NewBootNotificationConfirmation( types.NewDateTime(time.Now()), @@ -478,10 +496,23 @@ func (h *Handler) OnHeartbeat(id string, _ *core.HeartbeatRequest) (*core.Heartb return core.NewHeartbeatConfirmation(types.NewDateTime(time.Now())), nil } +// confirmAvailable publishes the 0 W of an Available connector again at the +// time of a message from the charger. telSuccess calls it for both dialects. +func (h *Handler) confirmAvailable(id string) { + s := h.state(id) + h.mu.Lock() + confirmed := s.confirmAvailablePower(time.Now()) + h.mu.Unlock() + if confirmed { + h.pushReading(id, s) + } +} + func (h *Handler) OnAuthorize(id string, req *core.AuthorizeRequest) (*core.AuthorizeConfirmation, error) { // Phase 1: auto-authorize every tag. RFID gating is a Phase 2 feature // alongside RemoteStartTransaction. slog.Debug("OCPP authorize", "charger", id, "tag", req.IdTag) + h.confirmAvailable(id) return core.NewAuthorizationConfirmation( types.NewIdTagInfo(types.AuthorizationStatusAccepted), ), nil @@ -490,15 +521,32 @@ func (h *Handler) OnAuthorize(id string, req *core.AuthorizeRequest) (*core.Auth func (h *Handler) OnDataTransfer(id string, req *core.DataTransferRequest) (*core.DataTransferConfirmation, error) { // Vendor-specific extensions — accept gracefully but do nothing. slog.Debug("OCPP DataTransfer ignored", "charger", id, "vendor", req.VendorId) + h.confirmAvailable(id) return core.NewDataTransferConfirmation(core.DataTransferStatusUnknownVendorId), nil } func (h *Handler) OnStatusNotification(id string, req *core.StatusNotificationRequest) (*core.StatusNotificationConfirmation, error) { + // Connector 0 is the charge point itself. Its Available, sent at boot and + // on reconnect, says the controller is up, not that a cable is out. + // Unavailable and Faulted there cover every connector. + if req.ConnectorId == 0 && req.Status == core.ChargePointStatusAvailable { + slog.Info("OCPP status", "charger", id, "connector", req.ConnectorId, "status", req.Status) + h.telSuccess(id) + return core.NewStatusNotificationConfirmation(), nil + } s := h.state(id) h.mu.Lock() s.connectedKnown = true + if req.Status != core.ChargePointStatusAvailable { + s.leaveAvailable() + } switch req.Status { - case core.ChargePointStatusAvailable, core.ChargePointStatusUnavailable: + case core.ChargePointStatusAvailable: + s.connected = false + s.charging = false + s.holdAvailablePower(time.Now(), statusTime(req.Timestamp)) + case core.ChargePointStatusUnavailable: + // Out of service, maybe broken: no evidence that nothing flows. s.connected = false s.charging = false s.clearMeasuredPower() @@ -590,6 +638,7 @@ func (h *Handler) OnStartTransaction(id string, req *core.StartTransactionReques s.sessionMeterWh = 0 s.connected = true s.connectedKnown = true + s.leaveAvailable() s.charging = true h.mu.Unlock() @@ -612,7 +661,10 @@ func (h *Handler) OnStopTransaction(id string, req *core.StopTransactionRequest) sessionWh := float64(req.MeterStop) - s.sessionStartMeterWh s.transactionID = -1 s.charging = false - s.clearMeasuredPower() + // A charger may report Available before the stop. The car is gone then. + if !s.available { + s.clearMeasuredPower() + } s.sessionMeterWh = sessionWh h.mu.Unlock() @@ -710,16 +762,66 @@ func meterPowerW(samples []powerSample) (float64, bool) { // accepted timestamp. Samples from before this socket or in the future cannot // revive a stale or synthesized reading. The caller holds h.mu. func (s *chargerState) recordPower(w float64, measured, received time.Time) bool { - if math.IsNaN(w) || math.IsInf(w, 0) || w < 0 || measured.IsZero() || measured.After(received) || measured.Before(s.powerConnectedAt) || measured.UnixMilli() <= s.forecastPower.MeasuredAtMS { + if math.IsNaN(w) || math.IsInf(w, 0) || w < 0 || measured.IsZero() || measured.After(received) || measured.Before(s.powerConnectedAt) || measured.UnixMilli() <= s.meterAtMS { return false } s.lastPowerW = w + s.meterAtMS = measured.UnixMilli() s.forecastPower = telemetry.ForecastPowerSample{Version: 1, Known: true, Watts: w, MeasuredAtMS: measured.UnixMilli(), ReceivedAtMS: received.UnixMilli()} return true } +// statusTime is the charger's own time of a status, when it sent one. +func statusTime(t *types.DateTime) time.Time { + if t == nil { + return time.Time{} + } + return t.Time +} + +// holdAvailablePower records the 0 W of an empty connector. A meter sample +// the charger took before the status cannot bring back the car's draw; a +// newer one wins. The gate uses the charger's clock, so a clock that runs +// behind Core's does not reject the next session's samples. Without a +// status time it stays where it was. The caller holds h.mu. +func (s *chargerState) holdAvailablePower(now, at time.Time) { + ms := now.UnixMilli() + s.available = true + s.lastPowerW = 0 + if !at.IsZero() { + s.meterAtMS = max(s.meterAtMS, min(at.UnixMilli(), ms)) + } + s.forecastPower = telemetry.ForecastPowerSample{Version: 1, Known: true, Watts: 0, MeasuredAtMS: ms, ReceivedAtMS: ms} +} + +// confirmAvailablePower moves a 0 W reading to now while the connector stays +// Available on a live socket. A meter that read more than 0 W is not +// confirmed; it ages out. At most once a second. The caller holds h.mu. +func (s *chargerState) confirmAvailablePower(now time.Time) bool { + ms := now.UnixMilli() + if !s.available || !s.online || !s.connectedKnown || !s.forecastPower.Known || s.forecastPower.Watts != 0 || ms-s.forecastPower.MeasuredAtMS < 1000 { + return false + } + s.forecastPower.MeasuredAtMS, s.forecastPower.ReceivedAtMS = ms, ms + return true +} + +// leaveAvailable ends the hold when a car may be there. A 0 W reading is no +// longer evidence then; power is unknown until the meter reports. The caller +// holds h.mu. +func (s *chargerState) leaveAvailable() { + if !s.available { + return + } + s.available = false + if s.forecastPower.Watts == 0 { + s.forecastPower.Known = false + } +} + // clearMeasuredPower zeros dispatch watts without recording a measured sample. func (s *chargerState) clearMeasuredPower() { s.lastPowerW = 0 s.forecastPower.Known = false + s.available = false } diff --git a/go/internal/ocpp/handlers_v201.go b/go/internal/ocpp/handlers_v201.go index 3e1b1009f..b89cbfa17 100644 --- a/go/internal/ocpp/handlers_v201.go +++ b/go/internal/ocpp/handlers_v201.go @@ -66,7 +66,7 @@ func (h *handlerV201) OnBootNotification(id string, req *provisioning.BootNotifi h.mu.Unlock() h.setVersion(id, Version201) h.noteIdentity(id) - h.telSuccess(id) + h.telHealth(id) return provisioning.NewBootNotificationResponse( types201.NewDateTime(time.Now()), @@ -98,8 +98,19 @@ func (h *handlerV201) OnStatusNotification(id string, req *availability.StatusNo s := h.state(id) h.mu.Lock() s.connectedKnown = true + if req.ConnectorStatus != availability.ConnectorStatusAvailable { + s.leaveAvailable() + } switch req.ConnectorStatus { - case availability.ConnectorStatusAvailable, availability.ConnectorStatusUnavailable: + case availability.ConnectorStatusAvailable: + s.connected = false + s.charging = false + var at time.Time + if req.Timestamp != nil { + at = req.Timestamp.Time + } + s.holdAvailablePower(time.Now(), at) + case availability.ConnectorStatusUnavailable: s.connected = false s.charging = false s.clearMeasuredPower() @@ -143,6 +154,10 @@ func (h *handlerV201) OnTransactionEvent(id string, req *transactions.Transactio h.mu.Lock() acceptedPower := false if req.EventType != transactions.TransactionEventEnded { + // A live transaction may mean a car, unless it says none is there. + if req.TransactionInfo.ChargingState != transactions.ChargingStateIdle { + s.leaveAvailable() + } acceptedPower = s.recordPowerV201(req.MeterValue, time.Now()) } switch req.EventType { @@ -178,7 +193,10 @@ func (h *handlerV201) OnTransactionEvent(id string, req *transactions.Transactio s.transactionID = -1 s.transactionRef = "" s.charging = false - s.clearMeasuredPower() + // Available may come first; the car has gone then. + if !s.available { + s.clearMeasuredPower() + } } powerW := s.lastPowerW diff --git a/go/internal/ocpp/read_wait_test.go b/go/internal/ocpp/read_wait_test.go new file mode 100644 index 000000000..7898d7c95 --- /dev/null +++ b/go/internal/ocpp/read_wait_test.go @@ -0,0 +1,50 @@ +package ocpp + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/gorilla/websocket" + + "github.com/srcfl/ftw/go/internal/telemetry" +) + +// An idle charger that sends only heartbeats must keep its socket. The +// library's own limit was 60 s, the same as a 60 s heartbeat, so the socket +// was closed as the heartbeat arrived. The limit now follows the heartbeat +// interval: with a 1 s heartbeat a silent charger is closed after about 5 s, +// not after 60 s. +func TestSilentChargerSocketFollowsHeartbeat(t *testing.T) { + port := freePort(t) + cfg := &Config{Enabled: true, Bind: "127.0.0.1", Port: port, HeartbeatIntervalS: 1, ApprovedIDs: []string{"garage"}} + srv, err := Start(context.Background(), cfg, telemetry.NewStore()) + if err != nil { + t.Fatalf("start: %v", err) + } + t.Cleanup(srv.Stop) + waitForListener(t, port) + + dialer := websocket.Dialer{Subprotocols: []string{"ocpp1.6"}, HandshakeTimeout: 2 * time.Second} + conn, _, err := dialer.Dial(fmt.Sprintf("ws://127.0.0.1:%d%sgarage", port, srv.Path()), nil) + if err != nil { + t.Fatalf("dial: %v", err) + } + defer conn.Close() + + start := time.Now() + _ = conn.SetReadDeadline(start.Add(12 * time.Second)) + for { + // Core may probe the charger; a silent charger never answers. + if _, _, err := conn.ReadMessage(); err != nil { + break + } + } + if got := time.Since(start); got < 4*time.Second || got > 8*time.Second { + t.Fatalf("silent charger socket closed after %v, want about 5 s (five 1 s heartbeats)", got.Round(100*time.Millisecond)) + } + if got := socketReadWait(30); got != 150*time.Second { + t.Fatalf("default 30 s heartbeat gives %v, want 150 s", got) + } +} diff --git a/go/internal/ocpp/server.go b/go/internal/ocpp/server.go index 992b3b1e7..4b0329712 100644 --- a/go/internal/ocpp/server.go +++ b/go/internal/ocpp/server.go @@ -207,6 +207,14 @@ func newListener(cfg *Config, auth *authorizer, sockets *socketSessions) (ws.WsS } else { srv = ws.NewServer() } + // The library closes a socket that has sent nothing for its PingWait, + // 60 s by default. An idle charger that sends only heartbeats, at our + // interval or its own 60 s, lands on that edge and is dropped. The + // telemetry watchdog decides when a quiet charger is offline; this timer + // only has to outlast a few missed heartbeats. + timeouts := ws.NewServerTimeoutConfig() + timeouts.PingWait = socketReadWait(cfg.HeartbeatIntervalS) + srv.SetTimeoutConfig(timeouts) if auth.requiresCredential() { srv.SetBasicAuthHandler(auth.basicAuth) } @@ -214,6 +222,12 @@ func newListener(cfg *Config, auth *authorizer, sockets *socketSessions) (ws.WsS return &guardedServer{Server: srv, check: auth.checkClient, sessions: sockets}, nil } +// socketReadWait is how long a charger socket may stay silent: five +// heartbeat intervals. +func socketReadWait(heartbeatS int) time.Duration { + return 5 * time.Duration(heartbeatS) * time.Second +} + // guardedServer keeps our connection check installed. // // ocppj.Server.Start unconditionally calls SetCheckClientHandler with its own diff --git a/go/internal/sitesim/fuse.go b/go/internal/sitesim/fuse.go new file mode 100644 index 000000000..6e33e75e9 --- /dev/null +++ b/go/internal/sitesim/fuse.go @@ -0,0 +1,46 @@ +package sitesim + +import ( + "fmt" + + "github.com/srcfl/device-simulator/fuse" +) + +// Fuse is a gG main fuse (IEC 60269) on each phase, as srcfl/device-simulator's +// fuse package models it: the thermal model the site rig judges Core by, so +// both simulators share one fuse. A phase starts cold in a 25 °C cabinet; +// Preheat starts it warm. Damage and Blown mirror each phase after a step. +type Fuse struct { + RatingA float64 + Phase [3]fuse.Phase + Damage [3]float64 + Blown [3]bool +} + +var gG, _ = fuse.Lookup(fuse.IecGg) + +// AnyBlown reports whether any phase has blown. +func (f *Fuse) AnyBlown() bool { return f.Blown[0] || f.Blown[1] || f.Blown[2] } + +// Preheat puts each phase in the steady state of amps held for hours, for a +// scenario that starts with the house near its limit. +func (f *Fuse) Preheat(amps [3]float64) error { + for i, a := range amps { + if err := gG.Preheat(&f.Phase[i], a, f.RatingA); err != nil { + return fmt.Errorf("L%d: %w", i+1, err) + } + f.Damage[i] = f.Phase[i].Damage + } + return nil +} + +// step panics on a step the model refuses, such as a rating it does not +// model: a scenario that cannot be judged is a broken test. +func (f *Fuse) step(amps [3]float64, dt float64) { + for i, a := range amps { + if st := gG.Step(&f.Phase[i], a, f.RatingA, dt); st.Invalid != "" { + panic(fmt.Sprintf("sitesim: L%d fuse: %s", i+1, st.Invalid)) + } + f.Damage[i], f.Blown[i] = f.Phase[i].Damage, f.Phase[i].Blown + } +} diff --git a/go/internal/sitesim/sitesim.go b/go/internal/sitesim/sitesim.go new file mode 100644 index 000000000..0ed284435 --- /dev/null +++ b/go/internal/sitesim/sitesim.go @@ -0,0 +1,314 @@ +// Package sitesim is a simulated house behind one main fuse, for testing Core +// in a closed loop. It holds the physics and the devices' own behaviour; Core +// reads what the devices report and sends them commands. A test advances it +// one second at a time on a pinned clock. Nothing here sleeps or starts a +// goroutine. +// +// Signs follow docs/site-convention.md: positive W flows into the site. +// Phase currents are signed the same way unless a meter drops the sign. +package sitesim + +import ( + "math" + "time" +) + +// Step is the plant's time step. +const Step = time.Second + +// Plant is one site. Set its fields before the first Step and change them +// from events. +type Plant struct { + Now time.Time + Voltage float64 + HouseW [3]float64 // house load per phase + PVW float64 // three-phase PV, zero or negative + Battery *Battery // nil: none + EVSE *EVSE // nil: none + Car *Car // the car on the EVSE; nil or unplugged: none + Meter Meter + Fuse Fuse + Events []Event // in time order; each fires once + + A [3]float64 // physical phase currents after the last step + start time.Time + offline map[string]time.Time +} + +// New returns a plant starting at start with a fuse of fuseA per phase. +func New(start time.Time, fuseA float64) *Plant { + return &Plant{Now: start, start: start, Voltage: 230, Fuse: Fuse{RatingA: fuseA}, offline: map[string]time.Time{}} +} + +// EventKind says what an event does. +type EventKind int + +const ( + Change EventKind = iota // run Do on the plant + CoreRestart // Core stops for For and starts with no memory + MeterSilent // the site meter sends nothing for For + DriverOffline // Device sends nothing for For; Offline says so, and callers send it nothing +) + +// Event is one change on the timeline, At after the plant started. +type Event struct { + At time.Duration + Kind EventKind + For time.Duration + Device string // "battery" or "evse" + Do func(*Plant) +} + +// Tick is what one second produced. A nil reading means the device sent +// nothing this second. +type Tick struct { + Meter *MeterReading + Battery *BatteryReading + EV *EVReading + Restart time.Duration // > 0: Core goes down now for this long +} + +// MeterReading is one site meter reading. +type MeterReading struct { + W float64 + PhaseA [3]float64 +} + +// BatteryReading is what the battery reports. +type BatteryReading struct{ W, SoC float64 } + +// EVReading is what the charger reports. W is ReadbackLag late. +type EVReading struct { + W, SessionWh float64 + Plugged bool +} + +// Offline reports whether device is unreachable now. +func (p *Plant) Offline(device string) bool { return p.Now.Before(p.offline[device]) } + +// Step advances one second: events, device responses, physics, fuse and +// readings, in that order. +func (p *Plant) Step() Tick { + p.Now = p.Now.Add(Step) + var tick Tick + for len(p.Events) > 0 && p.Now.Sub(p.start) >= p.Events[0].At { + e := p.Events[0] + p.Events = p.Events[1:] + switch e.Kind { + case Change: + e.Do(p) + case CoreRestart: + tick.Restart = e.For + case MeterSilent: + p.Meter.silentUntil = p.Now.Add(e.For) + case DriverOffline: + p.offline[e.Device] = p.Now.Add(e.For) + } + } + var batW [3]float64 + if b := p.Battery; b != nil { + b.step(p.Now) + batW = spread(b.W, b.Phases) + } + for i := range p.A { + p.A[i] = (p.HouseW[i] + p.PVW/3 + batW[i]) / p.Voltage + } + if e := p.EVSE; e != nil { + evA := spread(e.step(p.Now, p.Car, p.A, p.Voltage), e.DrawPhases) + for i := range p.A { + p.A[i] += evA[i] / p.Voltage + } + } + p.Fuse.step(p.A, Step.Seconds()) + if p.Meter.due(p.Now) { + r := &MeterReading{PhaseA: p.A} + for i, a := range p.A { + r.W += a * p.Voltage + if p.Meter.Unsigned { + r.PhaseA[i] = math.Abs(a) + } + } + tick.Meter = r + } + if b := p.Battery; b != nil && !p.Offline("battery") { + tick.Battery = &BatteryReading{W: b.W, SoC: b.SoC} + } + if e := p.EVSE; e != nil && !p.Offline("evse") { + tick.EV = &EVReading{W: e.readback(p.Now), SessionWh: e.SessionWh, Plugged: p.Car != nil && p.Car.Plugged} + } + return tick +} + +// spread puts w on L1 alone, evenly on L1 and L2, or evenly on three phases. +func spread(w float64, phases int) [3]float64 { + switch phases { + case 1: + return [3]float64{w, 0, 0} + case 2: + return [3]float64{w / 2, w / 2, 0} + } + return [3]float64{w / 3, w / 3, w / 3} +} + +type timedW struct { + at time.Time + w float64 +} + +// Meter is the site meter. It reads the boundary every Every, or every +// second when Every is zero. +type Meter struct { + Every time.Duration + Unsigned bool // phase currents as magnitudes + last time.Time + silentUntil time.Time +} + +func (m *Meter) due(now time.Time) bool { + if now.Before(m.silentUntil) || (!m.last.IsZero() && now.Sub(m.last) < m.Every) { + return false + } + m.last = now + return true +} + +// Battery is a battery inverter that follows a power command after Latency. +type Battery struct { + Phases int // 1 (on L1) or 3 + MaxChargeW, MaxDischargeW float64 + CapacityWh, SoC float64 + Latency time.Duration + W float64 // power now, positive charging + queue []timedW +} + +// Command asks for w from now + Latency. +func (b *Battery) Command(now time.Time, w float64) { + b.queue = append(b.queue, timedW{now.Add(b.Latency), w}) +} + +func (b *Battery) step(now time.Time) { + for len(b.queue) > 0 && !b.queue[0].at.After(now) { + b.W, b.queue = b.queue[0].w, b.queue[1:] + } + b.W = math.Max(-b.MaxDischargeW, math.Min(b.MaxChargeW, b.W)) + if (b.W > 0 && b.SoC >= 1) || (b.W < 0 && b.SoC <= 0) { + b.W = 0 + } + if b.CapacityWh > 0 { + b.SoC = math.Max(0, math.Min(1, b.SoC+b.W*Step.Hours()/b.CapacityWh)) + } +} + +// EVSE is a charger with its driver: it turns a power command into a +// current limit, as the Easee driver does, and applies it after Delay. +type EVSE struct { + Phases int // wired phases, 1 or 3 + MinA, MaxA float64 + Delay time.Duration // from command to current at the car + ReadbackLag time.Duration // the charger reports the draw this late + LoadBalancerA float64 // > 0: its own balancer keeps each phase under this + OfferA float64 // current limit in force + OfferPhases int + DrawW float64 // the car's draw now + DrawPhases int + SessionWh float64 // since the car plugged in + + // The Easee driver's own choice for "auto" (easee_cloud.lua pick_phases): + // without a phase_split_w it splits at the fuse's current on one phase, + // and it keeps a phase count PhaseHold before it changes it. + SplitAtA float64 // > 0: auto with no split splits at voltage × SplitAtA + PhaseHold time.Duration // > 0: an auto phase count stays this long + phasesSent int + phasesAt time.Time + queue []evCommand + draws []timedW +} + +type evCommand struct { + at time.Time + a float64 + phases int +} + +// Command sends powerW in phaseMode ("1p", "3p", "auto" or empty). Auto +// picks one phase below splitW, as the Easee driver does; PhaseHold adds its +// hysteresis. Below MinA the charger pauses. +func (e *EVSE) Command(now time.Time, powerW float64, phaseMode string, splitW, voltage float64) { + phases := e.Phases + if phaseMode == "auto" && splitW <= 0 && e.SplitAtA > 0 { + splitW = voltage * e.SplitAtA + } + if phaseMode == "1p" || (phaseMode == "auto" && powerW < math.Max(splitW, 3*voltage*e.MinA)) { + phases = 1 + } + if phaseMode == "auto" && e.PhaseHold > 0 && e.phasesSent > 0 && phases != e.phasesSent && now.Sub(e.phasesAt) < e.PhaseHold { + phases = e.phasesSent + } + if phases != e.phasesSent { + e.phasesSent, e.phasesAt = phases, now + } + a := math.Min(math.Floor(powerW/voltage/float64(phases)+.5), e.MaxA) + if a < e.MinA { + a = 0 + } + e.queue = append(e.queue, evCommand{now.Add(e.Delay), a, phases}) +} + +// step applies due commands and returns the car's draw. The car takes the +// offer, its own maximum and the balancer's room, whichever is least. +func (e *EVSE) step(now time.Time, car *Car, houseA [3]float64, voltage float64) float64 { + for len(e.queue) > 0 && !e.queue[0].at.After(now) { + e.OfferA, e.OfferPhases, e.queue = e.queue[0].a, e.queue[0].phases, e.queue[1:] + } + w := 0.0 + if car != nil && car.Plugged && e.OfferA > 0 { + e.DrawPhases = min(max(e.OfferPhases, 1), e.Phases, car.Phases) + a := math.Min(e.OfferA, car.MaxW/voltage/float64(e.DrawPhases)) + if e.LoadBalancerA > 0 { + for i := range e.DrawPhases { + a = math.Min(a, e.LoadBalancerA-houseA[i]) + } + } + if a >= e.MinA { + w = a * voltage * float64(e.DrawPhases) + } + } + e.DrawW, e.SessionWh = w, e.SessionWh+w*Step.Hours() + if car == nil || !car.Plugged { + e.SessionWh = 0 + } + if len(e.draws) == 0 || e.draws[len(e.draws)-1].w != w { + e.draws = append(e.draws, timedW{now, w}) + } + return w +} + +// Queued is the last order the charger has not applied yet, as current per +// phase and phases; false when every order is in force. +func (e *EVSE) Queued() (a float64, phases int, ok bool) { + if len(e.queue) == 0 { + return 0, 0, false + } + c := e.queue[len(e.queue)-1] + return c.a, c.phases, true +} + +// readback is the draw as it was ReadbackLag ago. +func (e *EVSE) readback(now time.Time) float64 { + seen := now.Add(-e.ReadbackLag) + for len(e.draws) > 1 && !e.draws[1].at.After(seen) { + e.draws = e.draws[1:] + } + if len(e.draws) == 0 || e.draws[0].at.After(seen) { + return 0 + } + return e.draws[0].w +} + +// Car is the vehicle on the EVSE. +type Car struct { + Phases int // phases its onboard charger uses + MaxW float64 // the most it takes, whatever it is offered + Plugged bool +} diff --git a/go/internal/sitesim/sitesim_test.go b/go/internal/sitesim/sitesim_test.go new file mode 100644 index 000000000..b9eda6300 --- /dev/null +++ b/go/internal/sitesim/sitesim_test.go @@ -0,0 +1,91 @@ +package sitesim + +import ( + "math" + "testing" + "time" + + "github.com/srcfl/device-simulator/fuse" +) + +var t0 = time.Date(2026, 10, 21, 12, 0, 0, 0, time.UTC) + +// The plant's fuse is the fuse package's gG at its default 25 °C, stepped a +// second at a time: it blows in the second the model melts, export heats it +// as import does, a warm fuse blows sooner and the other phases stay whole. +func TestFuseFollowsTheFuseModel(t *testing.T) { + gG, _ := fuse.Lookup(fuse.IecGg) + blowAfter := func(f Fuse, a float64) float64 { + for s := 1; s <= 7200; s++ { + f.step([3]float64{-a, 0, 0}, 1) + if f.Blown[0] { + if f.Blown[1] || f.Blown[2] { + t.Fatalf("%.0f A on L1 blew another phase", a) + } + return float64(s) + } + } + return math.Inf(1) + } + for _, tc := range []struct{ ratingA, multiple float64 }{{16, 2}, {20, 2}, {25, 2}, {25, 3}, {25, 1.6}} { + want := math.Ceil(gG.MeltSeconds(tc.multiple, tc.ratingA, fuse.DefaultAmbientC)) + if got := blowAfter(Fuse{RatingA: tc.ratingA}, tc.multiple*tc.ratingA); got != want { + t.Errorf("%.1f× %.0f A: blew after %v s, want %v s", tc.multiple, tc.ratingA, got, want) + } + } + if got := blowAfter(Fuse{RatingA: 25}, 1.25*25); !math.IsInf(got, 1) { + t.Errorf("1.25× 25 A blew after %v s, want never", got) + } + warm := Fuse{RatingA: 25} + if err := warm.Preheat([3]float64{23, 23, 23}); err != nil { + t.Fatal(err) + } + if cold, hot := blowAfter(Fuse{RatingA: 25}, 50), blowAfter(warm, 50); hot >= cold { + t.Errorf("2× 25 A: warm blew after %v s, cold after %v s", hot, cold) + } + defer func() { + if recover() == nil { + t.Error("a 10 A gG fuse, which the model does not cover, stepped without a panic") + } + }() + (&Fuse{RatingA: 10}).step([3]float64{5, 5, 5}, 1) +} + +// PV export shows as negative phase currents, or as magnitudes on an +// unsigned meter, and a slow charger and a slow battery answer late. +func TestPlantSignsAndDelays(t *testing.T) { + p := New(t0, 25) + p.HouseW = [3]float64{460, 460, 460} + p.PVW = -6900 + p.Meter.Unsigned = true + p.Battery = &Battery{Phases: 1, MaxChargeW: 3000, CapacityWh: 10000, SoC: .5, Latency: 2 * time.Second} + p.EVSE = &EVSE{Phases: 3, MinA: 6, MaxA: 16, Delay: 5 * time.Second, ReadbackLag: 10 * time.Second} + p.Car = &Car{Phases: 3, MaxW: 11040, Plugged: true} + p.Events = []Event{{At: 3 * time.Second, Kind: MeterSilent, For: 2 * time.Second}} + if tick := p.Step(); tick.Meter.W != -5520 || tick.Meter.PhaseA != [3]float64{8, 8, 8} || p.A[0] != -8 { + t.Fatalf("export: meter %+v, physical %v", tick.Meter, p.A) + } + p.Battery.Command(p.Now, 2300) + p.EVSE.Command(p.Now, 6900, "", 0, 230) + var heard []float64 + for s := 2; s <= 17; s++ { + tick := p.Step() + if (tick.Meter == nil) != (s == 3 || s == 4) { + t.Fatalf("%d s: meter reading %v", s, tick.Meter) + } + if want := s >= 3; (p.Battery.W == 2300) != want { + t.Fatalf("%d s: battery %.0f W", s, p.Battery.W) + } + wantL2 := -8.0 + if s >= 6 { + wantL2 = 2 + } + if p.A[1] != wantL2 { + t.Fatalf("%d s: L2 %.1f A, want %.0f A", s, p.A[1], wantL2) + } + heard = append(heard, tick.EV.W) + } + if heard[13] != 0 || heard[14] != 6900 { + t.Fatalf("readback %v, want the draw from 6 s at 16 s", heard) + } +} diff --git a/go/internal/state/archive_write.go b/go/internal/state/archive_write.go index 0fac2f982..203469101 100644 --- a/go/internal/state/archive_write.go +++ b/go/internal/state/archive_write.go @@ -42,15 +42,19 @@ func (s *Store) archiveTransaction(ctx context.Context, prepare, write func(cont } } -func (s *Store) tryArchiveBatch(parent context.Context, prepare, write func(context.Context, *sql.Tx) error) error { +func (s *Store) tryArchiveBatch(parent context.Context, prepare, write func(context.Context, *sql.Tx) error) (err error) { + timing := newHistoryWriteTiming("connection") conn, err := s.history.Conn(parent) if err != nil { - return err + return timing.failure(err) } var busyMS int var tx *sql.Tx var restoreBusy, viewLocked, writerLocked bool defer func() { + // Capture the operation before rollback, busy-timeout restoration and + // connection close. Those cleanup steps keep their existing order. + err = timing.failure(err) if tx != nil { _ = tx.Rollback() } @@ -72,6 +76,7 @@ func (s *Store) tryArchiveBatch(parent context.Context, prepare, write func(cont s.archiveViewMu.Unlock() } }() + timing.next("busy_timeout") if err := conn.QueryRowContext(parent, `PRAGMA busy_timeout`).Scan(&busyMS); err != nil { return err } @@ -81,11 +86,13 @@ func (s *Store) tryArchiveBatch(parent context.Context, prepare, write func(cont } txCtx, cancelTx := context.WithCancel(parent) defer cancelTx() + timing.next("begin") tx, err = conn.BeginTx(txCtx, nil) if err != nil { return err } if prepare != nil { + timing.next("prepare") readCtx, cancel := context.WithTimeout(parent, 30*time.Second) err := prepare(readCtx, tx) cancel() @@ -101,17 +108,21 @@ func (s *Store) tryArchiveBatch(parent context.Context, prepare, write func(cont defer stopDeadline() // Readers may postpone archive writes, but must never make an archive // operation wait while holding the live writer mutex. + timing.next("archive_view_lock") if err := lockContext(ctx, s.archiveViewMu.TryLock); err != nil { return err } viewLocked = true + timing.next("writer_lock") if err := lockContext(ctx, s.historyWriteMu.TryLock); err != nil { return err } writerLocked = true + timing.next("write") if err := write(ctx, tx); err != nil { return err } + timing.next("commit") err = tx.Commit() if err != nil && ctx.Err() != nil { return ctx.Err() diff --git a/go/internal/state/archive_write_timing_test.go b/go/internal/state/archive_write_timing_test.go new file mode 100644 index 000000000..4152c41fb --- /dev/null +++ b/go/internal/state/archive_write_timing_test.go @@ -0,0 +1,90 @@ +package state + +import ( + "context" + "database/sql" + "errors" + "strings" + "testing" + "time" +) + +func TestArchiveWriteFailureReportsPhaseAndPreservesRows(t *testing.T) { + for _, phase := range []string{"connection", "prepare", "archive_view_lock", "writer_lock", "write"} { + t.Run(phase, func(t *testing.T) { + s := freshStore(t) + if err := s.RecordSamples([]Sample{{Driver: "meter", Metric: "power", TsMs: 1000, Value: 1}}); err != nil { + t.Fatal(err) + } + if _, err := s.history.Exec(`CREATE TABLE archive_timing_probe(id INTEGER PRIMARY KEY); INSERT INTO archive_timing_probe VALUES(1)`); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond) + defer cancel() + var release func() + switch phase { + case "connection": + s.history.SetMaxOpenConns(1) + conn, err := s.history.Conn(context.Background()) + if err != nil { + t.Fatal(err) + } + release = func() { _ = conn.Close() } + case "archive_view_lock": + s.archiveViewMu.Lock() + release = s.archiveViewMu.Unlock + case "writer_lock": + s.historyWriteMu.Lock() + release = s.historyWriteMu.Unlock + } + var prepare func(context.Context, *sql.Tx) error + if phase == "prepare" { + prepare = func(ctx context.Context, tx *sql.Tx) error { + var count int + if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM archive_timing_probe`).Scan(&count); err != nil { + return err + } + <-ctx.Done() + return ctx.Err() + } + } + err := s.tryArchiveBatch(ctx, prepare, func(ctx context.Context, tx *sql.Tx) error { + if _, err := tx.ExecContext(ctx, `INSERT INTO archive_timing_probe VALUES(2)`); err != nil { + return err + } + if phase == "write" { + <-ctx.Done() + return ctx.Err() + } + return nil + }) + if release != nil { + release() + } + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("archive failure lost its deadline: %v", err) + } + for _, want := range []string{"phase=" + phase, "elapsed_ms=", "step_ms=", phase + "_max_ms=", phase + "_steps=1"} { + if !strings.Contains(err.Error(), want) { + t.Fatalf("missing %q: %v", want, err) + } + } + if strings.Contains(err.Error(), "archive_timing_probe") { + t.Fatal("archive timing exposed SQL contents") + } + var count int + if err := s.history.QueryRow(`SELECT COUNT(*) FROM archive_timing_probe`).Scan(&count); err != nil || count != 1 { + t.Fatalf("failed attempt changed committed rows: count=%d err=%v", count, err) + } + if err := s.tryArchiveBatch(context.Background(), nil, func(ctx context.Context, tx *sql.Tx) error { + _, err := tx.ExecContext(ctx, `INSERT INTO archive_timing_probe VALUES(2)`) + return err + }); err != nil { + t.Fatal("retry:", err) + } + if err := s.history.QueryRow(`SELECT COUNT(*) FROM archive_timing_probe`).Scan(&count); err != nil || count != 2 { + t.Fatalf("retry lost or duplicated rows: count=%d err=%v", count, err) + } + }) + } +} diff --git a/go/internal/state/backup_failure_test.go b/go/internal/state/backup_failure_test.go new file mode 100644 index 000000000..3dd226367 --- /dev/null +++ b/go/internal/state/backup_failure_test.go @@ -0,0 +1,86 @@ +package state + +import ( + "bytes" + "context" + "errors" + "log/slog" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// The home box's backup on 16 October 2026 copied on the SD card, ran into +// the live time limit and reported only "context deadline exceeded". +func TestLiveBackupPastItsLimitNamesTheLimitAndTable(t *testing.T) { + s := freshStore(t) + if err := s.BulkRecordHistory(pacedBackupPoints(5000)); err != nil { + t.Fatal(err) + } + orig := liveBackupTimeout + t.Cleanup(func() { liveBackupTimeout = orig }) + liveBackupTimeout = time.Second + // Each batch waits as a commit on the SD card did; four outlast the limit. + s.backupPause = func(ctx context.Context) error { + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(300 * time.Millisecond): + return nil + } + } + err := s.BackupToCompressed(filepath.Join(t.TempDir(), "full.gz")) + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("backup past its limit = %v", err) + } + for _, want := range []string{"live backup reached its time limit of 1s", "copy history_hot"} { + if !strings.Contains(err.Error(), want) { + t.Fatalf("reason %q does not say %q", err, want) + } + } +} + +func TestCanceledBackupIsNotBlamedOnTheLimit(t *testing.T) { + s := freshStore(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, _, err := s.BackupWithConfigurationContext(ctx, filepath.Join(t.TempDir(), "full.gz"), nil) + if !errors.Is(err, context.Canceled) || strings.Contains(err.Error(), "time limit") { + t.Fatalf("canceled backup = %v", err) + } +} + +func TestBackupScratchOnDiskNamesScratchLeftInShm(t *testing.T) { + shm := t.TempDir() + orig := backupShmDir + t.Cleanup(func() { backupShmDir = orig }) + backupShmDir = shm + var logs bytes.Buffer + previous := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil))) + defer slog.SetDefault(previous) + + path, cleanup := backupScratchFile(filepath.Join(t.TempDir(), "full.gz"), 1<<20) + cleanup() + if !strings.HasPrefix(path, shm) || logs.Len() != 0 { + t.Fatalf("fitting scratch = %s, logs %q", path, logs.String()) + } + + // A backup stopped by a restart leaves its scratch until reboot. + if err := os.WriteFile(filepath.Join(shm, "ftw-backup-4218622654.raw"), make([]byte, 4096), 0o600); err != nil { + t.Fatal(err) + } + dst := filepath.Join(t.TempDir(), "full.gz") + path, cleanup = backupScratchFile(dst, 1<<50) + defer cleanup() + if path != dst+".raw.tmp" { + t.Fatalf("oversized scratch = %s", path) + } + for _, want := range []string{"copying on the data disk", "backup_scratch_in_shm_bytes=4096"} { + if !strings.Contains(logs.String(), want) { + t.Fatalf("log %q does not say %q", logs.String(), want) + } + } +} diff --git a/go/internal/state/backup_scratch.go b/go/internal/state/backup_scratch.go new file mode 100644 index 000000000..304328087 --- /dev/null +++ b/go/internal/state/backup_scratch.go @@ -0,0 +1,100 @@ +package state + +import ( + "errors" + "io/fs" + "log/slog" + "os" + "path/filepath" + "strings" +) + +// HoldBackupScratch takes a shared lock on dir, a folder where a backup keeps +// scratch, until release. The kernel drops the lock when the process exits, +// also when systemd kills it before deferred cleanup runs. A folder that +// cannot be locked leaves the backup running without that protection. +func HoldBackupScratch(dir string) (release func()) { + f, err := lockBackupScratchDir(dir, false) + if err != nil { + if !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, errors.ErrUnsupported) { + slog.Debug("backup scratch folder not locked", "dir", dir, "err", err) + } + return func() {} + } + return func() { _ = f.Close() } +} + +// RemoveStaleBackupScratchInShm removes raw copies that stopped backups left +// in /dev/shm. On the home box one held 1.37 GB of RAM for days, so the next +// backup copied on the SD card and ran out of time. +func RemoveStaleBackupScratchInShm() { + RemoveStaleBackupScratch("/dev/shm", isShmBackupScratch) +} + +// RemoveStaleBackupScratch removes the entries in dir that isScratch accepts +// when it can lock dir exclusively, which means no backup is running there. +// It logs each entry it removes with its size and returns the totals. +func RemoveStaleBackupScratch(dir string, isScratch func(fs.DirEntry) bool) (removed int, bytes int64) { + lock, lockErr := lockBackupScratchDir(dir, true) + if lockErr == nil { + defer lock.Close() + } + entries, err := os.ReadDir(dir) + if err != nil { + return 0, 0 + } + var stale []string + for _, entry := range entries { + if isScratch(entry) { + stale = append(stale, filepath.Join(dir, entry.Name())) + } + } + if len(stale) == 0 { + return 0, 0 + } + if lockErr != nil { + slog.Info("kept backup scratch; a running backup may own it", "dir", dir, "entries", len(stale), "err", lockErr) + return 0, 0 + } + for _, path := range stale { + size := scratchBytes(path) + if err := os.RemoveAll(path); err != nil { + slog.Warn("could not remove scratch from a stopped backup", "path", path, "err", err) + continue + } + slog.Info("removed scratch from a stopped backup", "path", path, "bytes", size) + removed++ + bytes += size + } + return removed, bytes +} + +// isShmBackupScratch matches backupScratchFile's ftw-backup-.raw and the +// rollback journal SQLite keeps beside it. +func isShmBackupScratch(entry fs.DirEntry) bool { + stem, ok := strings.CutSuffix(strings.TrimSuffix(entry.Name(), "-journal"), ".raw") + return ok && entry.Type().IsRegular() && IsTempName(stem, "ftw-backup-") +} + +// IsTempName reports whether name is prefix plus the decimal suffix that +// os.CreateTemp and os.MkdirTemp add. +func IsTempName(name, prefix string) bool { + suffix, ok := strings.CutPrefix(name, prefix) + return ok && suffix != "" && strings.Trim(suffix, "0123456789") == "" +} + +func scratchBytes(path string) int64 { + var n int64 + _ = filepath.WalkDir(path, func(_ string, entry fs.DirEntry, err error) error { + if err != nil { + return nil + } + if entry.Type().IsRegular() { + if info, err := entry.Info(); err == nil { + n += info.Size() + } + } + return nil + }) + return n +} diff --git a/go/internal/state/backup_scratch_test.go b/go/internal/state/backup_scratch_test.go new file mode 100644 index 000000000..7a39f28b2 --- /dev/null +++ b/go/internal/state/backup_scratch_test.go @@ -0,0 +1,180 @@ +//go:build unix + +package state + +import ( + "bufio" + "bytes" + "context" + "fmt" + "log/slog" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "syscall" + "testing" + "time" +) + +func TestBackupScratchHolderHelper(t *testing.T) { + dir := os.Getenv("FTW_BACKUP_SCRATCH_DIR") + if dir == "" { + t.Skip("subprocess helper") + } + HoldBackupScratch(dir) + if err := os.WriteFile(filepath.Join(dir, "ftw-backup-2.raw"), make([]byte, 4096), 0o600); err != nil { + t.Fatal(err) + } + fmt.Println("BACKUP_SCRATCH_HELD") + <-time.After(time.Minute) +} + +// On the home box systemd killed Core mid-backup, so its deferred cleanup +// never ran and the raw copy stayed in /dev/shm. +func TestKilledBackupScratchIsRemovedAndLiveScratchKept(t *testing.T) { + dir := t.TempDir() + notOurs := []string{"ftw-backup-x.raw", "ftw-backup-3.raw.bak", "other.raw", "ftw-backup-4.raw-wal"} + for _, name := range notOurs { + if err := os.WriteFile(filepath.Join(dir, name), []byte("x"), 0o600); err != nil { + t.Fatal(err) + } + } + if err := os.Mkdir(filepath.Join(dir, "ftw-backup-5.raw"), 0o700); err != nil { + t.Fatal(err) + } + notOurs = append(notOurs, "ftw-backup-5.raw") + + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestBackupScratchHolderHelper$") + cmd.Env = append(os.Environ(), "FTW_BACKUP_SCRATCH_DIR="+dir) + stdout, err := cmd.StdoutPipe() + if err != nil { + t.Fatal(err) + } + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + defer cmd.Process.Kill() + held := false + lines := bufio.NewScanner(stdout) + for lines.Scan() { + if lines.Text() == "BACKUP_SCRATCH_HELD" { + held = true + break + } + } + if !held { + cmd.Wait() + t.Fatal("helper never held the folder", ctx.Err(), lines.Err()) + } + raw := filepath.Join(dir, "ftw-backup-2.raw") + journal := raw + "-journal" + if err := os.WriteFile(journal, make([]byte, 512), 0o600); err != nil { + t.Fatal(err) + } + + if removed, _ := RemoveStaleBackupScratch(dir, isShmBackupScratch); removed != 0 { + t.Fatalf("removed %d entries of a running backup", removed) + } + for _, path := range []string{raw, journal} { + if _, err := os.Stat(path); err != nil { + t.Fatalf("running backup lost %s: %v", path, err) + } + } + + if err := cmd.Process.Signal(syscall.SIGKILL); err != nil { + t.Fatal(err) + } + if err := cmd.Wait(); err == nil { + t.Fatal("helper exited cleanly") + } + logs := captureDefaultLog(t) + removed, n := RemoveStaleBackupScratch(dir, isShmBackupScratch) + if removed != 2 || n != 4096+512 { + t.Fatalf("removed %d entries, %d bytes; want 2, 4608", removed, n) + } + for _, path := range []string{raw, journal} { + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Fatalf("%s left after its backup was killed: %v", path, err) + } + } + if got := logs.String(); !strings.Contains(got, "path="+raw+" bytes=4096") { + t.Fatalf("log does not name the removed copy and its size:\n%s", got) + } + for _, name := range notOurs { + if _, err := os.Stat(filepath.Join(dir, name)); err != nil { + t.Fatalf("removed %s, which is not backup scratch: %v", name, err) + } + } +} + +// The raw copy is the large scratch file. On the home box it sat in /dev/shm. +func TestRawCopyFolderIsHeldWhileCopying(t *testing.T) { + s := freshStore(t) + dst := filepath.Join(t.TempDir(), "database.gz") + held := 0 + err := s.backupToCompressed(dst, func(BackupProgress) { + raw := ownRawCopy(dst) + if raw == "" { + return + } + lock, err := lockBackupScratchDir(filepath.Dir(raw), true) + if err == nil { + lock.Close() + t.Errorf("nothing holds %s while %s is written", filepath.Dir(raw), raw) + return + } + held++ + }, nil) + if err != nil { + t.Fatal(err) + } + if held == 0 { + t.Fatal("no progress report while the raw copy existed") + } +} + +// ownRawCopy finds this process's raw copy beside dst or, on Linux, among its +// open files in /dev/shm. +func ownRawCopy(dst string) string { + if _, err := os.Stat(dst + ".raw.tmp"); err == nil { + return dst + ".raw.tmp" + } + links, _ := filepath.Glob("/proc/self/fd/*") + for _, link := range links { + target, err := os.Readlink(link) + if err == nil && strings.HasPrefix(target, "/dev/shm/ftw-backup-") && strings.HasSuffix(target, ".raw") { + return target + } + } + return "" +} + +type lockedBuffer struct { + mu sync.Mutex + buf bytes.Buffer +} + +func (b *lockedBuffer) Write(p []byte) (int, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.Write(p) +} + +func (b *lockedBuffer) String() string { + b.mu.Lock() + defer b.mu.Unlock() + return b.buf.String() +} + +func captureDefaultLog(t *testing.T) *lockedBuffer { + t.Helper() + logs := &lockedBuffer{} + previous := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(logs, nil))) + t.Cleanup(func() { slog.SetDefault(previous) }) + return logs +} diff --git a/go/internal/state/backup_scratch_unix.go b/go/internal/state/backup_scratch_unix.go new file mode 100644 index 000000000..19f7866de --- /dev/null +++ b/go/internal/state/backup_scratch_unix.go @@ -0,0 +1,27 @@ +//go:build unix + +package state + +import ( + "os" + "syscall" +) + +// lockBackupScratchDir locks dir itself, so no lock file is left behind. A +// backup waits for a shared lock while a cleanup holds it; a cleanup does not +// wait for a backup. +func lockBackupScratchDir(dir string, exclusive bool) (*os.File, error) { + f, err := os.Open(dir) + if err != nil { + return nil, err + } + how := syscall.LOCK_SH + if exclusive { + how = syscall.LOCK_EX | syscall.LOCK_NB + } + if err := syscall.Flock(int(f.Fd()), how); err != nil { + _ = f.Close() + return nil, &os.PathError{Op: "flock", Path: dir, Err: err} + } + return f, nil +} diff --git a/go/internal/state/backup_scratch_windows.go b/go/internal/state/backup_scratch_windows.go new file mode 100644 index 000000000..9203129e5 --- /dev/null +++ b/go/internal/state/backup_scratch_windows.go @@ -0,0 +1,14 @@ +//go:build windows + +package state + +import ( + "errors" + "os" +) + +// Windows has no flock. A cleanup there cannot tell a stopped backup from a +// running one, so it removes nothing. +func lockBackupScratchDir(string, bool) (*os.File, error) { + return nil, errors.ErrUnsupported +} diff --git a/go/internal/state/backup_state.go b/go/internal/state/backup_state.go index b0760888e..16dcff2d3 100644 --- a/go/internal/state/backup_state.go +++ b/go/internal/state/backup_state.go @@ -5,6 +5,7 @@ import ( "database/sql" "errors" "fmt" + "log/slog" "os" "path/filepath" "sort" @@ -12,11 +13,20 @@ import ( "time" ) -const liveBackupTimeout = 2 * time.Hour +// liveBackupTimeout bounds a live Core backup. A test shortens it. +var liveBackupTimeout = 2 * time.Hour + +// errLiveBackupLimit names the limit in a failed backup's reason. A bare +// "context deadline exceeded" left the home box's failure unexplained. +var errLiveBackupLimit = errors.New("live backup reached its time limit") + const backupScratchHeadroom = 64 << 20 var backupDiskAvail = diskAvail +// backupShmDir is the tmpfs for raw scratch copies. A test moves it. +var backupShmDir = "/dev/shm" + // OfflineBackup reports whether this store is the read-only helper used by // ftw-backup. Live Core backups keep their 100 ms yield; the helper does not. func (s *Store) OfflineBackup() bool { @@ -31,7 +41,7 @@ func (s *Store) backupWorkContextParent(parent context.Context) (context.Context if s.OfflineBackup() { return context.WithCancel(parent) } - return context.WithTimeout(parent, liveBackupTimeout) + return context.WithTimeoutCause(parent, liveBackupTimeout, fmt.Errorf("%w of %s", errLiveBackupLimit, liveBackupTimeout)) } func (s *Store) backupCopyYield(ctx context.Context) func() error { @@ -133,8 +143,9 @@ func BackupArchiveScratch(sourceBytes, extraBytes int64) int64 { // filesystem and is what gets synced. func backupScratchFile(dstPath string, sourceBytes int64) (path string, cleanup func()) { need := sourceBytes + backupScratchHeadroom - if avail, err := diskAvail("/dev/shm"); err == nil && avail >= need { - f, err := os.CreateTemp("/dev/shm", "ftw-backup-*.raw") + avail, err := diskAvail(backupShmDir) + if err == nil && avail >= need { + f, err := os.CreateTemp(backupShmDir, "ftw-backup-*.raw") if err == nil { name := f.Name() _ = f.Close() @@ -142,10 +153,25 @@ func backupScratchFile(dstPath string, sourceBytes int64) (path string, cleanup return name, func() { _ = os.Remove(name) } } } + if err == nil && avail < need { + // A backup cut short by a restart leaves its scratch here until + // reboot; on the home box that sent the next copy to the SD card. + slog.Warn("backup scratch does not fit in /dev/shm; copying on the data disk is slower and delays history writes", + "need_bytes", need, "free_bytes", avail, "backup_scratch_in_shm_bytes", shmBackupScratchBytes()) + } path = dstPath + ".raw.tmp" return path, func() { _ = os.Remove(path) } } +func shmBackupScratchBytes() int64 { + names, _ := filepath.Glob(filepath.Join(backupShmDir, "ftw-backup-*.raw*")) + var n int64 + for _, name := range names { + n += fileSizeOrZero(name) + } + return n +} + // EnsureDiskSpace refuses to start a backup when dir cannot hold needed bytes. // A probe error (including Windows) does not block; the copy still fails if // the filesystem fills. diff --git a/go/internal/state/diagnostics.go b/go/internal/state/diagnostics.go index 5c4eb3625..5c38ef15a 100644 --- a/go/internal/state/diagnostics.go +++ b/go/internal/state/diagnostics.go @@ -86,6 +86,19 @@ func (s *Store) LoadDiagnosticsInRange(sinceMs, untilMs int64, return out, rows.Err() } +// PlannerDay sums the plans kept since sinceMs: how many, the longest any +// took to solve in milliseconds, and how many were a fallback plan. The +// anonymous fleet report coarsens these into buckets; nothing else of a plan +// leaves this query. +func (s *Store) PlannerDay(sinceMs int64) (plans int, slowestMs float64, fallbacks int, err error) { + err = s.db.QueryRow( + `SELECT COUNT(*), + COALESCE(MAX(json_extract(json, '$.solver.solve_ms')), 0), + COALESCE(SUM(json_extract(json, '$.solver.fallback') = 1), 0) + FROM planner_diagnostics WHERE ts_ms >= ?`, sinceMs).Scan(&plans, &slowestMs, &fallbacks) + return plans, slowestMs, fallbacks, err +} + // LoadDiagnosticAt returns the snapshot that was active at tsMs — the // most recent one with ts_ms <= tsMs. This is the correct semantics for // "what plan was driving the EMS at this moment?": if a replan ran at diff --git a/go/internal/state/diagnostics_test.go b/go/internal/state/diagnostics_test.go index 32cd4790d..a09314019 100644 --- a/go/internal/state/diagnostics_test.go +++ b/go/internal/state/diagnostics_test.go @@ -223,3 +223,30 @@ func itoa(i int) string { } return string(buf[pos:]) } + +// The fleet report's planner facts: plans kept since a time, the slowest +// solve and the fallbacks. Older plans and plans without solver details do +// not move the slowest. +func TestPlannerDaySumsThePlansSinceATime(t *testing.T) { + st := openTestStore(t) + day := time.Date(2026, 10, 24, 0, 0, 0, 0, time.UTC).UnixMilli() + for i, js := range []string{ + `{"solver":{"solve_ms":90000,"fallback":true}}`, // before the window + `{"solver":{"solve_ms":850}}`, + `{"solver":{"solve_ms":4200,"status":"fallback","fallback":true}}`, + `{"zone":"SE3"}`, + `{"solver":{"solve_ms":12.5,"fallback":false}}`, + } { + ts := day + int64(i-1)*3_600_000 + if err := st.SaveDiagnostic(ts, "scheduled", "SE3", 0, 96, js); err != nil { + t.Fatal(err) + } + } + plans, slowest, fallbacks, err := st.PlannerDay(day) + if err != nil || plans != 4 || slowest != 4200 || fallbacks != 1 { + t.Fatalf("plans %d, slowest %v, fallbacks %d, err %v", plans, slowest, fallbacks, err) + } + if plans, slowest, fallbacks, err := st.PlannerDay(day + 10*3_600_000); err != nil || plans != 0 || slowest != 0 || fallbacks != 0 { + t.Fatalf("an empty window: %d %v %d %v", plans, slowest, fallbacks, err) + } +} diff --git a/go/internal/state/history_write_timing.go b/go/internal/state/history_write_timing.go index 6f8efadba..f5392c7ef 100644 --- a/go/internal/state/history_write_timing.go +++ b/go/internal/state/history_write_timing.go @@ -15,6 +15,8 @@ type historyWriteTiming struct { phases [12]struct { name string elapsed time.Duration + max time.Duration + steps int } n int } @@ -34,11 +36,14 @@ func (t *historyWriteTiming) add(phase string, elapsed time.Duration) { for i := 0; i < t.n; i++ { if t.phases[i].name == phase { t.phases[i].elapsed += elapsed + t.phases[i].max = max(t.phases[i].max, elapsed) + t.phases[i].steps++ return } } if t.n < len(t.phases) { t.phases[t.n].name, t.phases[t.n].elapsed = phase, elapsed + t.phases[t.n].max, t.phases[t.n].steps = elapsed, 1 t.n++ } } @@ -51,7 +56,8 @@ func (t *historyWriteTiming) failure(err error) error { t.add(t.phase, now.Sub(t.step)) var details strings.Builder for i := 0; i < t.n; i++ { - fmt.Fprintf(&details, " %s_ms=%d", t.phases[i].name, t.phases[i].elapsed.Milliseconds()) + p := &t.phases[i] + fmt.Fprintf(&details, " %s_ms=%d %s_max_ms=%d %s_steps=%d", p.name, p.elapsed.Milliseconds(), p.name, p.max.Milliseconds(), p.name, p.steps) } return fmt.Errorf("%w [elapsed_ms=%d phase=%s step_ms=%d%s]", err, now.Sub(t.started).Milliseconds(), t.phase, now.Sub(t.step).Milliseconds(), details.String()) } diff --git a/go/internal/state/history_write_timing_test.go b/go/internal/state/history_write_timing_test.go index ee70b5dfb..8c3d8c2cd 100644 --- a/go/internal/state/history_write_timing_test.go +++ b/go/internal/state/history_write_timing_test.go @@ -60,11 +60,16 @@ func TestHistoryWriteFailureReportsSQLTimingsAndRetries(t *testing.T) { if !errors.Is(err, context.Canceled) { t.Fatalf("timings lost the original error: %v", err) } - for _, want := range []string{"history tick: samples:", "phase=" + phase, "elapsed_ms=", "step_ms=", "prepare_ms=", "writer_lock_ms=", "connection_ms=", "begin_ms=", "tick_ms="} { + for _, want := range []string{"history tick: samples:", "phase=" + phase, "elapsed_ms=", "step_ms=", "prepare_ms=", "writer_lock_ms=", "connection_ms=", "begin_ms=", "tick_ms=", phase + "_max_ms=", phase + "_steps=1"} { if !strings.Contains(err.Error(), want) { t.Fatalf("missing %q: %v", want, err) } } + for _, private := range []string{"timing", "payload", "grid_w", fmt.Sprint(base + 1000)} { + if strings.Contains(err.Error(), private) { + t.Fatalf("failure timings exposed a sample or batch value: %v", err) + } + } for i := 0; i < 2; i++ { if _, err := s.recordHistoryBatches(context.Background(), []historyBatch{batch}, 0); err != nil { t.Fatal("retry:", err) @@ -79,6 +84,21 @@ func TestHistoryWriteFailureReportsSQLTimingsAndRetries(t *testing.T) { } } +func TestHistoryWriteTimingSeparatesRepeatedStepMaximumFromTotal(t *testing.T) { + timing := newHistoryWriteTiming("other") + timing.add("read_bucket", 2*time.Millisecond) + timing.add("read_bucket", 7*time.Millisecond) + err := timing.failure(context.Canceled) + if !errors.Is(err, context.Canceled) { + t.Fatalf("timing lost cancellation: %v", err) + } + for _, want := range []string{"read_bucket_ms=9", "read_bucket_max_ms=7", "read_bucket_steps=2"} { + if !strings.Contains(err.Error(), want) { + t.Fatalf("repeated step timing missing %q: %v", want, err) + } + } +} + func TestHistoryWriteFailureReportsWriterLockWait(t *testing.T) { s := freshStore(t) if err := s.RecordSamples([]Sample{{Driver: "meter", Metric: "grid_w", TsMs: 1000, Value: 10}}); err != nil { diff --git a/go/internal/state/store.go b/go/internal/state/store.go index 0ab26cafb..8a42ba7e9 100644 --- a/go/internal/state/store.go +++ b/go/internal/state/store.go @@ -621,7 +621,7 @@ func (s *Store) backupToCompressed(dstPath string, report func(BackupProgress), return s.backupToCompressedContext(context.Background(), dstPath, report, capture) } -func (s *Store) backupToCompressedContext(parent context.Context, dstPath string, report func(BackupProgress), capture func(string) error) error { +func (s *Store) backupToCompressedContext(parent context.Context, dstPath string, report func(BackupProgress), capture func(string) error) (err error) { if s == nil || s.db == nil { return fmt.Errorf("store: backup on nil store") } @@ -633,6 +633,11 @@ func (s *Store) backupToCompressedContext(parent context.Context, dstPath string ctx, cancel := s.backupWorkContextParent(parent) defer cancel() + defer func() { + if cause := context.Cause(ctx); err != nil && errors.Is(cause, errLiveBackupLimit) { + err = fmt.Errorf("%w: %w", cause, err) + } + }() ctx = context.WithValue(ctx, backupReportKey{}, report) sourceBytes := s.BackupSourceBytes() if err := EnsureDiskSpace(filepath.Dir(dstPath), backupCopyScratch(sourceBytes)); err != nil { @@ -641,6 +646,8 @@ func (s *Store) backupToCompressedContext(parent context.Context, dstPath string rawPath, cleanupRaw := backupScratchFile(dstPath, sourceBytes) defer cleanupRaw() + release := HoldBackupScratch(filepath.Dir(rawPath)) + defer release() reportBackupProgress(report, BackupProgress{Phase: BackupPhaseCopying}) if err := s.copyStateForBackup(ctx, rawPath); err != nil { return fmt.Errorf("backup state: %w", err) diff --git a/go/internal/statsrelay/statsrelay.go b/go/internal/statsrelay/statsrelay.go new file mode 100644 index 000000000..44c81f515 --- /dev/null +++ b/go/internal/statsrelay/statsrelay.go @@ -0,0 +1,162 @@ +// Package statsrelay is how the box's anonymous daily reports leave it once a +// release names a relay: each report alone, sealed with Oblivious HTTP +// (RFC 9458) to FTW Cloud's gateway key and posted to a relay that someone +// other than Sourceful runs. The relay sees the box's address but not the +// report; FTW Cloud reads the report but not the address. +// +// Two reports use it, the fleet report (package fleetping) and the energy +// report (package energyreport), each at its own random time and never in +// one message. +package statsrelay + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/base64" + "errors" + "fmt" + "io" + "mime" + "net/http" + "net/url" + "strings" + "time" + + "github.com/srcfl/ftw/go/protocol/ohttp" + "github.com/srcfl/ftw/go/protocol/reportwire" +) + +// Compile-time settings, as in cloudlink: a relay a household could point +// elsewhere is a relay a support call could point at an attacker's. A local +// test build may set them with -ldflags -X. +var ( + // RelayURL is the OHTTP relay, run by someone other than Sourceful. + // Empty until that relay exists: the energy report is then not sent, + // and the fleet report goes straight to FTW Cloud as before. + RelayURL = "" + // GatewayKey is FTW Cloud's gateway key configuration (RFC 9458, + // section 3), base64url, as + // https://cloud.ftw.energy/.well-known/ohttp-gateway serves it. The box + // encrypts only to this key and never asks the service for one, so every + // box on a release uses the same key and a key swapped in at the service + // reads nothing. Empty means no gateway key exists for this build. + GatewayKey = "" +) + +const ( + sendTimeout = 30 * time.Second + maxResponseBytes = 1 << 10 +) + +// Client posts sealed reports to one relay under one gateway key. +type Client struct { + relay string + key ohttp.KeyConfig + http *http.Client + now func() time.Time +} + +// Pinned is this build's client, or nil when the build names no usable relay +// and gateway key. +func Pinned() *Client { + relay, key, ok := pinned(RelayURL, GatewayKey) + if !ok { + return nil + } + return New(relay, key, nil) +} + +func pinned(relay, key string) (string, ohttp.KeyConfig, bool) { + raw, err := base64.RawURLEncoding.DecodeString(key) + if err != nil || relay == "" || key == "" { + return "", ohttp.KeyConfig{}, false + } + cfg, err := ohttp.ParseKeyConfig(raw) + if err != nil { + return "", ohttp.KeyConfig{}, false + } + u, err := url.Parse(relay) + if err != nil { + return "", ohttp.KeyConfig{}, false + } + // Host names compare without case and without a trailing dot, so FTW + // Cloud cannot pass as its own relay under another spelling. + host := strings.TrimSuffix(u.Hostname(), ".") + if u.Scheme != "https" || u.Host == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || strings.EqualFold(host, reportwire.Authority) { + return "", ohttp.KeyConfig{}, false + } + return relay, cfg, true +} + +// New is a client for one relay and key. Tests point it at a fake relay. +func New(relay string, key ohttp.KeyConfig, client *http.Client) *Client { + if client == nil { + client = &http.Client{Timeout: sendTimeout} + } else { + c := *client + client = &c + } + // A redirect would resend the sealed report to an address nobody chose. + client.CheckRedirect = func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse } + return &Client{relay: relay, key: key, http: client, now: time.Now} +} + +// Relay is the relay's URL, for Settings. +func (c *Client) Relay() string { return c.relay } + +// Send seals one report body and reads the gateway's answer. Only a sealed +// 204 from the gateway the box encrypted to counts as delivered. +// +// Each call is one report: it draws a fresh id, which the gateway counts +// once. Nothing retries; a retry of the same report would have to reuse its +// id, or the gateway would count it twice. +func (c *Client) Send(ctx context.Context, body []byte) error { + id := make([]byte, reportwire.IDBytes) + if _, err := rand.Read(id); err != nil { + return err + } + inner, ok := reportwire.Request(reportwire.Report{ID: id, Day: c.now().UTC().Format(time.DateOnly), Body: body}) + if !ok { + return errors.New("statsrelay: the report does not fit the padded request") + } + sealed, reader, err := ohttp.Encapsulate(c.key, inner) + if err != nil { + return err + } + ctx, cancel := context.WithTimeout(ctx, sendTimeout) + defer cancel() + req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.relay, bytes.NewReader(sealed)) + if err != nil { + return err + } + req.Header.Set("Content-Type", ohttp.RequestType) + // Pinned, as for the fleet report's direct path: the relay sees it, and + // Go's default names the runtime's version. + req.Header.Set("User-Agent", "ftw-report") + resp, err := c.http.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + data, err := io.ReadAll(io.LimitReader(resp.Body, maxResponseBytes+1)) + if err != nil { + return err + } + media, _, _ := mime.ParseMediaType(resp.Header.Get("Content-Type")) + if resp.StatusCode != http.StatusOK || media != ohttp.ResponseType || len(data) > maxResponseBytes { + return fmt.Errorf("statsrelay: relay answered %d %q", resp.StatusCode, media) + } + msg, err := reader.Open(data) + if err != nil { + return err + } + answer, err := ohttp.ParseResponse(msg) + if err != nil { + return err + } + if answer.Status != http.StatusNoContent { + return fmt.Errorf("statsrelay: gateway answered %d", answer.Status) + } + return nil +} diff --git a/go/internal/statsrelay/statsrelay_test.go b/go/internal/statsrelay/statsrelay_test.go new file mode 100644 index 000000000..00d4559e1 --- /dev/null +++ b/go/internal/statsrelay/statsrelay_test.go @@ -0,0 +1,133 @@ +package statsrelay_test + +import ( + "bytes" + "context" + "crypto/ecdh" + "encoding/base64" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/statsrelay" + "github.com/srcfl/ftw/go/internal/statsrelay/statsrelaytest" + "github.com/srcfl/ftw/go/protocol/energywire" + "github.com/srcfl/ftw/go/protocol/ohttp" + "github.com/srcfl/ftw/go/protocol/reportwire" +) + +var now = time.Date(2026, 10, 24, 9, 0, 0, 0, time.UTC) + +// The release names no relay and no gateway key yet, so no report goes +// through one. Once both are pinned they must be a usable pair. +func TestTheReleasePinsBothOrNeither(t *testing.T) { + c := statsrelay.Pinned() + switch { + case statsrelay.RelayURL == "" && statsrelay.GatewayKey == "": + if c != nil { + t.Fatal("nothing pinned, yet a client") + } + case c == nil: + t.Fatalf("the pinned relay %q and gateway key do not make a usable pair", statsrelay.RelayURL) + } +} + +func TestAPinIsRefusedUnlessItIsARelayAndAKey(t *testing.T) { + k, _ := ecdh.X25519().GenerateKey(nil) + gw, _ := ohttp.NewGateway(ohttp.Key{ID: 1, Private: k}) + key := base64.RawURLEncoding.EncodeToString(gw.Configs()[0].Marshal()) + oldRelay, oldKey := statsrelay.RelayURL, statsrelay.GatewayKey + t.Cleanup(func() { statsrelay.RelayURL, statsrelay.GatewayKey = oldRelay, oldKey }) + set := func(relay, key string) *statsrelay.Client { + statsrelay.RelayURL, statsrelay.GatewayKey = relay, key + return statsrelay.Pinned() + } + if c := set("https://relay.example.net/ftw", key); c == nil || c.Relay() != "https://relay.example.net/ftw" { + t.Fatal("a good pair was refused") + } + for name, p := range map[string][2]string{ + "no relay": {"", key}, + "no key": {"https://relay.example.net/ftw", ""}, + "plain http": {"http://relay.example.net/ftw", key}, + "FTW Cloud itself": {"https://cloud.ftw.energy/v1/ohttp", key}, + "FTW Cloud, cased": {"https://Cloud.FTW.Energy/v1/ohttp", key}, + "FTW Cloud, dot": {"https://cloud.ftw.energy./v1/ohttp", key}, + "query": {"https://relay.example.net/ftw?box=1", key}, + "credentials": {"https://a:b@relay.example.net/ftw", key}, + "bad key": {"https://relay.example.net/ftw", "AAAA"}, + "padded base64": {"https://relay.example.net/ftw", key + "="}, + } { + if set(p[0], p[1]) != nil { + t.Fatalf("%s was accepted", name) + } + } +} + +// The relay sees one length and no report; the gateway reads it. +func TestAReportReachesTheGatewayThroughTheRelay(t *testing.T) { + client, gw, relay := statsrelaytest.New(t, now) + body, _ := json.Marshal(energywire.Report{Schema: energywire.Schema, Date: "2026-10-23", GridImport: 24}) + if err := client.Send(context.Background(), body); err != nil { + t.Fatal(err) + } + if got := gw.Energy(); len(got) != 1 || got[0].GridImport != 24 { + t.Fatalf("gateway got %+v", got) + } + seen := relay.Bodies()[0] + if bytes.Contains(seen, []byte("kwh")) || len(seen) != 7+32+reportwire.RequestSize+16 { + t.Fatalf("the relay saw %d bytes", len(seen)) + } + h := relay.Headers()[0] + if h.Get("Content-Type") != ohttp.RequestType || h.Get("User-Agent") != "ftw-report" { + t.Fatalf("relay headers %v", h) + } + if err := client.Send(context.Background(), make([]byte, reportwire.RequestSize)); err == nil { + t.Fatal("a report too large for the padding went out") + } +} + +// The box believes only a sealed 204 from the gateway it encrypted to. +func TestOnlyTheGatewaysSealed204Counts(t *testing.T) { + k, _ := ecdh.X25519().GenerateKey(nil) + gw, _ := ohttp.NewGateway(ohttp.Key{ID: 1, Private: k}) + sealed := func(status int) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + _, ctx, err := gw.Decapsulate(body) + if err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + answer, _ := ctx.Seal(reportwire.Response(status)) + w.Header().Set("Content-Type", ohttp.ResponseType) + _, _ = w.Write(answer) + } + } + body := []byte(`{"schema":"x"}`) + for name, h := range map[string]http.HandlerFunc{ + "relay error": func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusBadGateway) }, + "plain 204": func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNoContent) }, + "forged body": func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", ohttp.ResponseType) + _, _ = w.Write(make([]byte, 64)) + }, + "sealed 400": sealed(http.StatusBadRequest), + "redirect": func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, "https://elsewhere.example/", http.StatusTemporaryRedirect) + }, + } { + srv := httptest.NewServer(h) + if err := statsrelay.New(srv.URL, gw.Configs()[0], nil).Send(context.Background(), body); err == nil { + t.Fatalf("%s: accepted", name) + } + srv.Close() + } + srv := httptest.NewServer(sealed(http.StatusNoContent)) + defer srv.Close() + if err := statsrelay.New(srv.URL, gw.Configs()[0], nil).Send(context.Background(), body); err != nil { + t.Fatal(err) + } +} diff --git a/go/internal/statsrelay/statsrelaytest/statsrelaytest.go b/go/internal/statsrelay/statsrelaytest/statsrelaytest.go new file mode 100644 index 000000000..1a30e5bfa --- /dev/null +++ b/go/internal/statsrelay/statsrelaytest/statsrelaytest.go @@ -0,0 +1,146 @@ +// Package statsrelaytest is a fake relay and gateway for tests. The gateway +// reads a report as FTW Cloud's does: one padded inner request, its decoder +// chosen by schema. It keeps the reports for the test to read. +package statsrelaytest + +import ( + "bytes" + "crypto/ecdh" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" + "time" + + "github.com/srcfl/ftw/go/internal/statsrelay" + "github.com/srcfl/ftw/go/protocol/energywire" + "github.com/srcfl/ftw/go/protocol/fleetwire" + "github.com/srcfl/ftw/go/protocol/ohttp" + "github.com/srcfl/ftw/go/protocol/reportwire" +) + +// Gateway is a fake FTW Cloud gateway. +type Gateway struct { + *httptest.Server + Key ohttp.KeyConfig + + mu sync.Mutex + now time.Time + fleet []fleetwire.Report + energy []energywire.Report + last int +} + +// Relay forwards only the body and its content type, and records what it saw. +type Relay struct { + *httptest.Server + + mu sync.Mutex + bodies [][]byte + headers []http.Header +} + +// New starts a gateway whose clock reads now, a relay in front of it, and a +// client posting to the relay. +func New(t testing.TB, now time.Time) (*statsrelay.Client, *Gateway, *Relay) { + t.Helper() + k, err := ecdh.X25519().GenerateKey(nil) + if err != nil { + t.Fatal(err) + } + gw, err := ohttp.NewGateway(ohttp.Key{ID: 7, Private: k}) + if err != nil { + t.Fatal(err) + } + g := &Gateway{Key: gw.Configs()[0], now: now} + g.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + msg, sealer, err := gw.Decapsulate(body) + if err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + status := g.add(msg) + sealed, _ := sealer.Seal(reportwire.Response(status)) + w.Header().Set("Content-Type", ohttp.ResponseType) + _, _ = w.Write(sealed) + })) + t.Cleanup(g.Close) + + rl := &Relay{} + rl.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + rl.mu.Lock() + rl.bodies = append(rl.bodies, body) + rl.headers = append(rl.headers, r.Header.Clone()) + rl.mu.Unlock() + resp, err := http.Post(g.URL, r.Header.Get("Content-Type"), bytes.NewReader(body)) + if err != nil { + w.WriteHeader(http.StatusBadGateway) + return + } + defer resp.Body.Close() + w.Header().Set("Content-Type", resp.Header.Get("Content-Type")) + w.WriteHeader(resp.StatusCode) + _, _ = io.Copy(w, resp.Body) + })) + t.Cleanup(rl.Close) + return statsrelay.New(rl.URL, g.Key, nil), g, rl +} + +func (g *Gateway) add(msg []byte) int { + g.mu.Lock() + defer g.mu.Unlock() + g.last = http.StatusBadRequest + report, ok := reportwire.Read(msg) + if !ok || !reportwire.Fresh(report.Day, g.now) { + return g.last + } + body := report.Body + switch reportwire.Schema(body) { + case fleetwire.Schema: + if r, err := fleetwire.Decode(body); err == nil { + g.fleet, g.last = append(g.fleet, r), http.StatusNoContent + } + case energywire.Schema: + if r, err := energywire.Decode(body, g.now); err == nil { + g.energy, g.last = append(g.energy, r), http.StatusNoContent + } + } + return g.last +} + +// Fleet is every fleet report the gateway took. +func (g *Gateway) Fleet() []fleetwire.Report { + g.mu.Lock() + defer g.mu.Unlock() + return append([]fleetwire.Report(nil), g.fleet...) +} + +// Energy is every energy report the gateway took. +func (g *Gateway) Energy() []energywire.Report { + g.mu.Lock() + defer g.mu.Unlock() + return append([]energywire.Report(nil), g.energy...) +} + +// Answered is the gateway's last inner status. +func (g *Gateway) Answered() int { + g.mu.Lock() + defer g.mu.Unlock() + return g.last +} + +// Bodies is every body the relay forwarded, and Headers what came with each. +func (r *Relay) Bodies() [][]byte { + r.mu.Lock() + defer r.mu.Unlock() + return append([][]byte(nil), r.bodies...) +} + +func (r *Relay) Headers() []http.Header { + r.mu.Lock() + defer r.mu.Unlock() + return append([]http.Header(nil), r.headers...) +} diff --git a/go/internal/telemetry/battery_control_powers.go b/go/internal/telemetry/battery_control_powers.go new file mode 100644 index 000000000..f7416cd35 --- /dev/null +++ b/go/internal/telemetry/battery_control_powers.go @@ -0,0 +1,31 @@ +package telemetry + +import "time" + +// OnlineBatteryControlPowers returns qualified battery observations for +// diagnostic context. Callers may align them with another source, but cached +// readings alone never establish a live battery contribution. +func (s *Store) OnlineBatteryControlPowers(now time.Time, maxAge time.Duration) []ControlObservation { + s.mu.RLock() + defer s.mu.RUnlock() + if maxAge <= 0 { + return nil + } + var out []ControlObservation + for _, r := range s.readings { + if r == nil || r.DerType != DerBattery { + continue + } + h := s.health[r.Driver] + if h == nil || !h.TelemetryLive() || h.DeviceFault || r.UpdatedAt.After(now) || now.Sub(r.UpdatedAt) > maxAge { + continue + } + obs, known := ControlPowerObservation(r.RawW, r.Data, r.UpdatedAt) + age := now.Sub(obs.Seen()) + if !known || age < 0 || age > maxAge || age > ControlPowerMaxAge(r.Data) { + continue + } + out = append(out, obs) + } + return out +} diff --git a/go/internal/telemetry/battery_control_powers_test.go b/go/internal/telemetry/battery_control_powers_test.go new file mode 100644 index 000000000..277b7e4c2 --- /dev/null +++ b/go/internal/telemetry/battery_control_powers_test.go @@ -0,0 +1,54 @@ +package telemetry + +import ( + "math" + "testing" + "time" +) + +func TestOnlineBatteryControlPowersQualifiesDiagnosticContext(t *testing.T) { + for _, mode := range []string{"fresh", "offline", "fault", "missing health", "stale receipt", "future receipt", "stale source", "source validity", "unavailable", "nonfinite", "invalid age"} { + t.Run(mode, func(t *testing.T) { + now := time.Now().UTC() + s := NewStore() + s.Update("battery", DerBattery, 1500, nil, nil) + s.RecordDriverSuccess("battery") + r := s.Get("battery", DerBattery) + r.UpdatedAt = now + maxAge := 15 * time.Second + switch mode { + case "offline": + s.DriverHealthMut("battery").SetOffline() + case "fault": + s.DriverHealthMut("battery").SetDeviceFault(true, "test") + case "missing health": + delete(s.health, "battery") + case "stale receipt": + r.UpdatedAt = now.Add(-16 * time.Second) + case "future receipt": + r.UpdatedAt = now.Add(time.Second) + case "stale source": + r.Data = []byte(`{"power_observed_at":"` + now.Add(-16*time.Second).Format(time.RFC3339Nano) + `"}`) + case "source validity": + r.Data = []byte(`{"power_max_age_s":1,"power_observed_at":"` + now.Add(-2*time.Second).Format(time.RFC3339Nano) + `"}`) + case "unavailable": + r.Data = []byte(`{"control_power_available":false}`) + case "nonfinite": + r.RawW = math.NaN() + case "invalid age": + maxAge = 0 + } + // Another DER must not enter battery context. + s.Update("car", DerEV, 8000, nil, nil) + s.RecordDriverSuccess("car") + got := s.OnlineBatteryControlPowers(now, maxAge) + if mode == "fresh" { + if len(got) != 1 || got[0].PowerW != 1500 { + t.Fatalf("observations %+v", got) + } + } else if len(got) != 0 { + t.Fatalf("unqualified context %+v", got) + } + }) + } +} diff --git a/go/internal/telemetry/command_evidence.go b/go/internal/telemetry/command_evidence.go index 6523c2bf9..5761d98a8 100644 --- a/go/internal/telemetry/command_evidence.go +++ b/go/internal/telemetry/command_evidence.go @@ -26,10 +26,13 @@ type CommandEvidence struct { Result string ReadbackMismatchSince time.Time PowerMismatchSince time.Time - PowerMatchSince time.Time - PowerObservedSince time.Time - LastObservation time.Time - LastMeasuredAt time.Time + // PowerOverSince is the start of the current run of readings above + // every command in force, a part of PowerMismatchSince's run. + PowerOverSince time.Time + PowerMatchSince time.Time + PowerObservedSince time.Time + LastObservation time.Time + LastMeasuredAt time.Time // NoPower is an observed EV stop during uninterrupted positive orders. // Retuning the current cannot restart this clock; a pause, failed command, // missing measurement or observed power ends it. @@ -101,10 +104,16 @@ func (s *Store) BeginCommand(driver string, payload []byte, now time.Time) Comma var request struct { Action string `json:"action"` PowerW *float64 `json:"power_w"` + // ExpectedW is what the device should draw when it differs from + // the order: a one-phase car takes a third of a three-phase one. + ExpectedW *float64 `json:"expected_w"` } if json.Unmarshal(payload, &request) != nil { return CommandEvidence{} } + if request.ExpectedW != nil && finite(*request.ExpectedW) && *request.ExpectedW >= 0 { + request.PowerW = request.ExpectedW + } // Resume changes permission to draw, not the current limit. Keep the // accepted limit and its measured response while this optional call runs. if request.Action == "ev_resume" && request.PowerW == nil { @@ -152,10 +161,13 @@ func (s *Store) BeginCommand(driver string, payload []byte, now time.Time) Comma if !materialStep(old.PowerW, c.PowerW) { c.Since, c.Baseline, c.StepAfter, c.StepAfterAt = old.Since, old.Baseline, old.StepAfter, old.StepAfterAt c.ReadbackMismatchSince, c.PowerMismatchSince, c.PowerMatchSince = old.ReadbackMismatchSince, old.PowerMismatchSince, old.PowerMatchSince + c.PowerOverSince = old.PowerOverSince c.PowerObservedSince, c.LastObservation, c.LastGapW = old.PowerObservedSince, old.LastObservation, old.LastGapW } } - if c.PowerW != nil { + // A repeat of the last order is the same order: keeping it would push + // the earlier ones out of Recent long before their response time ends. + if n := len(c.Recent); c.PowerW != nil && (n == 0 || c.Recent[n-1].PowerW != *c.PowerW) { c.Recent = append(c.Recent, CommandPoint{PowerW: *c.PowerW, At: now}) } if len(c.Recent) > 8 { @@ -292,6 +304,7 @@ func (s *Store) endCommandControlAt(driver string, failed bool, now time.Time) { c.Result = "released" } c.ReadbackMismatchSince, c.PowerMismatchSince, c.PowerMatchSince = time.Time{}, time.Time{}, time.Time{} + c.PowerOverSince = time.Time{} c.PowerObservedSince, c.LastObservation, c.LastGapW = time.Time{}, time.Time{}, nil c.Recent, c.StepAfter, c.StepAfterAt = nil, nil, time.Time{} if !failed { @@ -375,14 +388,15 @@ func (s *Store) observeCommand(driver string, kind DerType, power float64, data if json.Unmarshal(data, &d) != nil { c.NoPowerSince, c.NoPowerAt = time.Time{}, time.Time{} c.ReadbackMismatchSince, c.PowerMismatchSince, c.PowerMatchSince = time.Time{}, time.Time{}, time.Time{} + c.PowerOverSince = time.Time{} c.PowerObservedSince, c.LastObservation = time.Time{}, time.Time{} s.commands[k] = c return } - c.MaxAge = controlPowerDeclaredMaxAge(data) + c.MaxAge = DeclaredPowerMaxAge(data) if now.Sub(c.LastObservation) > max(time.Minute, c.MaxAge) { c.ReadbackMismatchSince, c.PowerMismatchSince, c.PowerMatchSince = time.Time{}, time.Time{}, time.Time{} - c.PowerObservedSince = time.Time{} + c.PowerOverSince, c.PowerObservedSince = time.Time{}, time.Time{} } mark := func(since *time.Time, mismatch bool, at time.Time) { if !mismatch { @@ -423,7 +437,7 @@ func (s *Store) observeCommand(driver string, kind DerType, power float64, data // delivery of one source sample also cannot extend a measured time window. if !fresh || !finite(power) || observedAt.Before(c.Since) { c.PowerMismatchSince, c.PowerMatchSince, c.LastObservation = time.Time{}, time.Time{}, time.Time{} - c.PowerObservedSince, c.LastGapW = time.Time{}, nil + c.PowerOverSince, c.PowerObservedSince, c.LastGapW = time.Time{}, time.Time{}, nil s.commands[k] = c return } @@ -441,6 +455,8 @@ func (s *Store) observeCommand(driver string, kind DerType, power float64, data tolerance := ControlToleranceW(*c.PowerW) mismatch := !finite(gap) || gap > tolerance mark(&c.PowerMismatchSince, mismatch, observedAt) + // Above the latest command and off every one in force is above them all. + mark(&c.PowerOverSince, mismatch && kind != DerPV && power > *c.PowerW, observedAt) follows := !mismatch if kind == DerPV { // Weak sun alone cannot prove that a ceiling took effect. diff --git a/go/internal/telemetry/command_evidence_test.go b/go/internal/telemetry/command_evidence_test.go index 1e2c7410e..84125808b 100644 --- a/go/internal/telemetry/command_evidence_test.go +++ b/go/internal/telemetry/command_evidence_test.go @@ -178,3 +178,43 @@ func TestLastMeasuredResponseSurvivesNewCommandButNotRelease(t *testing.T) { t.Fatal("intentional release kept loss alarm armed") } } + +// Core sends a charger its order on every tick. A repeated order must not +// push the one before it out of the response window: a cloud charger may +// still show it for two minutes. +func TestRepeatedOrderKeepsEarlierOrderInForce(t *testing.T) { + s := NewStore() + start := time.Now() + send := func(w string, at time.Time) { + c := s.BeginCommand("easee", []byte(`{"action":"ev_set_current","power_w":`+w+`}`), at) + s.CompleteCommand(c, "accepted") + } + send("7590", start) + for i := range 30 { + send("0", start.Add(time.Duration(10+2*i)*time.Second)) + } + at := start.Add(70 * time.Second) + s.mu.Lock() + s.observeCommand("easee", DerEV, 3130, json.RawMessage(`{}`), at) + s.mu.Unlock() + got, _ := s.CommandEvidence("easee", "ev") + if got.LastGapW == nil || *got.LastGapW != 0 || !got.PowerMismatchSince.IsZero() { + t.Fatalf("1 min after a cut the earlier 7590 W order was out of force: gap %v, mismatch since %v", got.LastGapW, got.PowerMismatchSince) + } +} + +// A one-phase car takes a third of a three-phase order. The command says +// what it should draw, and the evidence compares the car with that. +func TestExpectedDrawReplacesTheOrder(t *testing.T) { + s := NewStore() + start := time.Now() + c := s.BeginCommand("easee", []byte(`{"action":"ev_set_current","power_w":11100,"expected_w":3700}`), start) + s.CompleteCommand(c, "accepted") + s.mu.Lock() + s.observeCommand("easee", DerEV, 3680, json.RawMessage(`{}`), start.Add(3*time.Minute)) + s.mu.Unlock() + got, _ := s.CommandEvidence("easee", "ev") + if got.PowerW == nil || *got.PowerW != 3700 || !got.PowerMismatchSince.IsZero() { + t.Fatalf("evidence %+v", got) + } +} diff --git a/go/internal/telemetry/control_observations.go b/go/internal/telemetry/control_observations.go index 59612953f..2da936546 100644 --- a/go/internal/telemetry/control_observations.go +++ b/go/internal/telemetry/control_observations.go @@ -183,13 +183,15 @@ func (p ControlObservation) Seen() time.Time { // source's power_max_age_s when it declares one, at most three minutes, // otherwise one minute. func ControlPowerMaxAge(data json.RawMessage) time.Duration { - if age := controlPowerDeclaredMaxAge(data); age > 0 { + if age := DeclaredPowerMaxAge(data); age > 0 { return age } return time.Minute } -func controlPowerDeclaredMaxAge(data json.RawMessage) time.Duration { +// DeclaredPowerMaxAge is the power_max_age_s a source declares, at most +// three minutes, otherwise 0. +func DeclaredPowerMaxAge(data json.RawMessage) time.Duration { var d struct { PowerMaxAgeS float64 `json:"power_max_age_s"` } diff --git a/go/internal/telemetry/forecast.go b/go/internal/telemetry/forecast.go index b5a31152f..386023715 100644 --- a/go/internal/telemetry/forecast.go +++ b/go/internal/telemetry/forecast.go @@ -45,8 +45,33 @@ type ForecastOptions struct { // Live display may use the driver's measured site-side control power // (for example Pixii AC instead of battery DC). Raw history stays separate. UseControlPower bool `json:",omitempty"` + // PVDark says how dark it is at the site at a time. Live Home use counts + // an unknown PV source as 0 W after sunset. Nil, or no site position, + // infers nothing. Not part of a forecast revision's identity. + PVDark func(time.Time) Darkness `json:"-"` } +// Darkness is the sun's state at the site, for unknown PV. +type Darkness int + +const ( + // DarknessUnknown: daylight, or no site position. + DarknessUnknown Darkness = iota + // DarknessTwilight: after sunset, before civil dusk. Diffuse light only. + DarknessTwilight + // DarknessNight: past civil dusk. PV produces nothing. + DarknessNight +) + +// Live Home use quality. Measured: every flow read, each by its driver's latest +// poll. Estimated: the site meter is read and a flow is left out, assumed or +// held after a failed read. Unknown: no usable site meter. +const ( + LoadMeasured = "measured" + LoadEstimated = "estimated" + LoadUnknown = "unknown" +) + // ForecastReading is one coherent snapshot, in site signs and raw watts. // Valid describes the household balance. PVValid only describes the PV sources. // Neither flag claims that the instantaneous sample covers a time interval. @@ -54,6 +79,8 @@ type ForecastSource struct { Flow ForecastFlow PowerW *float64 SoC *float64 + // Dark: an unread PV source after sunset, shown as 0 W. Not a reading. + Dark bool } type ForecastReading struct { @@ -63,6 +90,13 @@ type ForecastReading struct { GridW, PVW, BatteryW, EVW, V2XW, HouseholdW float64 Valid, PVValid bool Reason, PVReason string + // Live Home use, separate from history and forecast qualification + // (Valid/Reason). Known whenever the site meter is read; flows that cannot + // be read are left out and LoadQuality says so. PVLiveKnown: every PV + // source is read, or dark. + LoadW float64 + LoadKnown, PVLiveKnown bool + LoadQuality, LoadReason string } // ForecastMeasurement reads power and health under one lock. A command fault @@ -128,6 +162,7 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast out.PVValid = false out.Reason = "no_telemetry" out.PVReason = out.Reason + out.LoadQuality, out.LoadReason = LoadUnknown, out.Reason return out } if opts.MaxAge <= 0 { @@ -136,9 +171,20 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast if opts.MaxSkew <= 0 { opts.MaxSkew = 30 * time.Second } + liveIssue := "" + note := func(reason string) { + if liveIssue == "" { + liveIssue = reason + } + } + pvLiveKnown := true + darkness := DarknessUnknown + if opts.PVDark != nil && !alignSources { + darkness = opts.PVDark(now) + } flows := map[string]ForecastFlow{} key := func(f ForecastFlow) string { return f.Driver + ":" + f.DerType.String() } - fail := func(reason string, pv bool) { + failQuiet := func(reason string, pv bool) { if out.Valid { out.Valid = false out.Reason = reason @@ -148,6 +194,10 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast out.PVReason = reason } } + fail := func(reason string, pv bool) { + note(reason) + failQuiet(reason, pv) + } if opts.PVInvalidReason != "" { out.PVValid = false out.PVReason = opts.PVInvalidReason @@ -165,6 +215,9 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast flows[key(f)] = f } } + // Kinds configured twice: live Home use leaves them out, never subtracts twice. + duplicate := map[DerType]bool{} + var duplicateW float64 ids := map[string]ForecastFlow{} for _, f := range opts.ExpectedFlows { if f.DerType == DerVehicle || (f.DerType == DerMeter && f.Driver != siteMeter) { @@ -174,6 +227,7 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast if old, ok := ids[f.FlowID]; ok { fail("duplicate_flow:"+f.FlowID, f.DerType == DerPV || old.DerType == DerPV) out.Known[f.DerType], out.Known[old.DerType] = false, false + duplicate[f.DerType], duplicate[old.DerType] = true, true } ids[f.FlowID] = f } @@ -185,9 +239,15 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast if _, ok := flows[f.Driver+":"+DerV2X.String()]; ok { fail("duplicate_charger:"+f.Driver, false) out.Known[DerEV], out.Known[DerV2X] = false, false + duplicate[DerEV], duplicate[DerV2X] = true, true } } } + // Configuration and duplicate flows leave PV unknown for live use too. + if !out.Known[DerPV] { + pvLiveKnown = false + } + meterIssue := "" keys := make([]string, 0, len(flows)) for k := range flows { keys = append(keys, k) @@ -246,10 +306,34 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast reason = "invalid_sign:" } if reason != "" { + switch { + case pv && darkness != DarknessUnknown && pvUnread(reason): + // Asleep after sunset: 0 W for live use. In twilight Home + // use is an estimate. History and forecasts keep their own + // evidence through Valid/Known. + if darkness == DarknessTwilight { + note("pv_twilight:" + k) + } + failQuiet(reason+k, pv) + out.Known[f.DerType] = false + zero := 0.0 + source.PowerW, source.Dark = &zero, true + out.Sources[k] = source + continue + case pv: + pvLiveKnown = false + case f.DerType == DerMeter: + meterIssue = reason + k + } fail(reason+k, pv) out.Known[f.DerType] = false continue } + if s.health[f.Driver].LastReadFailed() { + // Held from an earlier read until the age limit: shown, not + // measured. History and forecasts keep their own policy. + note("read_failed:" + k) + } source.PowerW = &powerW if r.SoC != nil && !math.IsNaN(*r.SoC) && !math.IsInf(*r.SoC, 0) && *r.SoC >= 0 && *r.SoC <= 1 { soc := *r.SoC @@ -270,6 +354,9 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast pvLast = powerAt } } + if duplicate[f.DerType] { + duplicateW += powerW + } switch f.DerType { case DerMeter: out.GridW = powerW @@ -306,6 +393,8 @@ func (s *Store) measurementLocked(now time.Time, siteMeter string, opts Forecast if math.IsNaN(out.HouseholdW) || math.IsInf(out.HouseholdW, 0) || out.HouseholdW < 0 { fail("invalid_balance", false) } + out.PVLiveKnown = pvLiveKnown && pvCount > 0 && !math.IsNaN(out.PVW) && !math.IsInf(out.PVW, 0) + out.setLiveLoad(out.HouseholdW+duplicateW, liveIssue, meterIssue) return out } @@ -409,3 +498,42 @@ func (r ForecastReading) Error() error { } return fmt.Errorf("forecast measurement: %s", r.Reason) } + +// setLiveLoad derives live Home use from the site meter and every flow that +// was read. A missing battery, EV, V2X or daytime PV is left out, as Core did +// before history qualification, and the result is an estimate. It is unknown +// only without a usable site meter. Quality is a hint and never hides it. +func (out *ForecastReading) setLiveLoad(w float64, issue, meterIssue string) { + if !out.Known[DerMeter] || meterIssue != "" { + if meterIssue == "" { + meterIssue = "site_meter_unknown" + } + out.LoadKnown, out.LoadQuality, out.LoadReason = false, LoadUnknown, meterIssue + return + } + if math.IsNaN(w) || math.IsInf(w, 0) { + out.LoadKnown, out.LoadQuality, out.LoadReason = false, LoadUnknown, "nonfinite_balance" + return + } + // Shown as is. A negative balance is a hint of production Core cannot + // see; zero or blank would hide it. + out.LoadKnown, out.LoadW = true, w + if w < 0 && issue == "" { + issue = "invalid_balance" + } + if issue == "" { + out.LoadQuality = LoadMeasured + return + } + out.LoadQuality, out.LoadReason = LoadEstimated, issue +} + +// pvUnread: the PV source gave no usable reading. Only then can darkness stand +// in for it. A malformed reading (wrong sign, non-finite, future) stays a fault. +func pvUnread(reason string) bool { + switch reason { + case "missing:", "offline:", "stale:", "unknown_power:": + return true + } + return false +} diff --git a/go/internal/telemetry/home_cloud_timing_test.go b/go/internal/telemetry/home_cloud_timing_test.go deleted file mode 100644 index c7a0f7aa5..000000000 --- a/go/internal/telemetry/home_cloud_timing_test.go +++ /dev/null @@ -1,78 +0,0 @@ -package telemetry - -import ( - "math" - "testing" - "time" -) - -func homeReceiptSample(s *Store, name string, kind DerType, watts float64, received time.Time) { - s.mu.Lock() - defer s.mu.Unlock() - s.recordHomeObservation(name, kind, watts, nil, received) - s.readings[key(name, kind)] = &DerReading{Driver: name, DerType: kind, RawW: watts, UpdatedAt: received} - s.health[name] = &DriverHealth{Name: name} -} - -func TestHomeEstimateCloudPollDoesNotAdvancePowerTime(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 120; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - power := 6000.0 - if sec >= 10 && sec <= 20 { - power += float64(sec-10) * 76.8 - } else if sec > 20 { - power = 6768 - } - homeReceiptSample(s, "site", DerMeter, 1000+power, at) - if sec == 10 { - homeSample(s, "cloud", DerEV, 6000, at, at, true) - } - if sec >= 20 { - homeSample(s, "cloud", DerEV, 6768, base.Add(20*time.Second), at, true) - } - } - got := s.HomeUseEstimate(base.Add(120*time.Second), "site", ForecastOptions{}) - if got.Watts == nil || got.ToMS != base.Add(20*time.Second).UnixMilli() || got.AgeMS != 105000 || got.TimingBasis != "mixed" { - t.Fatalf("fresh HTTP replies manufactured fresh power evidence: %+v", got) - } -} - -func TestHomeEstimateOneMinuteCloudDelayUsesPastWindow(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 150; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - ev := func(sec int) float64 { - if sec >= 30 && sec < 70 { - return 9714 - } - return 0 - } - homeReceiptSample(s, "site", DerMeter, 1000+ev(sec), at) - if sec >= 60 { - homeSample(s, "cloud", DerEV, ev(sec-60), at.Add(-time.Minute), at, true) - } - if sec < 70 { - continue - } - got := s.HomeUseEstimate(at, "site", ForecastOptions{}) - if got.Watts == nil || got.ToMS != at.Add(-time.Minute).UnixMilli() || got.AgeMS != 65000 || math.Abs(*got.Watts-1000) > 1e-8 || got.SourceAgeMS != 60000 { - t.Fatalf("second %d: cannot compare the reported past window: %+v", sec, got) - } - } -} - -func TestHomeEstimateRejectsChargerWithoutSourceTime(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 20; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 8000, at) - homeReceiptSample(s, "cloud", DerEV, 7000, at) - } - if got := s.HomeUseEstimate(base.Add(20*time.Second), "site", ForecastOptions{}); got.Watts != nil { - t.Fatalf("a charger receipt time was presented as measurement alignment: %+v", got) - } -} diff --git a/go/internal/telemetry/home_energy.go b/go/internal/telemetry/home_energy.go deleted file mode 100644 index 9bab53d3f..000000000 --- a/go/internal/telemetry/home_energy.go +++ /dev/null @@ -1,101 +0,0 @@ -package telemetry - -import ( - "encoding/json" - "time" -) - -type homeEnergyPoint struct { - At time.Time - Wh float64 -} - -type homeEnergySeries struct { - Session string - Points []homeEnergyPoint -} - -// Easee already supplies these three fields together. A counter without its -// observation time cannot distinguish a fresh read from a cached slow register. -// In particular, do not receipt-stamp local counters or mix battery DC energy -// with the AC power used in the site balance. Those flows use sampled power. -func (s *Store) recordHomeEnergy(driver string, kind DerType, data json.RawMessage, now time.Time) { - if kind != DerEV { - return - } - k := key(driver, kind) - var d struct { - Wh *float64 `json:"session_wh"` - At string `json:"energy_observed_at"` - Session string `json:"session_id"` - } - err := json.Unmarshal(data, &d) - series := s.homeEnergy[k] - if series == nil { - if d.At == "" { - return - } - if s.homeEnergy == nil { - s.homeEnergy = map[string]*homeEnergySeries{} - } - series = &homeEnergySeries{} - s.homeEnergy[k] = series - } - at, timeErr := time.Parse(time.RFC3339Nano, d.At) - if err != nil || timeErr != nil || at.After(now) || d.Wh == nil || !finite(*d.Wh) || *d.Wh < 0 || d.Session == "" { - series.Points = nil - return - } - if !s.health[driver].TelemetryLive() || series.Session != d.Session { - series.Points = nil - } - series.Session = d.Session - p := homeEnergyPoint{At: at, Wh: *d.Wh} - points := series.Points - if len(points) > 0 { - last := points[len(points)-1] - if at.Before(last.At) || (at.Equal(last.At) && p.Wh != last.Wh) { - series.Points = nil - return - } - if at.Equal(last.At) { - return // A poll of the same observation adds neither energy nor time. - } - if p.Wh < last.Wh || at.Sub(last.At) > homeMaxWindow { - points = nil // Reset, missed session boundary, or unsupported long gap. - } - } - points = append(points, p) - cutoff := now.Add(-homeLookback - homeMaxWindow) - for len(points) > 1 && points[1].At.Before(cutoff) { - points = points[1:] - } - if len(points) > 1024 { - points = points[len(points)-1024:] - } - series.Points = points -} - -// Counter differences give energy over their own interval. At an interior -// boundary, linear cumulative-energy interpolation is an estimate, not evidence -// of constant instantaneous power. Never extend beyond the reported endpoints. -func homeCounterEnergy(points []homeEnergyPoint, from, to time.Time) (float64, bool) { - energyAt := func(at time.Time) (float64, bool) { - for i, p := range points { - if p.At.Equal(at) { - return p.Wh, true - } - if i+1 < len(points) { - next := points[i+1] - if p.At.Before(at) && next.At.After(at) && next.At.Sub(p.At) <= homeMaxWindow { - f := at.Sub(p.At).Seconds() / next.At.Sub(p.At).Seconds() - return p.Wh + (next.Wh-p.Wh)*f, true - } - } - } - return 0, false - } - a, okA := energyAt(from) - b, okB := energyAt(to) - return b - a, okA && okB && to.After(from) && finite(b-a) && b >= a -} diff --git a/go/internal/telemetry/home_energy_test.go b/go/internal/telemetry/home_energy_test.go deleted file mode 100644 index c30bf556c..000000000 --- a/go/internal/telemetry/home_energy_test.go +++ /dev/null @@ -1,275 +0,0 @@ -package telemetry - -import ( - "encoding/json" - "math" - "testing" - "time" -) - -func homeSessionSample(s *Store, wh float64, session string, measured, received time.Time) { - data, _ := json.Marshal(map[string]any{ - "session_wh": wh, "session_id": session, "energy_observed_at": measured.Format(time.RFC3339Nano), - // Simulate a change-only power report that has not moved for an hour. - "control_power_observed_at": received.Add(-time.Hour).Format(time.RFC3339Nano), - "control_power_confirmed": true, - }) - homeDataSample(s, "cloud", DerEV, 9714, data, received) -} - -func homeDataSample(s *Store, driver string, kind DerType, w float64, data json.RawMessage, at time.Time) { - s.mu.Lock() - defer s.mu.Unlock() - s.recordHomeObservation(driver, kind, w, data, at) - s.readings[key(driver, kind)] = &DerReading{Driver: driver, DerType: kind, RawW: w, Data: data, UpdatedAt: at} - s.health[driver] = &DriverHealth{Name: driver} -} - -func TestHomeEstimatePrefersSessionEnergyWithMinuteDeliveryDelay(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 180; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - // A triangular charge ramp: exactly 50 Wh in each full minute. - phase := sec % 60 - ev := float64(phase) * 200 - if phase > 30 { - ev = float64(60-phase) * 200 - } - homeReceiptSample(s, "site", DerMeter, 1000+ev, at) - if sec >= 60 && sec%60 == 0 { - homeSessionSample(s, float64(sec/60-1)*50, "session-a", at.Add(-time.Minute), at) - } - if sec < 120 { - continue - } - got := s.HomeUseEstimate(at, "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.EnergyWh == nil || math.Abs(*got.EnergyWh-1000.0/60) > 1e-8 || got.EnergyBasis != "mixed" { - t.Fatalf("second %d: energy balance failed: %+v", sec, got) - } - if got.ToMS-got.FromMS != 60000 || got.EndAgeMS < 60000 || got.MinW != nil || got.MaxW != nil { - t.Fatalf("counter energy presented as instantaneous power/range: %+v", got) - } - } - // Display energy must not repair the bad raw power or change control evidence. - raw := s.ForecastMeasurement(base.Add(180*time.Second), "site", ForecastOptions{}) - if raw.Valid || raw.Reason != "invalid_balance" || raw.HouseholdW != -8714 { - t.Fatal(raw) - } -} - -func TestHomeEstimateAdaptsToCounterReportsAndDoesNotRefreshOnPoll(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 250; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - stamp := 0 - switch { - case sec >= 180: - stamp = 180 - case sec >= 60: - stamp = 60 - } - homeSessionSample(s, float64(stamp)*6000/3600, "session-a", base.Add(time.Duration(stamp)*time.Second), at) - if sec != 60 && sec != 180 && sec != 250 { - continue - } - got := s.HomeUseEstimate(at, "site", ForecastOptions{}) - period := int64(60000) - if sec >= 180 { - period = 120000 - } - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.ToMS-got.FromMS != period || got.EndAgeMS != int64(sec-stamp)*1000 { - t.Fatal(sec, got) - } - } -} - -func TestHomeSessionCounterBreaksAtInvalidEvidence(t *testing.T) { - base := time.Unix(100000, 0) - for _, scenario := range []string{"session", "reset", "missing_session", "missing_time", "missing_counter", "backward", "conflict", "future", "long_gap", "offline"} { - t.Run(scenario, func(t *testing.T) { - s := NewStore() - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - if sec%60 == 0 { - homeSessionSample(s, float64(sec)*6000/3600, "session-a", at, at) - } - } - now := base.Add(61 * time.Second) - d := map[string]any{"session_wh": 101, "session_id": "session-a", "energy_observed_at": now.Format(time.RFC3339Nano)} - switch scenario { - case "session": - d["session_id"] = "session-b" - case "reset": - d["session_wh"] = 0 - case "missing_session": - delete(d, "session_id") - case "missing_time": - delete(d, "energy_observed_at") - case "missing_counter": - delete(d, "session_wh") - case "backward": - d["energy_observed_at"] = base.Add(59 * time.Second).Format(time.RFC3339Nano) - case "conflict": - d["energy_observed_at"] = base.Add(60 * time.Second).Format(time.RFC3339Nano) - case "future": - d["energy_observed_at"] = now.Add(time.Second).Format(time.RFC3339Nano) - case "long_gap": - now = base.Add(361 * time.Second) - d["energy_observed_at"] = now.Format(time.RFC3339Nano) - case "offline": - s.health["cloud"].SetOffline() - } - data, _ := json.Marshal(d) - homeDataSample(s, "cloud", DerEV, 6000, data, now) - homeReceiptSample(s, "site", DerMeter, 7000, now) - if got := s.HomeUseEstimate(now, "site", ForecastOptions{}); got.Watts != nil { - t.Fatalf("crossed %s: %+v", scenario, got) - } - }) - } -} - -func TestHomeEnergyCannotFillLocalPowerGap(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 120; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - if sec < 30 || sec > 90 { - homeReceiptSample(s, "site", DerMeter, 7000, at) - } - if sec%60 == 0 { - homeSessionSample(s, float64(sec)*6000/3600, "session-a", at, at) - } - } - if got := s.HomeUseEstimate(base.Add(120*time.Second), "site", ForecastOptions{}); got.Watts != nil { - t.Fatal("counter filled missing meter energy", got) - } -} - -func TestHomePowerPeriodFollowsDistinctSourceReports(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - if sec%20 == 0 { - homeSample(s, "cloud", DerEV, 6000, at, at, false) - } - } - got := s.HomeUseEstimate(base.Add(time.Minute), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.ToMS-got.FromMS != 20000 || got.EnergyBasis != "power_integral" { - t.Fatal(got) - } -} - -func TestHomeEnergyKeepsWholeReferenceCounterInterval(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 90; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - if sec == 0 || sec == 60 || sec == 90 { - homeSessionSample(s, float64(sec)*6000/3600, "session-a", at, at) - } - } - got := s.HomeUseEstimate(base.Add(90*time.Second), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.FromMS != base.UnixMilli() || got.ToMS-got.FromMS != 90000 { - t.Fatal("split the reference counter interval after a cadence change", got) - } -} - -func TestHomeEnergyUsesCountersWithoutPowerTimeAndDoesNotMixDCEnergy(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 5000, at) - // Pixii: raw DC and cabinet counters do not describe the AC site flow. - homeDataSample(s, "battery", DerBattery, -2500, json.RawMessage(`{"control_power_w":-2000,"charge_wh":0,"discharge_wh":9000}`), at) - if sec%60 == 0 { - data, _ := json.Marshal(map[string]any{"session_wh": float64(sec) * 6000 / 3600, "session_id": "session-a", "energy_observed_at": at.Format(time.RFC3339Nano)}) - homeDataSample(s, "cloud", DerEV, 12345, data, at) - } - } - got := s.HomeUseEstimate(base.Add(time.Minute), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 { - t.Fatal("counter path required power time or used DC energy", got) - } -} - -func TestHomeEnergyBalancesTwoCounterSchedules(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 135; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 11000, at) - for name, interval := range map[string]int{"charger-a": 60, "charger-b": 45} { - if sec%interval != 0 { - continue - } - w := 6000.0 - if name == "charger-b" { - w = 4000 - } - data, _ := json.Marshal(map[string]any{"session_wh": float64(sec) * w / 3600, "session_id": name, "energy_observed_at": at.Format(time.RFC3339Nano)}) - homeDataSample(s, name, DerEV, w, data, at) - } - } - for i := 0; i < 20; i++ { - got := s.HomeUseEstimate(base.Add(135*time.Second), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.ToMS != base.Add(120*time.Second).UnixMilli() || got.FromMS != base.Add(time.Minute).UnixMilli() { - t.Fatal("counter schedules did not share a stable period", got) - } - } -} - -func TestHomeEstimateDropsSessionEnergyWhenDriverIsReplaced(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - if sec%60 == 0 { - homeSessionSample(s, float64(sec)*6000/3600, "session-a", at, at) - } - } - s.Remove("cloud") - // The replacement has source power but does not expose session energy. - for sec := 61; sec <= 81; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - homeSample(s, "cloud", DerEV, 6000, at, at, false) - } - got := s.HomeUseEstimate(base.Add(81*time.Second), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.EnergyBasis != "power_integral" { - t.Fatal("replacement inherited the old counter requirement", got) - } -} - -func TestHomeEnergyRepeatedPollsCannotKeepAnOldIntervalFresh(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 241; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 7000, at) - stamp := 0 - if sec >= 60 { - stamp = 60 - } - homeSessionSample(s, float64(stamp)*6000/3600, "session-a", base.Add(time.Duration(stamp)*time.Second), at) - if sec < 240 { - continue - } - got := s.HomeUseEstimate(at, "site", ForecastOptions{}) - if sec == 240 && (got.Watts == nil || got.EndAgeMS != 180000) { - t.Fatal("lost supported past interval", got) - } - if sec == 241 && got.Watts != nil { - t.Fatal("fresh HTTP reply extended energy age", got) - } - } -} diff --git a/go/internal/telemetry/home_estimate.go b/go/internal/telemetry/home_estimate.go deleted file mode 100644 index 34eeb3315..000000000 --- a/go/internal/telemetry/home_estimate.go +++ /dev/null @@ -1,312 +0,0 @@ -package telemetry - -import ( - "encoding/json" - "math" - "strings" - "time" -) - -const homeWindow = 10 * time.Second -const homeMaxGap = 30 * time.Second -const homeLookback = 3 * time.Minute -const homeMaxWindow = 5 * time.Minute - -// HomeEstimate describes a past interval, never a new instantaneous measurement. -// MinW/MaxW describe variation in the reported window, not error bounds. -// Reported source clocks and local receipt clocks are not verified as aligned. -type HomeEstimate struct { - Watts *float64 `json:"watts"` - EnergyWh *float64 `json:"energy_wh"` - EnergyBasis string `json:"energy_basis"` - MinW *float64 `json:"window_min_w"` - MaxW *float64 `json:"window_max_w"` - FromMS int64 `json:"from_ms"` - ToMS int64 `json:"to_ms"` - AgeMS int64 `json:"age_ms"` - EndAgeMS int64 `json:"end_age_ms"` - Method string `json:"method"` - TimingBasis string `json:"timing_basis"` - SourceAgeMS int64 `json:"source_age_ms"` - Reason string `json:"reason,omitempty"` -} - -// Keep a separate bounded buffer: neither control evidence nor raw readings may -// depend on how often a client asks for an estimate. Caller holds s.mu. -func (s *Store) recordHomeObservation(driver string, kind DerType, w float64, data json.RawMessage, now time.Time) { - if kind == DerVehicle { - return - } - s.recordHomeEnergy(driver, kind, data, now) - k := key(driver, kind) - if s.homeObservations == nil { - s.homeObservations = map[string][]ControlObservation{} - s.homeSourceTime = map[string]bool{} - s.homeIdleState = map[string]time.Time{} - } - p, ok := ControlPowerObservation(w, data, now) - var timing struct { - ControlPowerObservedAt string `json:"control_power_observed_at"` - PowerObservedAt string `json:"power_observed_at"` - ForecastPower *ForecastPowerSample `json:"forecast_power"` - } - _ = json.Unmarshal(data, &timing) - sourceTime := timing.ControlPowerObservedAt != "" || timing.PowerObservedAt != "" || timing.ForecastPower != nil - idleState := time.Time{} - if ok && kind == DerEV && w == 0 && p.PowerW == 0 { - idleState = homeDisconnectedState(data, now) - } - if !ok || ((kind == DerEV || kind == DerV2X) && !sourceTime && idleState.IsZero()) { - delete(s.homeObservations, k) - delete(s.homeSourceTime, k) - delete(s.homeIdleState, k) - return - } - if !idleState.IsZero() { - // A confirmed disconnected poll supports a zero-EV-load estimate at - // receipt time. It does not refresh the old power/session observation. - // Bracket only these polls, and start over after any state transition. - p.At, sourceTime = now, false - delete(s.homeEnergy, k) - } - // A successful cloud request can return an old observation. The driver's - // control confirmation is not a new power timestamp and cannot extend it. - p.ConfirmedAt = time.Time{} - points := s.homeObservations[k] - if !s.health[driver].TelemetryLive() || s.homeSourceTime[k] != sourceTime || !s.homeIdleState[k].Equal(idleState) { - points = nil - } - s.homeSourceTime[k] = sourceTime - s.homeIdleState[k] = idleState - if len(points) > 0 { - last := &points[len(points)-1] - if p.At.Before(last.At) || p.At.Sub(last.At) > homeMaxGap { - points = nil - } else if p.At.Equal(last.At) { - if p.PowerW != last.PowerW { - delete(s.homeObservations, k) - return - } - // A repeated change-only timestamp adds no time, even when a - // fresh HTTP reply carries a control confirmation. - s.homeObservations[k] = points - return - } - } - points = append(points, p) - cutoff := now.Add(-homeLookback - homeMaxWindow) - for len(points) > 1 && points[1].At.Before(cutoff) { - points = points[1:] - } - if len(points) > 1024 { - points = points[len(points)-1024:] - } - s.homeObservations[k] = points -} - -func homeDisconnectedState(data json.RawMessage, now time.Time) time.Time { - var d struct { - Connected *bool `json:"connected"` - Charging *bool `json:"charging"` - ControlPowerAvailable bool `json:"control_power_available"` - ControlPowerConfirmed bool `json:"control_power_confirmed"` - StateObservedAt string `json:"state_observed_at"` - SessionID string `json:"session_id"` - } - if json.Unmarshal(data, &d) != nil || d.Connected == nil || *d.Connected || d.Charging == nil || *d.Charging || !d.ControlPowerAvailable || !d.ControlPowerConfirmed || d.SessionID != "" { - return time.Time{} - } - at, err := time.Parse(time.RFC3339Nano, d.StateObservedAt) - if err != nil || at.After(now) { - return time.Time{} - } - return at -} - -// HomeUseEstimate divides signed energy by a common reported interval. Prefer -// timestamped session counters; otherwise integrate bracketed power. The period -// is at least ten seconds and at least the largest of each source's last three -// distinct reporting intervals. This is observed coverage, not a cadence promise. -// Clocks and sampling delay remain unverified. This path serves display only. -func (s *Store) HomeUseEstimate(now time.Time, siteMeter string, opts ForecastOptions) HomeEstimate { - out := HomeEstimate{Method: "reported_energy_balance_v3"} - if s == nil { - out.Reason = "no_telemetry" - return out - } - s.mu.RLock() - defer s.mu.RUnlock() - r := s.forecastMeasurementLocked(now, siteMeter, opts) - if !r.Valid && r.Reason != "invalid_balance" && r.Reason != "time_skew" && !strings.HasPrefix(r.Reason, "stale:") && !strings.HasPrefix(r.Reason, "unknown_power:") { - out.Reason = r.Reason - return out - } - to := now - window := homeWindow - counterCount := 0 - var reference *homeEnergySeries - var referenceKey string - var referenceGap time.Duration - var sourceClocks, receiptClocks bool - for k, source := range r.Sources { - points := s.homeObservations[k] - counter := s.homeEnergy[k] - if counter == nil && (source.Flow.DerType == DerEV || source.Flow.DerType == DerV2X) && !s.homeSourceTime[k] && s.homeIdleState[k].IsZero() { - out.Reason = "charger_source_time_unavailable" - return out - } - if !s.health[source.Flow.Driver].TelemetryLive() { - out.Reason = "missing_source_window" - return out - } - var times []time.Time - usesSourceTime := s.homeSourceTime[k] - if counter != nil { - for _, p := range counter.Points { - times = append(times, p.At) - } - usesSourceTime = true - counterCount++ - } else { - for _, p := range points { - times = append(times, p.At) - } - } - if len(times) < 2 { - out.Reason = "incomplete_source_window" - return out - } - last := times[len(times)-1] - if usesSourceTime && now.Sub(last).Milliseconds() > out.SourceAgeMS { - out.SourceAgeMS = now.Sub(last).Milliseconds() - } - if last.After(now) || now.Sub(last) > homeLookback { - out.Reason = "stale_source_window" - return out - } - if last.Before(to) { - to = last - } - var reportGap time.Duration - for i := len(times) - 1; i > 0 && i >= len(times)-3; i-- { - if gap := times[i].Sub(times[i-1]); gap > reportGap { - reportGap = gap - } - } - if reportGap > window { - window = reportGap - } - if counter != nil && (reference == nil || reportGap > referenceGap || (reportGap == referenceGap && k < referenceKey)) { - reference, referenceKey, referenceGap = counter, k, reportGap - } - sourceClocks = sourceClocks || usesSourceTime - receiptClocks = receiptClocks || !usesSourceTime - } - out.TimingBasis = "source" - if receiptClocks { - out.TimingBasis = "receipt" - if sourceClocks { - out.TimingBasis = "mixed" - } - } - from := to.Add(-window) - if reference != nil { - // Keep the slowest counter's complete interval wherever possible. - // Other counter clocks may require cumulative-energy interpolation. - endFound, startFound := false, false - for i := len(reference.Points) - 1; i >= 0; i-- { - at := reference.Points[i].At - if !endFound && !at.After(to) { - to, endFound = at, true - from = to.Add(-window) - } - if endFound && !at.After(from) { - from, startFound = at, true - break - } - } - if !startFound || to.Sub(from) > homeMaxWindow || now.Sub(to) > homeLookback { - out.Reason = "incomplete_source_window" - return out - } - window = to.Sub(from) - } - out.FromMS, out.ToMS = from.UnixMilli(), to.UnixMilli() - out.AgeMS = now.Sub(from.Add(window / 2)).Milliseconds() - out.EndAgeMS = now.Sub(to).Milliseconds() - out.EnergyBasis = "power_integral" - if counterCount > 0 { - out.EnergyBasis = "counters" - if counterCount < len(r.Sources) { - out.EnergyBasis = "mixed" - } - } - var energy, low, high float64 - for k, source := range r.Sources { - var wh, minW, maxW float64 - var ok bool - if counter := s.homeEnergy[k]; counter != nil { - wh, ok = homeCounterEnergy(counter.Points, from, to) - } else { - var avg float64 - avg, minW, maxW, ok = homeWindowPower(s.homeObservations[k], from, to) - wh = avg * window.Hours() - } - if !ok { - out.Reason = "incomplete_source_window" - return out - } - if source.Flow.DerType == DerMeter { - energy += wh - low += minW - high += maxW - } else { - energy -= wh - low -= maxW - high -= minW - } - } - mean := energy / window.Hours() - if !finite(mean) || !finite(low) || !finite(high) || mean < 0 { - out.Reason = "invalid_balance" - return out - } - out.Watts, out.EnergyWh = &mean, &energy - if counterCount == 0 { - out.MinW, out.MaxW = &low, &high - } - return out -} - -func homeWindowPower(points []ControlObservation, from, to time.Time) (mean, low, high float64, ok bool) { - covered := time.Duration(0) - low, high = math.Inf(1), math.Inf(-1) - add := func(a, b time.Time, wa, wb float64) { - left, right := a, b - if left.Before(from) { - left = from - } - if right.After(to) { - right = to - } - if !right.After(left) { - return - } - span := b.Sub(a).Seconds() - v0 := wa + (wb-wa)*left.Sub(a).Seconds()/span - v1 := wa + (wb-wa)*right.Sub(a).Seconds()/span - mean += (v0 + v1) / 2 * right.Sub(left).Seconds() - low = math.Min(low, math.Min(v0, v1)) - high = math.Max(high, math.Max(v0, v1)) - covered += right.Sub(left) - } - for i, p := range points { - if i+1 < len(points) { - next := points[i+1] - if next.At.After(p.At) && next.At.Sub(p.At) <= homeMaxGap { - add(p.At, next.At, p.PowerW, next.PowerW) - } - } - } - return mean / to.Sub(from).Seconds(), low, high, covered == to.Sub(from) -} diff --git a/go/internal/telemetry/home_estimate_test.go b/go/internal/telemetry/home_estimate_test.go deleted file mode 100644 index 423713e02..000000000 --- a/go/internal/telemetry/home_estimate_test.go +++ /dev/null @@ -1,201 +0,0 @@ -package telemetry - -import ( - "encoding/json" - "fmt" - "math" - "testing" - "time" -) - -func homeSample(s *Store, name string, kind DerType, w float64, measured, received time.Time, confirmed bool) { - data, _ := json.Marshal(map[string]any{"power_observed_at": measured.Format(time.RFC3339Nano), "control_power_confirmed": confirmed}) - s.mu.Lock() - defer s.mu.Unlock() - s.recordHomeObservation(name, kind, w, data, received) - s.readings[key(name, kind)] = &DerReading{Driver: name, DerType: kind, RawW: w, UpdatedAt: received, Data: data} - s.health[name] = &DriverHealth{Name: name} -} - -// This is a synthetic, timestamped replay of the reported failure class, not -// Göran's missing raw 19:32 trace. A four-second delivery delay is a test input. -func TestHomeEstimateAlignedReplayVersusIndependentKalman(t *testing.T) { - s := NewStore() - base := time.Date(2026, 10, 17, 17, 32, 0, 0, time.UTC) - gridFilter, evFilter := NewKalman(100, 50), NewKalman(100, 50) - var alignedError, filteredError float64 - var count int - for second := 0; second <= 100; second++ { - at := base.Add(time.Duration(second) * time.Second) - ev := func(sec int) float64 { - if sec >= 10 && sec < 40 || sec >= 60 && sec < 75 { - return 9714 - } - return 0 - } - homeSample(s, "site", DerMeter, 1000+ev(second), at, at, false) - grid := gridFilter.Update(1000 + ev(second)) - if second >= 4 { - measured := at.Add(-4 * time.Second) - homeSample(s, "ev", DerEV, ev(second-4), measured, at, false) - evFilter.Update(ev(second - 4)) - } - if second < 14 { - continue - } - out := s.HomeUseEstimate(at, "site", ForecastOptions{}) - if out.Watts == nil { - t.Fatalf("second %d: %+v", second, out) - } - alignedError += math.Abs(*out.Watts - 1000) - filteredError += math.Abs(grid - evFilter.Estimate - 1000) - if out.ToMS != at.Add(-4*time.Second).UnixMilli() || out.AgeMS != 9000 { - t.Fatal(out) - } - count++ - } - if alignedError > 1e-8 || filteredError/float64(count) < 500 { - t.Fatal(alignedError, filteredError, count) - } - t.Logf("synthetic replay MAE: aligned %.3f W, independent scalar Kalman %.1f W", alignedError/float64(count), filteredError/float64(count)) -} - -func TestHomeWindowTimeWeightingAndSignedEnergy(t *testing.T) { - base := time.Unix(1000, 0) - for _, offsets := range [][]int{{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, {0, 2, 9, 10}} { - var p []ControlObservation - for _, sec := range offsets { - p = append(p, ControlObservation{PowerW: float64(sec) * 100, At: base.Add(time.Duration(sec) * time.Second)}) - } - mean, low, high, ok := homeWindowPower(p, base, base.Add(10*time.Second)) - if !ok || math.Abs(mean-500) > 1e-9 || low != 0 || high != 1000 { - t.Fatal(mean, low, high, ok) - } - } - // Integrating then subtracting must equal subtracting the signed energies. - p := []ControlObservation{{PowerW: -1000, At: base}, {PowerW: 3000, At: base.Add(10 * time.Second)}} - mean, _, _, ok := homeWindowPower(p, base, base.Add(10*time.Second)) - if !ok || mean != 1000 { - t.Fatal(mean, ok) - } -} - -func TestHomeEstimateChangeOnlyConfirmationAndCachedRepeats(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - original := base.Add(-time.Hour) - for sec := 0; sec <= 20; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeSample(s, "site", DerMeter, 1500, at, at, false) - homeSample(s, "ev", DerEV, 500, original, at, true) - } - got := s.HomeUseEstimate(base.Add(20*time.Second), "site", ForecastOptions{}) - if got.Watts != nil || len(s.homeObservations["ev:ev"]) != 1 { - t.Fatal(got) - } - // Asking repeatedly must not advance the interval or modify its source. - for i := 0; i < 100; i++ { - again := s.HomeUseEstimate(base.Add(25*time.Second), "site", ForecastOptions{}) - if again.Watts != nil || again.ToMS != got.ToMS { - t.Fatal(again) - } - } - if got := s.HomeUseEstimate(base.Add(51*time.Second), "site", ForecastOptions{}); got.Watts != nil { - t.Fatal(got) - } - // An unconfirmed duplicate cannot extend a plateau. - u := NewStore() - for sec := 0; sec <= 20; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeSample(u, "site", DerMeter, 1000, base, at, false) - } - if got := u.HomeUseEstimate(base.Add(20*time.Second), "site", ForecastOptions{}); got.Watts != nil { - t.Fatal(got) - } -} - -func TestHomeEstimateRejectsGapsAndConflicts(t *testing.T) { - base := time.Unix(100000, 0) - for _, scenario := range []string{"offline", "long_gap", "backward", "conflict", "future", "unknown", "restart", "duplicate_topology"} { - t.Run(scenario, func(t *testing.T) { - s := NewStore() - for sec := 0; sec <= 10; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeSample(s, "site", DerMeter, 1000, at, at, false) - } - now := base.Add(10 * time.Second) - opts := ForecastOptions{} - switch scenario { - case "offline": - s.health["site"].SetOffline() - case "long_gap": - now = base.Add(50 * time.Second) - homeSample(s, "site", DerMeter, 1000, now, now, false) - case "backward": - homeSample(s, "site", DerMeter, 1000, base.Add(5*time.Second), now, false) - case "conflict": - homeSample(s, "site", DerMeter, 2000, now, now, false) - case "future": - homeSample(s, "site", DerMeter, 1000, now.Add(time.Second), now, false) - case "unknown": - s.recordHomeObservation("site", DerMeter, 1000, json.RawMessage(`{"control_power_available":false}`), now) - case "restart": - s = NewStore() - homeSample(s, "site", DerMeter, 1000, now, now, false) - case "duplicate_topology": - opts.ExpectedFlows = []ForecastFlow{{Driver: "site", DerType: DerMeter, FlowID: "x"}, {Driver: "site", DerType: DerMeter, FlowID: "x"}} - } - if got := s.HomeUseEstimate(now, "site", opts); got.Watts != nil { - t.Fatal(got) - } - }) - } -} - -func TestHomeEstimateDoesNotRepairRawNegativeBalance(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 20; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - ev := 9714.0 - grid := 10600.0 - bat := 0.0 - if sec >= 20 { - grid = -267 - bat = -6035 - } - homeSample(s, "site", DerMeter, grid, at, at, false) - homeSample(s, "bat", DerBattery, bat, at, at, false) - // The slow source has not reported the stop yet. - if sec <= 16 { - homeSample(s, "ev", DerEV, ev, at, at, false) - } - } - now := base.Add(20 * time.Second) - for i := 0; i < 5; i++ { - raw := s.ForecastMeasurement(now, "site", ForecastOptions{}) - if raw.Valid || raw.Reason != "invalid_balance" || raw.HouseholdW != -3946 { - t.Fatal(raw) - } - got := s.HomeUseEstimate(now, "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-886) > 1e-8 || got.AgeMS != 9000 { - t.Fatal(fmt.Sprintf("%+v", got)) - } - } -} - -func TestHomeWindowCannotExtendThroughControlConfirmations(t *testing.T) { - base := time.Unix(100000, 0) - points := []ControlObservation{ - {At: base, ConfirmedAt: base.Add(20 * time.Second), PowerW: 1000}, - {At: base.Add(19 * time.Second), ConfirmedAt: base.Add(40 * time.Second), PowerW: 2000}, - } - // Control confirmations are HTTP/status evidence, not source brackets. - if _, _, _, ok := homeWindowPower(points, base.Add(15*time.Second), base.Add(25*time.Second)); ok { - t.Fatal("overlap treated as certain") - } - mean, _, _, ok := homeWindowPower(points, base.Add(30*time.Second), base.Add(40*time.Second)) - if ok { - t.Fatal("confirmation extended source evidence", mean, ok) - } -} diff --git a/go/internal/telemetry/home_idle_test.go b/go/internal/telemetry/home_idle_test.go deleted file mode 100644 index 2cf6128dc..000000000 --- a/go/internal/telemetry/home_idle_test.go +++ /dev/null @@ -1,227 +0,0 @@ -package telemetry - -import ( - "encoding/json" - "math" - "testing" - "time" -) - -func homeDisconnectedData(base time.Time) map[string]any { - return map[string]any{ - "connected": false, "charging": false, - "control_power_observed_at": base.Add(-time.Hour).Format(time.RFC3339Nano), - "control_power_available": true, "control_power_confirmed": true, - "state_observed_at": base.Add(-time.Hour).Format(time.RFC3339Nano), - } -} - -func homeIdleSample(t *testing.T, s *Store, w float64, fields map[string]any, at time.Time) { - t.Helper() - data, err := json.Marshal(fields) - if err != nil { - t.Fatal("cannot encode charger evidence") - } - homeDataSample(s, "charger", DerEV, w, data, at) -} - -// Easee can retain the last session's energy and change-only power timestamps -// for hours after unplugging. Confirmed disconnected polls describe zero EV -// load, not fresh measurements of that old session. -func TestHomeEstimateDisconnectedChargerWithOldObservations(t *testing.T) { - for _, oldEnergy := range []bool{false, true} { - t.Run(map[bool]string{false: "power_only", true: "previous_session_energy"}[oldEnergy], func(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - fields := homeDisconnectedData(base) - if oldEnergy { - fields["session_wh"] = 5000 - fields["energy_observed_at"] = base.Add(-time.Hour).Format(time.RFC3339Nano) - } - for sec := 0; sec <= 600; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, -1500, at) - homeReceiptSample(s, "pv", DerPV, -3000, at) - homeReceiptSample(s, "battery", DerBattery, 500, at) - if sec%15 == 0 { - homeIdleSample(t, s, 0, fields, at) - } - } - now := base.Add(600 * time.Second) - got := s.HomeUseEstimate(now, "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.Reason != "" { - t.Fatalf("disconnected charger hid household load: %+v", got) - } - if got.TimingBasis != "receipt" || got.ToMS != now.UnixMilli() || got.ToMS-got.FromMS != 15000 || got.EnergyBasis != "power_integral" { - t.Fatalf("wrong interval or clock basis: %+v", got) - } - // Display estimates must leave raw power and control evidence alone. - reading := s.readings[key("charger", DerEV)] - p, ok := ControlPowerObservation(reading.RawW, reading.Data, now) - if !ok || !p.At.Equal(base.Add(-time.Hour)) || !p.ConfirmedAt.Equal(now) || reading.RawW != 0 { - t.Fatal("display estimate changed source/control evidence") - } - // Asking again cannot extend the interval without another poll. - again := s.HomeUseEstimate(now.Add(time.Second), "site", ForecastOptions{}) - if again.ToMS != got.ToMS || again.EndAgeMS != 1000 { - t.Fatal("client reads extended disconnected coverage") - } - if stale := s.HomeUseEstimate(now.Add(homeLookback+time.Second), "site", ForecastOptions{}); stale.Watts != nil { - t.Fatal("old disconnected polls remained usable") - } - }) - } -} - -func TestHomeEstimateDoesNotInferDisconnectedZeroFromOtherEvidence(t *testing.T) { - for _, scenario := range []string{"connected", "charging", "missing_connected", "missing_charging", "unconfirmed", "unavailable", "missing_availability", "nonzero", "control_nonzero", "session", "missing_state_time", "bad_state_time", "future_state_time", "future_power_time", "offline"} { - t.Run(scenario, func(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - fields := homeDisconnectedData(base) - w := 0.0 - switch scenario { - case "connected": - fields["connected"] = true - case "charging": - fields["charging"] = true - case "missing_connected": - delete(fields, "connected") - case "missing_charging": - delete(fields, "charging") - case "unconfirmed": - fields["control_power_confirmed"] = false - case "unavailable": - fields["control_power_available"] = false - case "missing_availability": - delete(fields, "control_power_available") - case "nonzero": - w = 500 - case "control_nonzero": - fields["control_power_w"] = 500 - case "session": - fields["session_id"] = "active-session" - case "missing_state_time": - delete(fields, "state_observed_at") - case "bad_state_time": - fields["state_observed_at"] = "invalid" - case "future_state_time": - fields["state_observed_at"] = base.Add(time.Hour).Format(time.RFC3339Nano) - case "future_power_time": - fields["control_power_observed_at"] = base.Add(time.Hour).Format(time.RFC3339Nano) - } - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 1000+w, at) - if sec%15 == 0 { - homeIdleSample(t, s, w, fields, at) - } - } - if scenario == "offline" { - s.health["charger"].SetOffline() - } - if got := s.HomeUseEstimate(base.Add(time.Minute), "site", ForecastOptions{}); got.Watts != nil { - t.Fatalf("%s became a zero-load interval: %+v", scenario, got) - } - }) - } -} - -func TestHomeDisconnectedCoverageBreaksAtTransitions(t *testing.T) { - for _, scenario := range []string{"reconnect", "unknown", "state_change", "offline", "gap", "remove"} { - t.Run(scenario, func(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - fields := homeDisconnectedData(base) - for sec := 0; sec <= 60; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 1000, at) - if sec%15 == 0 { - homeIdleSample(t, s, 0, fields, at) - } - } - if got := s.HomeUseEstimate(base.Add(time.Minute), "site", ForecastOptions{}); got.Watts == nil { - t.Fatal("missing initial disconnected interval") - } - now := base.Add(61 * time.Second) - switch scenario { - case "reconnect": - fields["connected"] = true - case "unknown": - fields["control_power_available"] = false - case "state_change": - fields["state_observed_at"] = now.Format(time.RFC3339Nano) - case "offline": - s.health["charger"].SetOffline() - case "gap": - now = base.Add(91 * time.Second) - case "remove": - s.Remove("charger") - } - homeReceiptSample(s, "site", DerMeter, 1000, now) - homeIdleSample(t, s, 0, fields, now) - if got := s.HomeUseEstimate(now, "site", ForecastOptions{}); got.Watts != nil { - t.Fatalf("crossed %s: %+v", scenario, got) - } - // A newly bracketed disconnected interval may become usable again. - for sec := 1; sec <= 30; sec++ { - at := now.Add(time.Duration(sec) * time.Second) - homeReceiptSample(s, "site", DerMeter, 1000, at) - if sec%15 == 0 { - homeIdleSample(t, s, 0, homeDisconnectedData(base), at) - } - } - got := s.HomeUseEstimate(now.Add(30*time.Second), "site", ForecastOptions{}) - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 || got.FromMS < now.UnixMilli() { - t.Fatalf("new interval did not recover after %s: %+v", scenario, got) - } - }) - } -} - -func TestHomeEstimateKeepsChargingCountersSeparateFromDisconnectedCoverage(t *testing.T) { - s := NewStore() - base := time.Unix(100000, 0) - for sec := 0; sec <= 180; sec++ { - at := base.Add(time.Duration(sec) * time.Second) - grid := 7000.0 - if sec > 60 && sec <= 90 { - grid = 1000 - } - homeReceiptSample(s, "site", DerMeter, grid, at) - if sec%15 != 0 { - continue - } - fields := homeDisconnectedData(base) - w := 6000.0 - fields["session_wh"] = float64(sec) * w / 3600 - fields["energy_observed_at"] = at.Format(time.RFC3339Nano) - switch { - case sec <= 60: - fields["connected"], fields["charging"], fields["session_id"] = true, true, "first-session" - case sec <= 90: - w = 0 - fields["state_observed_at"] = base.Add(61 * time.Second).Format(time.RFC3339Nano) - fields["energy_observed_at"] = base.Add(time.Minute).Format(time.RFC3339Nano) - fields["session_wh"] = 100 - default: - fields["connected"], fields["charging"], fields["session_id"] = true, true, "second-session" - fields["session_wh"] = float64(sec-90) * w / 3600 - } - homeIdleSample(t, s, w, fields, at) - got := s.HomeUseEstimate(at, "site", ForecastOptions{}) - switch sec { - case 60, 90, 180: - if got.Watts == nil || math.Abs(*got.Watts-1000) > 1e-8 { - t.Fatalf("lost a supported interval at %d: %+v", sec, got) - } - if sec != 90 && got.EnergyBasis != "mixed" { - t.Fatal("charging no longer uses session energy") - } - case 75, 105: - if got.Watts != nil { - t.Fatal("estimate crossed a plug/session transition") - } - } - } -} diff --git a/go/internal/telemetry/live_load_quality_test.go b/go/internal/telemetry/live_load_quality_test.go new file mode 100644 index 000000000..cf51f85c1 --- /dev/null +++ b/go/internal/telemetry/live_load_quality_test.go @@ -0,0 +1,248 @@ +package telemetry + +import ( + "testing" + "time" +) + +// Anders' site: Pixii reports the site meter and its battery; a Growatt +// inverter drops off the bus after sunset. +func andersSite(darkness Darkness) (*Store, ForecastOptions) { + s := NewStore() + for _, d := range []string{"pixii", "growatt"} { + s.EnsureDriverHealth(d) + } + s.Update("pixii", DerMeter, 780, nil, nil) + s.Update("pixii", DerBattery, -200, nil, nil) + s.RecordDriverSuccess("pixii") + s.Update("growatt", DerPV, -1500, nil, nil) + s.DriverHealthMut("growatt").SetOffline() + return s, ForecastOptions{PVDark: func(time.Time) Darkness { return darkness }} +} + +func TestLiveLoadAndersNightOfflinePVIsZero(t *testing.T) { + s, opts := andersSite(DarknessNight) + r := s.LiveMeasurementNow("pixii", opts) + if !r.LoadKnown || r.LoadW != 980 || r.LoadQuality != LoadMeasured || r.LoadReason != "" { + t.Fatalf("night load = %v known=%t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } + if !r.PVLiveKnown || r.PVW != 0 { + t.Fatalf("dark PV not shown as 0 W: known=%t pv=%v", r.PVLiveKnown, r.PVW) + } + if src := r.Sources["growatt:pv"]; !src.Dark || src.PowerW == nil || *src.PowerW != 0 { + t.Fatal("dark source not marked 0 W") + } + // History and forecasts keep their own evidence. + if r.Valid || r.Known[DerPV] || r.Reason != "offline:growatt:pv" { + t.Fatalf("live display changed history qualification: valid=%t reason=%q", r.Valid, r.Reason) + } + if f := s.ForecastMeasurementNow("pixii", opts); f.Valid || f.Known[DerPV] { + t.Fatal("forecast path assumed dark PV") + } +} + +func TestLiveLoadTwilightOfflinePVIsEstimate(t *testing.T) { + s, opts := andersSite(DarknessTwilight) + r := s.LiveMeasurementNow("pixii", opts) + if !r.LoadKnown || r.LoadW != 980 || r.LoadQuality != LoadEstimated || r.LoadReason != "pv_twilight:growatt:pv" || !r.PVLiveKnown { + t.Fatalf("twilight: %v %t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } +} + +func TestLiveLoadDaytimeOfflinePVIsEstimateWithUnknownPV(t *testing.T) { + for _, opts := range []ForecastOptions{{}, {PVDark: func(time.Time) Darkness { return DarknessUnknown }}} { + s, _ := andersSite(DarknessUnknown) + r := s.LiveMeasurementNow("pixii", opts) + if !r.LoadKnown || r.LoadW != 980 || r.LoadQuality != LoadEstimated || r.LoadReason != "offline:growatt:pv" { + t.Fatalf("day: %v %t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } + if r.PVLiveKnown { + t.Fatal("daytime offline PV shown as known") + } + } +} + +func TestLiveLoadOfflineSiteMeterIsUnknown(t *testing.T) { + s, opts := andersSite(DarknessNight) + s.DriverHealthMut("pixii").SetOffline() + r := s.LiveMeasurementNow("pixii", opts) + if r.LoadKnown || r.LoadQuality != LoadUnknown || r.LoadReason != "offline:pixii:meter" { + t.Fatalf("offline meter: known=%t %s %q", r.LoadKnown, r.LoadQuality, r.LoadReason) + } + if r := s.LiveMeasurementNow("nometer", ForecastOptions{}); r.LoadKnown || r.LoadReason != "missing:nometer:meter" { + t.Fatalf("missing meter: known=%t %q", r.LoadKnown, r.LoadReason) + } +} + +func TestLiveLoadStaleSiteMeterIsUnknown(t *testing.T) { + s, opts := andersSite(DarknessNight) + s.Get("pixii", DerMeter).UpdatedAt = time.Now().Add(-10 * time.Minute) + if r := s.LiveMeasurementNow("pixii", opts); r.LoadKnown || r.LoadReason != "stale:pixii:meter" { + t.Fatalf("stale meter: known=%t %q", r.LoadKnown, r.LoadReason) + } +} + +// #1744: a loadpoint bound to a charger name with no driver. +func TestLiveLoadNonexistentChargerIsLeftOut(t *testing.T) { + s := NewStore() + s.EnsureDriverHealth("meter") + s.Update("meter", DerMeter, 1500, nil, nil) + opts := ForecastOptions{ExpectedFlows: []ForecastFlow{{Driver: "easee", DerType: DerEV}}} + r := s.LiveMeasurementNow("meter", opts) + if !r.LoadKnown || r.LoadW != 1500 || r.LoadQuality != LoadEstimated || r.LoadReason != "missing:easee:ev" { + t.Fatalf("#1744: %v %t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } + if r.Valid || r.Known[DerEV] { + t.Fatal("history qualification accepted the missing charger") + } +} + +func TestLiveLoadOfflineBatteryLeftOut(t *testing.T) { + s, opts := andersSite(DarknessNight) + s.EnsureDriverHealth("bat2") + s.Update("bat2", DerBattery, -3000, nil, nil) + s.DriverHealthMut("bat2").SetOffline() + r := s.LiveMeasurementNow("pixii", opts) + if !r.LoadKnown || r.LoadW != 980 || r.LoadQuality != LoadEstimated || r.LoadReason != "offline:bat2:battery" { + t.Fatalf("offline battery: %v %t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } +} + +func TestLiveLoadNegativeBalanceIsShown(t *testing.T) { + s, opts := andersSite(DarknessNight) + s.Update("pixii", DerBattery, 2000, nil, nil) + if r := s.LiveMeasurementNow("pixii", opts); !r.LoadKnown || r.LoadW != -1220 || r.LoadQuality != LoadEstimated || r.LoadReason != "invalid_balance" { + t.Fatalf("negative balance: %v known=%t %q", r.LoadW, r.LoadKnown, r.LoadReason) + } +} + +func TestLiveLoadMisconfigurationStillShowsValue(t *testing.T) { + s, opts := andersSite(DarknessNight) + opts.HouseholdInvalidReason = "unresolved_topology" + opts.ExpectedFlows = []ForecastFlow{{Driver: "ghost", DerType: DerBattery}, {Driver: "ghost-ev", DerType: DerEV}} + if r := s.LiveMeasurementNow("pixii", opts); !r.LoadKnown || r.LoadW != 980 || r.LoadQuality != LoadEstimated { + t.Fatalf("misconfiguration hid the balance: %v known=%t %q", r.LoadW, r.LoadKnown, r.LoadReason) + } +} + +func TestLiveLoadDarkDoesNotHideMalformedPV(t *testing.T) { + s, opts := andersSite(DarknessNight) + s.DriverHealthMut("growatt").RecordSuccess() + s.Get("growatt", DerPV).RawW = 300 // positive PV: a sign fault Update would refuse + r := s.LiveMeasurementNow("pixii", opts) + if r.PVLiveKnown || r.Sources["growatt:pv"].Dark || r.LoadQuality != LoadEstimated || r.LoadReason != "invalid_sign:growatt:pv" { + t.Fatalf("sign fault hidden at night: pv=%t %s %q", r.PVLiveKnown, r.LoadQuality, r.LoadReason) + } +} + +func TestLiveLoadDuplicateChargerIsNotSubtractedTwice(t *testing.T) { + s := NewStore() + for _, d := range []string{"meter", "wallbox"} { + s.EnsureDriverHealth(d) + } + s.Update("meter", DerMeter, 5000, nil, nil) + s.Update("wallbox", DerEV, 3000, nil, nil) + s.Update("wallbox", DerV2X, 3000, nil, nil) + r := s.LiveMeasurementNow("meter", ForecastOptions{}) + if !r.LoadKnown || r.LoadW != 5000 || r.LoadQuality != LoadEstimated || r.LoadReason != "duplicate_charger:wallbox" { + t.Fatalf("duplicate charger: %v known=%t %q", r.LoadW, r.LoadKnown, r.LoadReason) + } +} + +// Simulator run 21 Oct 2026, scenario 6: the inverter stops answering while +// the site meter keeps reading. Its last PV and battery are held, so Home use +// stays forgiving, but it is an estimate from the first failed read. +func TestLiveLoadFailedReadHoldsValueAsEstimate(t *testing.T) { + s := NewStore() + for _, d := range []string{"sdm", "sungrow"} { + s.EnsureDriverHealth(d) + } + s.Update("sdm", DerMeter, 0, nil, nil) + s.RecordDriverSuccess("sdm") + s.Update("sungrow", DerPV, -1500, nil, nil) + s.Update("sungrow", DerBattery, -570, nil, nil) + s.RecordDriverSuccess("sungrow") + if r := s.LiveMeasurementNow("sdm", ForecastOptions{}); r.LoadW != 2070 || r.LoadQuality != LoadMeasured { + t.Fatalf("before: %v %s %q", r.LoadW, r.LoadQuality, r.LoadReason) + } + + // Silent inverter: the house now draws from the grid. + s.RecordDriverError("sungrow", "modbus: timeout") + s.Update("sdm", DerMeter, 2070, nil, nil) + s.RecordDriverSuccess("sdm") + r := s.LiveMeasurementNow("sdm", ForecastOptions{}) + if !r.LoadKnown || r.LoadW != 4140 || r.LoadQuality != LoadEstimated || r.LoadReason != "read_failed:sungrow:battery" { + t.Fatalf("first failed read: %v known=%t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } + if !r.PVLiveKnown || r.PVW != -1500 { + t.Fatalf("held PV not shown: known=%t pv=%v", r.PVLiveKnown, r.PVW) + } + // History keeps its own policy. + if f := s.ForecastMeasurementNow("sdm", ForecastOptions{}); !f.Valid || f.Reason != "" || f.HouseholdW != 4140 { + t.Fatalf("history qualification changed: valid=%t %q %v", f.Valid, f.Reason, f.HouseholdW) + } + + // Errors go on past degraded, then the watchdog drops the held flows. + // Every later failed read keeps them dropped. + for range 5 { + s.RecordDriverError("sungrow", "modbus: timeout") + if r := s.LiveMeasurementNow("sdm", ForecastOptions{}); r.LoadW != 4140 || r.LoadQuality != LoadEstimated { + t.Fatalf("degraded: %v %s %q", r.LoadW, r.LoadQuality, r.LoadReason) + } + } + past := time.Now().Add(-2 * time.Minute) + s.DriverHealthMut("sungrow").LastSuccess = &past + if tr := s.WatchdogScan(time.Minute); len(tr) != 1 || tr[0].Name != "sungrow" || tr[0].Online { + t.Fatalf("watchdog transitions: %+v", tr) + } + for range 5 { + s.RecordDriverError("sungrow", "modbus: timeout") + if r := s.LiveMeasurementNow("sdm", ForecastOptions{}); r.LoadW != 2070 || r.LoadQuality != LoadEstimated || r.LoadReason != "offline:sungrow:battery" { + t.Fatalf("offline: %v %s %q", r.LoadW, r.LoadQuality, r.LoadReason) + } + if tr := s.WatchdogScan(time.Minute); len(tr) != 0 { + t.Fatalf("offline driver came back on a failed read: %+v", tr) + } + } + + // Reads succeed again. + s.Update("sungrow", DerPV, -1500, nil, nil) + s.Update("sungrow", DerBattery, -570, nil, nil) + s.RecordDriverSuccess("sungrow") + s.Update("sdm", DerMeter, 0, nil, nil) + if r := s.LiveMeasurementNow("sdm", ForecastOptions{}); r.LoadW != 2070 || r.LoadQuality != LoadMeasured || r.LoadReason != "" { + t.Fatalf("after: %v %s %q", r.LoadW, r.LoadQuality, r.LoadReason) + } +} + +// A failed read of the site meter holds its value too: an estimate, not blank. +func TestLiveLoadFailedSiteMeterReadIsEstimate(t *testing.T) { + s := NewStore() + for _, d := range []string{"sdm", "solar"} { + s.EnsureDriverHealth(d) + } + s.Update("sdm", DerMeter, 500, nil, nil) + s.Update("solar", DerPV, -1000, nil, nil) + s.RecordDriverSuccess("solar") + s.RecordDriverError("sdm", "modbus: timeout") + if r := s.LiveMeasurementNow("sdm", ForecastOptions{}); !r.LoadKnown || r.LoadW != 1500 || r.LoadQuality != LoadEstimated || r.LoadReason != "read_failed:sdm:meter" { + t.Fatalf("failed meter read: %v known=%t %s %q", r.LoadW, r.LoadKnown, r.LoadQuality, r.LoadReason) + } +} + +// A charger that refuses commands still reads: its draw is measured. +func TestLiveLoadCommandFaultStaysMeasured(t *testing.T) { + s := NewStore() + for _, d := range []string{"meter", "easee"} { + s.EnsureDriverHealth(d) + } + s.Update("meter", DerMeter, 12000, nil, nil) + s.Update("easee", DerEV, 11000, nil, nil) + s.RecordDriverSuccess("meter") + s.RecordDriverSuccess("easee") + s.SetDriverCommandFault("easee", true, "refused 3 commands") + s.SetDriverDeviceFault("easee", true, "charger reports a fault") + if r := s.LiveMeasurementNow("meter", ForecastOptions{}); r.LoadW != 1000 || r.LoadQuality != LoadMeasured || r.LoadReason != "" { + t.Fatalf("command fault: %v %s %q", r.LoadW, r.LoadQuality, r.LoadReason) + } +} diff --git a/go/internal/telemetry/polls_test.go b/go/internal/telemetry/polls_test.go new file mode 100644 index 000000000..14dca625a --- /dev/null +++ b/go/internal/telemetry/polls_test.go @@ -0,0 +1,51 @@ +package telemetry + +import ( + "testing" + "time" +) + +// The fleet report's driver facts: polls and failed polls over the last 24 +// clock hours, all drivers together, forgetting the hour a day back, and how +// much of the day the count covers. +func TestPollsLastDayCountsADayOfPollsAndErrors(t *testing.T) { + s := NewStore() + now := time.Date(2026, 10, 24, 9, 30, 0, 0, time.UTC) + s.born = now.Add(-30 * time.Hour) + s.mu.Lock() + for i := range 10 { + s.countPollLocked(now.Add(-25*time.Hour), i < 5) // a day and an hour ago: gone + s.countPollLocked(now.Add(-23*time.Hour), i < 2) + s.countPollLocked(now, false) + } + s.mu.Unlock() + if polls, errors, covered := s.PollsLastDay(now); polls != 20 || errors != 2 || covered != 24*time.Hour { + t.Fatalf("polls %d, errors %d, covered %s", polls, errors, covered) + } + if polls, _, _ := s.PollsLastDay(now.Add(48 * time.Hour)); polls != 0 { + t.Fatalf("two days on, %d polls remain", polls) + } + s.born = now.Add(-2 * time.Hour) + if _, _, covered := s.PollsLastDay(now); covered != 2*time.Hour { + t.Fatalf("two hours after a restart the count covers %s", covered) + } +} + +// Only the poll loop counts. An OCPP charger's messages record success and +// errors on its health, but they are not polls, so a site whose one driver +// is OCPP has nothing to report rather than a 100 % failure from one +// faulted connector. +func TestOnlyPollsAreCounted(t *testing.T) { + s := NewStore() + s.RecordDriverError("ocpp-charger", "connector faulted") + s.RecordDriverSuccess("ocpp-charger") + s.RecordDriverTick("inverter") + if polls, errors, _ := s.PollsLastDay(time.Now()); polls != 0 || errors != 0 { + t.Fatalf("health records were counted as polls: %d %d", polls, errors) + } + s.CountPoll(false) + s.CountPoll(true) + if polls, errors, _ := s.PollsLastDay(time.Now()); polls != 2 || errors != 1 { + t.Fatalf("polls %d, errors %d", polls, errors) + } +} diff --git a/go/internal/telemetry/store.go b/go/internal/telemetry/store.go index d25cebcbf..d9dc84309 100644 --- a/go/internal/telemetry/store.go +++ b/go/internal/telemetry/store.go @@ -217,11 +217,13 @@ func (h *DriverHealth) RecordTick() { } // RecordError bumps the error counter and degrades the status after 3 in a row. +// An offline driver stays offline: only fresh data brings it back. Lifting it +// to degraded made its held readings live again until the next watchdog scan. func (h *DriverHealth) RecordError(err string) { h.ConsecutiveErrors++ h.LastError = err h.TickCount++ - if h.ConsecutiveErrors >= 3 { + if h.ConsecutiveErrors >= 3 && h.Status != StatusOffline { h.Status = StatusDegraded } } @@ -274,6 +276,13 @@ func (h *DriverHealth) TelemetryLive() bool { return h != nil && h.Status != StatusOffline } +// LastReadFailed reports whether the driver's latest poll failed, so its +// readings are held from an earlier one. Only failed reads count: a refused +// command sets DeviceFault and leaves this alone. +func (h *DriverHealth) LastReadFailed() bool { + return h != nil && h.ConsecutiveErrors > 0 +} + // MetricSample is one (driver, metric, ts, value) tuple buffered for the // long-format TS database. State.Store consumes these via FlushSamples. type MetricSample struct { @@ -296,10 +305,6 @@ type Store struct { health map[string]*DriverHealth commands map[string]CommandEvidence controlObservations map[string][]ControlObservation - homeObservations map[string][]ControlObservation // display estimates only - homeSourceTime map[string]bool // reported source time, otherwise receipt time - homeIdleState map[string]time.Time // disconnected state time; zero means unqualified - homeEnergy map[string]*homeEnergySeries // timestamped session counters, display only processNoise float64 measurementNoise float64 @@ -319,6 +324,17 @@ type Store struct { // arrivals hear of each finished poll (see Arrived). arrivalsMu sync.Mutex arrivals map[chan struct{}]struct{} + + // pollHours counts driver polls and their errors, all drivers together, + // one slot per clock hour of the last day, since born. Guarded by mu. + pollHours [24]pollHour + born time.Time +} + +// pollHour is one clock hour's driver polls, all drivers together. +type pollHour struct { + hour int64 // Unix hours + polls, errors uint64 } type metricSnap struct { @@ -336,6 +352,7 @@ func NewStore() *Store { filters: make(map[string]*KalmanFilter1D), health: make(map[string]*DriverHealth), latestMetric: make(map[string]metricSnap), + born: time.Now(), processNoise: 100, // W of expected change between samples measurementNoise: 50, // W of sensor noise } @@ -478,7 +495,6 @@ func (s *Store) Update(driver string, t DerType, rawW float64, soc *float64, dat s.observeCommand(driver, t, rawW, data, now) s.recordControlObservation(driver, t, rawW, data, now) - s.recordHomeObservation(driver, t, rawW, data, now) s.freezeStepWindows(now) // Auto-buffer the standard fields (raw, not smoothed — we store ground @@ -766,6 +782,48 @@ func (s *Store) RecordDriverError(name, err string) { s.driverHealthLocked(name).RecordError(err) } +// CountPoll counts one finished driver poll, failed or not. The driver +// registry's poll loop is the one caller: an OCPP charger's messages are not +// polls and are counted neither way. +func (s *Store) CountPoll(failed bool) { + s.mu.Lock() + defer s.mu.Unlock() + s.countPollLocked(time.Now(), failed) +} + +// countPollLocked counts one poll in its clock hour, reusing the slot of the +// same hour a day before. +func (s *Store) countPollLocked(now time.Time, failed bool) { + hour := now.Unix() / 3600 + slot := &s.pollHours[hour%int64(len(s.pollHours))] + if slot.hour != hour { + *slot = pollHour{hour: hour} + } + slot.polls++ + if failed { + slot.errors++ + } +} + +// PollsLastDay is how many driver polls ran in the 24 clock hours up to now, +// all drivers together, how many of them failed, and how much of the day the +// count covers: the counts live in memory, so after a restart they cover +// only the time since. The anonymous fleet report coarsens the share into a +// bucket; no driver's name is counted. +func (s *Store) PollsLastDay(now time.Time) (polls, errors uint64, covered time.Duration) { + s.mu.RLock() + defer s.mu.RUnlock() + covered = min(24*time.Hour, max(0, now.Sub(s.born))) + hour := now.Unix() / 3600 + for _, slot := range s.pollHours { + if slot.hour > hour-int64(len(s.pollHours)) && slot.hour <= hour { + polls += slot.polls + errors += slot.errors + } + } + return polls, errors, covered +} + // Remove drops all in-memory state for a driver: readings, Kalman // filters, the health entry and the latest emit_metric snapshots. Called // from the driver Registry when a driver is removed from config (or @@ -785,10 +843,6 @@ func (s *Store) Remove(driver string) { delete(s.health, driver) for _, kind := range allDerTypes { delete(s.controlObservations, key(driver, kind)) - delete(s.homeObservations, key(driver, kind)) - delete(s.homeSourceTime, key(driver, kind)) - delete(s.homeIdleState, key(driver, kind)) - delete(s.homeEnergy, key(driver, kind)) } for k, c := range s.commands { c.Baseline, c.StepAfter = nil, nil diff --git a/go/protocol/cloudwire/allowlist.go b/go/protocol/cloudwire/allowlist.go index f4b21e66f..c9653865c 100644 --- a/go/protocol/cloudwire/allowlist.go +++ b/go/protocol/cloudwire/allowlist.go @@ -39,6 +39,10 @@ var Fields = map[string][]string{ MsgHB: {"clock", "srcRev", "liveSeq", "lpSeq", "ages"}, + // The app protocol's hist.chunk: power means over 5-minute buckets of + // the four series the app draws (TileSeries), and nothing else. + MsgTiles: {"tileId", "etag", "res", "startMs", "stepMs", "series", "data", "partial"}, + MsgEvent: {"eventId", "ts", "kind", "severity", "args", "src", "lp", "cars"}, MsgRoster: {"rosterSeq", "ttlS", "devices", "grant", "pending", "noise", "webpush"}, @@ -63,10 +67,13 @@ var Fields = map[string][]string{ // tun carries no header and no signature: d is Noise ciphertext its two // ends authenticate, and the service reads only dev and sid. MsgTun: {"dev", "sid", "d"}, + // x is the same: d is a watch's exchange with the box, and the service + // reads only dev and id. + MsgX: {"dev", "id", "d"}, } // Bodies maps each class the box sends to its body type. Classes the box does -// not send yet (plan, health, tiles) have no entry; the service treats their +// not send yet (plan, health) have no entry; the service treats their // bytes as opaque either way. tun is not here: it has no header and is not a // class, and Unheaded holds it. var Bodies = map[string]any{ @@ -76,6 +83,7 @@ var Bodies = map[string]any{ MsgCar: Cars{}, MsgPrices: Prices{}, MsgHB: HB{}, + MsgTiles: Tiles{}, MsgEvent: Event{}, MsgRoster: Roster{}, MsgBye: Bye{}, @@ -86,4 +94,5 @@ var Bodies = map[string]any{ // Unheaded are the box's messages without the common header. var Unheaded = map[string]any{ MsgTun: Tun{}, + MsgX: X{}, } diff --git a/go/protocol/cloudwire/exchange.go b/go/protocol/cloudwire/exchange.go new file mode 100644 index 000000000..dbfa1ba8d --- /dev/null +++ b/go/protocol/cloudwire/exchange.go @@ -0,0 +1,69 @@ +package cloudwire + +import "time" + +// The exchange through FTW Cloud: a watch's commands to its box (ADR 0009, +// decisions 1 and 2). docs/protocol-cloud.md, "Exchange", is the normative +// text. +// +// A watch has HTTPS and nothing lower, so it cannot hold a box session the +// way a phone holds one in the tunnel. It runs the box's exchange instead: +// short Noise sessions (ftw.exchange.v1, IK to the box's static key) that +// live on the box, one request and one answer at a time. Each request rides +// one one-shot request to the service as x {d}. The service adds the device +// id its own handshake proved and the request's number, sends +// x {dev, id, d} down /v1/box, and waits for the box's x {dev, id, d} with +// the same number. Its d goes back to the watch in message 2, as x {d}. +// The service keeps nothing after the answer, and reads neither d. +// +// An empty d from the service means no answer came: the box is not +// connected, does not take x, kept silent, or did not answer in time. An +// empty d from the box is its silence: a stranger, a spent session or a +// message it could not read. + +// MsgX is the exchange message on /v1/box and in a one-shot request. +const MsgX = "x" + +// CapX in hello.caps says the box answers x. The service sends x to no +// other box: an older one closes the uplink on a type it does not know. +const CapX = MsgX + +// Sizes. An exchange message is a handshake message 1 with one lane 0 frame +// (608 bytes), a request of one lane 0 frame (536) or an answer of two +// (1,072). XDataMax leaves room above the largest without letting d carry +// anything else. +const ( + XDataMax = 2048 + // MaxXFrame bounds one x frame on the uplink: d, the device id, the + // number and the envelope around them. + MaxXFrame = XDataMax + 256 +) + +// XAnswerWithin is how long the service waits for the box's answer. The box +// answers a command once its result settles, within 3 s, and no sooner than +// 400 ms after it arrived; a read and a handshake at once. +const XAnswerWithin = 5 * time.Second + +// XInFlight is how many exchanges the service has open at one box at a time, +// as many as the box answers at once. Past it the service answers an empty +// d at once. +const XInFlight = 8 + +// X is the exchange message on /v1/box, both ways. +type X struct { + // Dev is the roster device id of the watch. The service sets it from + // the key its own handshake with the watch proved. + Dev string `cbor:"dev"` + // ID is the service's number for one request, which the box copies + // into its answer. It means nothing else. + ID uint32 `cbor:"id"` + // D is one exchange message, or empty for no answer. + D []byte `cbor:"d"` +} + +// AppX is the exchange message in a one-shot request and its answer. It +// has no device id and no number: the service knows the device from its +// handshake, and one request has one answer. +type AppX struct { + D []byte `cbor:"d"` +} diff --git a/go/protocol/cloudwire/messages.go b/go/protocol/cloudwire/messages.go index 8ace59fe0..335ea26a1 100644 --- a/go/protocol/cloudwire/messages.go +++ b/go/protocol/cloudwire/messages.go @@ -42,9 +42,9 @@ const ( ) // Downlink is the closed set of types the service may send the box. None of -// the first five asks the box to act. tun carries ciphertext the box decrypts -// itself and judges as it judges the same session over the relay. -var Downlink = []string{MsgChallenge, MsgHelloOK, MsgWatch, MsgResync, MsgPurged, MsgTun} +// the first five asks the box to act. tun and x carry ciphertext the box +// decrypts itself and judges as it judges the same session anywhere else. +var Downlink = []string{MsgChallenge, MsgHelloOK, MsgWatch, MsgResync, MsgPurged, MsgTun, MsgX} // ClassScope is the scope a device needs to read each class the service // forwards. hb maps to "" because every device in the roster gets it: it diff --git a/go/protocol/cloudwire/once.go b/go/protocol/cloudwire/once.go index e76fce0df..39334bf5d 100644 --- a/go/protocol/cloudwire/once.go +++ b/go/protocol/cloudwire/once.go @@ -15,8 +15,9 @@ import ( // application/octet-stream. The body is IK message 1 to a service Noise key, // with OncePrologue(home) as the prologue, so a once handshake and an app // session handshake can never stand in for each other. Message 1 carries one -// unsigned envelope: read, token.get, push.register or push.unregister. The -// answer body is message 2 with the reply and then, for a read, the +// unsigned envelope: read, token.get, push.register, push.unregister or x +// (exchange.go). The answer body is message 2 with the reply and then, for a +// read, the // envelopes that did not fit, each message after its length as u16 big // endian. The service keeps no session and no Noise state after the answer. @@ -48,7 +49,7 @@ const ( // OnceMessages are the types message 1 may carry. The service refuses every // other type, an owner's suspend among them, before message 2. -var OnceMessages = []string{MsgRead, MsgTokenGet, MsgPushRegister, MsgPushUnregister} +var OnceMessages = []string{MsgRead, MsgTokenGet, MsgPushRegister, MsgPushUnregister, MsgX} // Read asks for the snapshot, less what the watch already holds. type Read struct { diff --git a/go/protocol/cloudwire/once_vectors_test.go b/go/protocol/cloudwire/once_vectors_test.go index d2e638314..445a1f20c 100644 --- a/go/protocol/cloudwire/once_vectors_test.go +++ b/go/protocol/cloudwire/once_vectors_test.go @@ -320,3 +320,165 @@ func TestPushRegisterNamesTheApp(t *testing.T) { t.Fatalf("a watch registration read app %q: %v", back.App, err) } } + +// testdata/cloud-once-x-vectors.json freezes one exchange request through +// the service byte for byte: the watch's x {d} in message 1, the x {dev, id, +// d} the service sends the box and the one the box sends back, and message 2 +// with the box's d as the service writes it. d is filler of the exchange's +// real lengths; the exchange's own bytes are in srcfl/ftw +// go/internal/appuplink/testdata/exchange-vectors.json. +const onceXVectorsPath = "testdata/cloud-once-x-vectors.json" + +type onceXVectorFile struct { + ServiceStatic string `json:"serviceStatic"` + ServicePublic string `json:"servicePublic"` + ServiceEphemeral string `json:"serviceEphemeral"` + WatchStatic string `json:"watchStatic"` + WatchPublic string `json:"watchPublic"` + WatchEphemeral string `json:"watchEphemeral"` + Home string `json:"home"` + Prologue string `json:"prologue"` + Msg1Payload string `json:"msg1Payload"` + Request string `json:"request"` + Down string `json:"down"` + Up string `json:"up"` + Msg2Payload string `json:"msg2Payload"` + Answer string `json:"answer"` + Notes map[string]string `json:"notes"` +} + +func TestOnceExchangeVectors(t *testing.T) { + labels := map[string]string{ + "serviceStatic": "ftw.cloud once vector service static", + "serviceEphemeral": "ftw.cloud once x vector service ephemeral", + "watchStatic": "ftw.cloud once vector watch static", + "watchEphemeral": "ftw.cloud once x vector watch ephemeral", + } + serviceStatic := fixedX25519(t, labels["serviceStatic"]) + serviceEph := fixedX25519(t, labels["serviceEphemeral"]) + watchStatic := fixedX25519(t, labels["watchStatic"]) + watchEph := fixedX25519(t, labels["watchEphemeral"]) + raw, err := os.ReadFile(vectorsPath) + if err != nil { + t.Fatal(err) + } + var signed vectorFile + if err := json.Unmarshal(raw, &signed); err != nil { + t.Fatal(err) + } + home := signed.BoxKey.Home + prologue := OncePrologue(home) + dev := DeviceID(watchStatic.Public()) + + // An exchange handshake's message 1, and its message 2 from the box. + request := bytes.Repeat([]byte{0x5a}, appwire.Message1Overhead+512) + answer := bytes.Repeat([]byte{0xa5}, appwire.Message2Overhead+1024) + msg1Payload, err := Unsigned(MsgX, AppX{D: request}) + if err != nil { + t.Fatal(err) + } + msg2Payload, err := Unsigned(MsgX, AppX{D: answer}) + if err != nil { + t.Fatal(err) + } + frame := func(x X) []byte { + body, err := Marshal(x) + if err != nil { + t.Fatal(err) + } + raw, err := Envelope{T: MsgX, B: body}.Encode() + if err != nil { + t.Fatal(err) + } + f, err := EncodeFrame(raw) + if err != nil { + t.Fatal(err) + } + if len(f) > MaxXFrame { + t.Fatalf("an x frame is %d bytes, over %d", len(f), MaxXFrame) + } + return f + } + down := frame(X{Dev: dev, ID: 1, D: request}) + up := frame(X{Dev: dev, ID: 1, D: answer}) + + watch, err := appwire.NewInitiator(appwire.InitiatorOptions{ + StaticKey: watchStatic, RemoteStatic: serviceStatic.Public(), Prologue: prologue, Ephemeral: &watchEph, + }) + if err != nil { + t.Fatal(err) + } + service, err := appwire.NewResponder(appwire.HandshakeOptions{StaticKey: serviceStatic, Prologue: prologue, Ephemeral: &serviceEph}) + if err != nil { + t.Fatal(err) + } + msg1, err := watch.WriteMessage(msg1Payload) + if err != nil { + t.Fatal(err) + } + if len(msg1) > OnceMessage1MaxBytes { + t.Fatalf("message 1 is %d bytes", len(msg1)) + } + got1, err := service.ReadMessage(msg1) + if err != nil || !bytes.Equal(got1, msg1Payload) { + t.Fatalf("message 1: %v", err) + } + msg2, err := service.WriteMessage(msg2Payload) + if err != nil { + t.Fatal(err) + } + body, err := OnceAnswer(msg2) + if err != nil { + t.Fatal(err) + } + parts, err := SplitOnceAnswer(body) + if err != nil || len(parts) != 1 { + t.Fatalf("split %d parts: %v", len(parts), err) + } + got2, err := watch.ReadMessage(parts[0]) + if err != nil { + t.Fatal(err) + } + env, err := DecodeEnvelope(got2) + var x AppX + if err != nil || env.T != MsgX || len(env.Sig) != 0 || Unmarshal(env.B, &x) != nil || !bytes.Equal(x.D, answer) { + t.Fatalf("the watch read message 2 as %+v: %v", env, err) + } + + out := onceXVectorFile{ + ServiceStatic: secretHex(labels["serviceStatic"]), ServicePublic: hex.EncodeToString(serviceStatic.Public()), + ServiceEphemeral: secretHex(labels["serviceEphemeral"]), + WatchStatic: secretHex(labels["watchStatic"]), WatchPublic: hex.EncodeToString(watchStatic.Public()), + WatchEphemeral: secretHex(labels["watchEphemeral"]), + Home: home, Prologue: hex.EncodeToString(prologue), + Msg1Payload: hex.EncodeToString(msg1Payload), Request: hex.EncodeToString(msg1), + Down: hex.EncodeToString(down), Up: hex.EncodeToString(up), + Msg2Payload: hex.EncodeToString(msg2Payload), Answer: hex.EncodeToString(body), + Notes: map[string]string{ + "keys": "Secrets are SHA-256 of the labels in once_vectors_test.go. X25519 clamps them.", + "request": "The POST body is message 1, which carries x {d}: one exchange message.", + "down": "The frame the service sends the box: x {dev, id, d}, dev from its own handshake, id its own number.", + "up": "The box's answer under the same dev and id; d goes to the watch unchanged.", + "answer": "Message 2 after its length as u16 big endian. An empty d means no answer came.", + }, + } + gotJSON, err := json.MarshalIndent(out, "", " ") + if err != nil { + t.Fatal(err) + } + gotJSON = append(gotJSON, '\n') + if *update { + if err := os.WriteFile(onceXVectorsPath, gotJSON, 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(onceXVectorsPath) + if err != nil { + t.Fatalf("%v (run with -update to create it)", err) + } + if !bytes.Equal(gotJSON, want) { + t.Fatalf("the bytes no longer match %s. If the wire change is intended, "+ + "regenerate with -update and pair the change with srcfl/ftw-cloud and the watch", onceXVectorsPath) + } +} diff --git a/go/protocol/cloudwire/testdata/cloud-once-x-vectors.json b/go/protocol/cloudwire/testdata/cloud-once-x-vectors.json new file mode 100644 index 000000000..248deab7e --- /dev/null +++ b/go/protocol/cloudwire/testdata/cloud-once-x-vectors.json @@ -0,0 +1,23 @@ +{ + "serviceStatic": "88b1d5e4491b3de125574d2eaa27a47229fc2c51d19fcb5dc6435b5ef61c380f", + "servicePublic": "0e9bd61ff213665b5e31282e6f2098d467245d224b7b4ad374a6ab4a8e5b594d", + "serviceEphemeral": "609f1c22cad0a21854fb399b775dbd1fad0939359bdd90e8cee3e801e762bd65", + "watchStatic": "8a87ac4f7d277626a8c36f7311c2bcd833e9abacefd9c0df90d39a916f19da94", + "watchPublic": "7ef44864cb0a1f0b998b0ad161d41dac007454a5717c2447c4021449a7873819", + "watchEphemeral": "bec12c753582d65382ef12695e6a90e1557c5d7f39b31f6d50fbf790bc6da93b", + "home": "fjzyr67buzpyncqmjliwgezvde", + "prologue": "6674772e636c6f75642e6f6e63652e76313a666a7a7972363762757a70796e63716d6a6c697767657a766465", + "msg1Payload": "a36162590266a161645902605a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a617461786373696740", + "request": "81f81f71eb08d9d76b6bd377987ba10cee30e50e899e9e3a41c892c69502c67736aadede071b5f30e77532d7ddea315bf2a3ff9fc054728a1cd79efd33d728eda7fa37dca20c64da5ff63d3508c222ef9b3d8f0fae16ff1a690ed5c62e44be646212200ec59c8312dca2b150a78708a30066978db0498c4eb38bce8d1a4cd34a8d114a72ca9776c7c4ffae03edd651f5bb87f9fe87b57b497711e2416cb3b3f9c239c4c56c3475ab46e7fa5a3c79efa74bbcd81b574a9512d071dfddc9afe1267826a5907ef5e9c7d154f85d53a275b9aa4948bc54a85a5acd4ce62080aea614992006a1310ed99e7be28f4d046f63a3bbbea80f6f0346b57962d3152209b6701c939a08bc22edb800bcb6790c2c5721469270f07db2a2f83a3e400922a5fe46c8d653df5e2bfa05c898e0d86dc0ab30fc1626d8a3a7f5a769c872055320f55003ad3bd35c1b6e56f98109091a51cdc8d07526829c0b1b3a4c8b604da85907e458c5664d7b1a0e012fd04a8a3e63038fb2dbf29cf10e367d0965ebcd7eb5bdc4a135c0378e798698de4801209fceb6e5c1fc7bf8eed107663313fdbc29ba233f8ff0087b52046b5a8bcd364068ddd97d345b855c22b020d87c6c6da188a9d04a24e5f9b31dc36cb4461122500b39b54cdbd0a5df53c40e3233f1bdaba63cd5f62b8ce94bf4ba48fb6c533b564b05d6803ae0ff5cbdd0270634372c52ae189a60756f9199f456655557781b13c387ed6d982f1794a8239cb8c4cc1da8992ff6f0b69fc4dacfbeec5539a912b60bfd0965bcc1a4466aacfbc4a73a6ebe09caf61e32cd2e6423d317e25292431fe30b38774982e236cc05f786e42c681287e2cd74a08620c1a409f498c6de7c2d64bb277cd835f65ac92ec404546ab234ff3c909ba442c2cf33870aa55ccce88f229c7c5f8b82b596e0552aec4c88f2e91b1203a0a8f83b09df6d4bd4033a2a9c5348c4fb9cd12e0d44178229f3421371d65ae6ea2686a1a44ba216826284d442a5d2895a789688d706", + "down": "010100000286a36162590277a361645902605a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a626964016364657668667652495a4d734b617461786373696740", + "up": "010100000456a36162590447a36164590430a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5626964016364657668667652495a4d734b617461786373696740", + "msg2Payload": "a36162590436a16164590430a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5a5617461786373696740", + "answer": "04758694100821300a95e0fb303174248d6fd825e563b58802430522b289579e372b871feab2457830ec66640e595692fef54ef6590377ad4f7c82c47fd877997fac51d04ac4d3af08053ea488b250bcc3ffb74ad54479fa117285210baa02566b7b202a6c7ffc23642f0537954e0ef0fa21fcd651283b7fbeeaa88862cdf2f3024330566ff98cc85a21b3156555bb7a5cc2493649591026392d224190e42159e459b2fa5f2831c7fa21b3a9bdb537fbc265095dfc94da3ee315492966a78ae5c0f5a9d089ee77f9e752df6c44cb96e4bdb76f65cd066e3f27855449a03e49ee1da96b560b5763afc2bb99a097e6bfece8db687fa4fe5c1280d3a8ff7d1365ce79b7762c0d9c74e8679ea232c5d9ea07094a1d17d202b2040cbbcb031056d0f9234052e531dd5707e13fcb1db62c2b2e01ab94858d5fab02bc48809db77b6aad49aa4949b566593ff67c93f79665498ff1d6dbfd5f21f6fd36e831c036a6dd6a4a90c6d970f2fcd152730c3961dbd9949dd7c0f740f660b9a21387b75ca03fc358abe1e2a6f8186b3942b99e4933ffa69b7c4be3ad45e667f13f3218ac2fe135b6c7de6621ac2350a6c065d2d83b8147d6b7e8cf24c7274a7653851940a74841d1365da6d3f63d0e9a343b7ae4c83bcd688ffa77ebac59c48c4f856b17f8a4b04606400a150014ac34e02a862e6e96b58c753f3775c092acba2ac47a8240fe94baf02d289dd7956027af2b1bb08e6ff4d4ff5c2dc8266ea3d1be9aa28f1d6ec74375daf3ffe9bb2c64036a092f7a0cd9bcfa1988c617b1a4c3f059e5601827ff738b88223666349e7a958294b8083a652a8599f5d065d5b16bc8a938038f5664ec6212cc6a739bbbf4de2c4e69dfe9064bc74ad2830636edab24d032cee34a7d3c8fac1d824a47d5577cacfedad4e55bb3615fad6434016e849184964178fea9a515ed0741d8225cbd05b8ee0c2340b7d07536029c7fc1fe00196f4b418ebf0ace805b97afc936688fcfcda5a3677b090f6dac8ff9b632eb312da27b73af219118acf2aeb94b1748c36c94c446b27ccf0fd7419f6fdcd08122572048671c6ed9089577ecba46ff10dde908081aee7f8bf9096ca9f4bb9856274181f8b4bcc2b0f30fe5001f3f0205068090ed9215aa28d1adfcb27fb934ffa66a5f751a1ea6a3c61821855b358fc6eb2672f2cb7d791309be412c5c326ab315b0d744cd593fbf7c46a470ab7d4d0f5a7dc788c928d7e20dfdfeceda97c673c37ffcbcef7d28047623e9a7c48887f9852802db41ccffe2e9b3eae74a31739b754cf4f0cd14cf23116a974245b06dd80ad7802891827507e4c2cf82d06bb4754901db9fbd67a1c64564ade957375718c67ec894c11bcba45c515aafd6d0112e83f2be2d19a907cc40d68e876921eae1c7d0dae4f68b467f141975d2259886f25cac7b51ee5c6b5ab90afd42fd3ce32c1d902a54af485e40c20bb488856119829c42424ab4ef0c0f6e1657aab82ae8735c162f1688317dfb89ae8de6c69a660f2363a96c15042ffa0fed76607c82eebad90d1bc7026d2cf96966eded6cd4e04787c641dd1c9db2f463f63d98b35a0609346f513638b33c82b905b926a44823", + "notes": { + "answer": "Message 2 after its length as u16 big endian. An empty d means no answer came.", + "down": "The frame the service sends the box: x {dev, id, d}, dev from its own handshake, id its own number.", + "keys": "Secrets are SHA-256 of the labels in once_vectors_test.go. X25519 clamps them.", + "request": "The POST body is message 1, which carries x {d}: one exchange message.", + "up": "The box's answer under the same dev and id; d goes to the watch unchanged." + } +} diff --git a/go/protocol/cloudwire/testdata/cloud-tile-vectors.json b/go/protocol/cloudwire/testdata/cloud-tile-vectors.json new file mode 100644 index 000000000..67ce8a6d5 --- /dev/null +++ b/go/protocol/cloudwire/testdata/cloud-tile-vectors.json @@ -0,0 +1,177 @@ +{ + "boxKey": { + "d": "6c50811aa47e688ecbb01b2c1e91aca4a2ad7c879526d1c9679ef3fd25560074", + "pk": "04cd7a7c09b7c1fed854fcba642c1da25b48df5bf3208c67bcc2158374011e8c84bf6490130896243c527842441c170a64252874fdf719aa61444e329fe6bc02b8", + "home": "fjzyr67buzpyncqmjliwgezvde" + }, + "series": [ + "grid_w", + "pv_w", + "battery_w", + "load_w" + ], + "tiles": [ + { + "name": "closed", + "tileId": "5m/1/41476", + "etag": "986e18cd", + "startMs": 1791763200000, + "points": 144, + "partial": false, + "seq": 20, + "body": "ac6372657362356d637365711464626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900200300001103000002030000f3020000e4020000d5020000c6020000b7020000a8020000990200008a0200007b0200006c0200005d0200004e0200003f02000030020000210200001202000003020000f4010000e5010000d6010000c7010000b8010000a90100009a0100008b0100007c0100006d0100005e0100004f0100004001000031010000220100001301000004010000f5000000e6000000d7000000c8000000b9000000aa0000009b0000008c0000007d0000006e0000005f000000500000004100000032000000230000001400000005000000f6ffffffe7ffffffd8ffffffc9ffffffbaffffffabffffff9cffffff8dffffff7effffff6fffffff60ffffff51ffffff42ffffff33ffffff24ffffff15ffffff06fffffff7feffffe8feffffd9feffffcafeffffbbfeffffacfeffff9dfeffff8efeffff7ffeffff70feffff61feffff52feffff43feffff34feffff25feffff16feffff07fefffff8fdffffe9fdffffdafdffffcbfdffffbcfdffffadfdffff9efdffff8ffdffff80fdffff71fdffff62fdffff53fdffff44fdffff35fdffff26fdffff17fdffff08fdfffff9fcffffeafcffffdbfcffffccfcffffbdfcffffaefcffff9ffcffff90fcffff81fcffff72fcffff63fcffff54fcffff45fcffff36fcffff27fcffff18fcffff09fcfffffafbffffebfbffffdcfbffffcdfbffffbefbffffaffbffffa0fbffff91fbffff82fbffff73fbffff64fbffff55fbffff46fbffff37fbffff28fbffff19fbffff0afbfffffbfaffffecfaffffddfaffffcefaffffbffaffff00000000b5ffffff6affffff1fffffffd4feffff8afeffff40fefffff6fdffffacfdffff63fdffff1afdffffd1fcffff89fcffff42fcfffffbfbffffb5fbffff70fbffff2bfbffffe7faffffa4faffff62faffff20faffffe0f9ffffa0f9ffff62f9ffff25f9ffffe8f8ffffadf8ffff73f8ffff3bf8ffff03f8ffffcdf7ffff98f7ffff64f7ffff32f7ffff01f7ffffd2f6ffffa4f6ffff78f6ffff4df6ffff24f6fffffcf5ffffd6f5ffffb1f5ffff8ef5ffff6df5ffff4ef5ffff30f5ffff14f5fffffaf4ffffe1f4ffffcaf4ffffb5f4ffffa2f4ffff91f4ffff81f4ffff74f4ffff68f4ffff5ef4ffff56f4ffff50f4ffff4bf4ffff49f4ffff48f4ffff49f4ffff4cf4ffff51f4ffff58f4ffff61f4ffff6cf4ffff78f4ffff86f4ffff96f4ffffa8f4ffffbcf4ffffd2f4ffffe9f4ffff02f5ffff1df5ffff3af5ffff58f5ffff78f5ffff9af5ffffbdf5ffffe2f5ffff09f6ffff31f6ffff5bf6ffff87f6ffffb3f6ffffe2f6ffff12f7ffff43f7ffff76f7ffffaaf7ffffdff7ffff16f8ffff4ef8ffff87f8ffffc1f8fffffdf8ffff39f9ffff77f9ffffb6f9fffff5f9ffff36faffff78faffffbafafffffefaffff42fbffff87fbffffcdfbffff13fcffff5afcffffa2fcffffeafcffff32fdffff7bfdffffc5fdffff0ffeffff59feffffa3feffffeefeffff38ffffff83ffffffceffffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900100008d0100008a0100008701000084010000810100007e0100007b0100007801000075010000720100006f0100006c010000690100006601000063010000600100005d0100005a0100005701000054010000510100004e0100004b0100004801000045010000420100003f0100003c010000390100003601000033010000300100002d0100002a0100002701000024010000210100001e0100001b0100001801000015010000120100000f0100000c01000009010000060100000301000000010000fd000000fa000000f7000000f4000000f1000000ee000000eb000000e8000000e5000000e2000000df000000dc000000d9000000d6000000d3000000d0000000cd000000ca000000c7000000c4000000c1000000be000000bb000000b8000000b5000000b2000000af000000ac000000a9000000a6000000a3000000a00000009d0000009a0000009700000094000000910000008e0000008b0000008800000085000000820000007f0000007c000000790000007600000073000000700000006d0000006a0000006700000064000000610000005e0000005b0000005800000055000000520000004f0000004c000000490000004600000043000000400000003d0000003a0000003700000034000000310000002e0000002b0000002800000025000000220000001f0000001c000000190000001600000013000000100000000d0000000a000000070000000400000001000000fefffffffbfffffff8fffffff5fffffff2ffffffefffffffecffffffe9ffffffe6ffffffe3ffffffc2010000c5010000c7010000ca010000cc010000cf010000d1010000d4010000d6010000d9010000db010000de010000e0010000e3010000e5010000e8010000ea010000ed010000ef010000f2010000f4010000f7010000f9010000fc010000fe010000010200000302000006020000080200000b0200000d020000100200001202000015020000170200001a0200000000008000000080000000800000008000000080000000802b0200002e020000300200003302000035020000380200003a0200003d0200003f020000420200004402000047020000490200004c0200004e020000510200005302000056020000580200005b0200005d020000600200006202000065020000670200006a0200006c0200006f020000710200007402000076020000790200007b0200007e020000800200008302000085020000880200008a0200008d0200008f020000920200009402000097020000990200009c0200009e020000a1020000a3020000a6020000a8020000ab020000ad020000b0020000b2020000b5020000b7020000ba020000bc020000bf020000c1020000c4020000c6020000c9020000cb020000ce020000d0020000d3020000d5020000d8020000da020000dd020000df020000e2020000e4020000e7020000e9020000ec020000ee020000f1020000f3020000f6020000f8020000fb020000fd020000000300000203000005030000070300000a0300000c0300000f030000110300001403000016030000190300001b0300001e030000200300002303000025030000280300006465746167683938366531386364667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437366677616c6c4d731b000001a12ffc5b30677061727469616cf46773746172744d731b000001a12d68b80068757074696d654d7319eb28", + "signedInputSha256": "3a55085d157a3c0db66c9463ece48a18f6d956eeaa6e403e8cabdfabf4dc7731", + "sig": "ef58627fcdebc178aa551ec1a64223d45d22488c58d699f0c26d9db6a989249248ac33233b4aa8d5196b4d5ff71046b54f700fc07c71084e08be200eb2af25d5", + "envelope": "a361625909b2ac6372657362356d637365711464626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900200300001103000002030000f3020000e4020000d5020000c6020000b7020000a8020000990200008a0200007b0200006c0200005d0200004e0200003f02000030020000210200001202000003020000f4010000e5010000d6010000c7010000b8010000a90100009a0100008b0100007c0100006d0100005e0100004f0100004001000031010000220100001301000004010000f5000000e6000000d7000000c8000000b9000000aa0000009b0000008c0000007d0000006e0000005f000000500000004100000032000000230000001400000005000000f6ffffffe7ffffffd8ffffffc9ffffffbaffffffabffffff9cffffff8dffffff7effffff6fffffff60ffffff51ffffff42ffffff33ffffff24ffffff15ffffff06fffffff7feffffe8feffffd9feffffcafeffffbbfeffffacfeffff9dfeffff8efeffff7ffeffff70feffff61feffff52feffff43feffff34feffff25feffff16feffff07fefffff8fdffffe9fdffffdafdffffcbfdffffbcfdffffadfdffff9efdffff8ffdffff80fdffff71fdffff62fdffff53fdffff44fdffff35fdffff26fdffff17fdffff08fdfffff9fcffffeafcffffdbfcffffccfcffffbdfcffffaefcffff9ffcffff90fcffff81fcffff72fcffff63fcffff54fcffff45fcffff36fcffff27fcffff18fcffff09fcfffffafbffffebfbffffdcfbffffcdfbffffbefbffffaffbffffa0fbffff91fbffff82fbffff73fbffff64fbffff55fbffff46fbffff37fbffff28fbffff19fbffff0afbfffffbfaffffecfaffffddfaffffcefaffffbffaffff00000000b5ffffff6affffff1fffffffd4feffff8afeffff40fefffff6fdffffacfdffff63fdffff1afdffffd1fcffff89fcffff42fcfffffbfbffffb5fbffff70fbffff2bfbffffe7faffffa4faffff62faffff20faffffe0f9ffffa0f9ffff62f9ffff25f9ffffe8f8ffffadf8ffff73f8ffff3bf8ffff03f8ffffcdf7ffff98f7ffff64f7ffff32f7ffff01f7ffffd2f6ffffa4f6ffff78f6ffff4df6ffff24f6fffffcf5ffffd6f5ffffb1f5ffff8ef5ffff6df5ffff4ef5ffff30f5ffff14f5fffffaf4ffffe1f4ffffcaf4ffffb5f4ffffa2f4ffff91f4ffff81f4ffff74f4ffff68f4ffff5ef4ffff56f4ffff50f4ffff4bf4ffff49f4ffff48f4ffff49f4ffff4cf4ffff51f4ffff58f4ffff61f4ffff6cf4ffff78f4ffff86f4ffff96f4ffffa8f4ffffbcf4ffffd2f4ffffe9f4ffff02f5ffff1df5ffff3af5ffff58f5ffff78f5ffff9af5ffffbdf5ffffe2f5ffff09f6ffff31f6ffff5bf6ffff87f6ffffb3f6ffffe2f6ffff12f7ffff43f7ffff76f7ffffaaf7ffffdff7ffff16f8ffff4ef8ffff87f8ffffc1f8fffffdf8ffff39f9ffff77f9ffffb6f9fffff5f9ffff36faffff78faffffbafafffffefaffff42fbffff87fbffffcdfbffff13fcffff5afcffffa2fcffffeafcffff32fdffff7bfdffffc5fdffff0ffeffff59feffffa3feffffeefeffff38ffffff83ffffffceffffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900100008d0100008a0100008701000084010000810100007e0100007b0100007801000075010000720100006f0100006c010000690100006601000063010000600100005d0100005a0100005701000054010000510100004e0100004b0100004801000045010000420100003f0100003c010000390100003601000033010000300100002d0100002a0100002701000024010000210100001e0100001b0100001801000015010000120100000f0100000c01000009010000060100000301000000010000fd000000fa000000f7000000f4000000f1000000ee000000eb000000e8000000e5000000e2000000df000000dc000000d9000000d6000000d3000000d0000000cd000000ca000000c7000000c4000000c1000000be000000bb000000b8000000b5000000b2000000af000000ac000000a9000000a6000000a3000000a00000009d0000009a0000009700000094000000910000008e0000008b0000008800000085000000820000007f0000007c000000790000007600000073000000700000006d0000006a0000006700000064000000610000005e0000005b0000005800000055000000520000004f0000004c000000490000004600000043000000400000003d0000003a0000003700000034000000310000002e0000002b0000002800000025000000220000001f0000001c000000190000001600000013000000100000000d0000000a000000070000000400000001000000fefffffffbfffffff8fffffff5fffffff2ffffffefffffffecffffffe9ffffffe6ffffffe3ffffffc2010000c5010000c7010000ca010000cc010000cf010000d1010000d4010000d6010000d9010000db010000de010000e0010000e3010000e5010000e8010000ea010000ed010000ef010000f2010000f4010000f7010000f9010000fc010000fe010000010200000302000006020000080200000b0200000d020000100200001202000015020000170200001a0200000000008000000080000000800000008000000080000000802b0200002e020000300200003302000035020000380200003a0200003d0200003f020000420200004402000047020000490200004c0200004e020000510200005302000056020000580200005b0200005d020000600200006202000065020000670200006a0200006c0200006f020000710200007402000076020000790200007b0200007e020000800200008302000085020000880200008a0200008d0200008f020000920200009402000097020000990200009c0200009e020000a1020000a3020000a6020000a8020000ab020000ad020000b0020000b2020000b5020000b7020000ba020000bc020000bf020000c1020000c4020000c6020000c9020000cb020000ce020000d0020000d3020000d5020000d8020000da020000dd020000df020000e2020000e4020000e7020000e9020000ec020000ee020000f1020000f3020000f6020000f8020000fb020000fd020000000300000203000005030000070300000a0300000c0300000f030000110300001403000016030000190300001b0300001e030000200300002303000025030000280300006465746167683938366531386364667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437366677616c6c4d731b000001a12ffc5b30677061727469616cf46773746172744d731b000001a12d68b80068757074696d654d7319eb2861746574696c6573637369675840ef58627fcdebc178aa551ec1a64223d45d22488c58d699f0c26d9db6a989249248ac33233b4aa8d5196b4d5ff71046b54f700fc07c71084e08be200eb2af25d5", + "frame": "010100000a06a361625909b2ac6372657362356d637365711464626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900200300001103000002030000f3020000e4020000d5020000c6020000b7020000a8020000990200008a0200007b0200006c0200005d0200004e0200003f02000030020000210200001202000003020000f4010000e5010000d6010000c7010000b8010000a90100009a0100008b0100007c0100006d0100005e0100004f0100004001000031010000220100001301000004010000f5000000e6000000d7000000c8000000b9000000aa0000009b0000008c0000007d0000006e0000005f000000500000004100000032000000230000001400000005000000f6ffffffe7ffffffd8ffffffc9ffffffbaffffffabffffff9cffffff8dffffff7effffff6fffffff60ffffff51ffffff42ffffff33ffffff24ffffff15ffffff06fffffff7feffffe8feffffd9feffffcafeffffbbfeffffacfeffff9dfeffff8efeffff7ffeffff70feffff61feffff52feffff43feffff34feffff25feffff16feffff07fefffff8fdffffe9fdffffdafdffffcbfdffffbcfdffffadfdffff9efdffff8ffdffff80fdffff71fdffff62fdffff53fdffff44fdffff35fdffff26fdffff17fdffff08fdfffff9fcffffeafcffffdbfcffffccfcffffbdfcffffaefcffff9ffcffff90fcffff81fcffff72fcffff63fcffff54fcffff45fcffff36fcffff27fcffff18fcffff09fcfffffafbffffebfbffffdcfbffffcdfbffffbefbffffaffbffffa0fbffff91fbffff82fbffff73fbffff64fbffff55fbffff46fbffff37fbffff28fbffff19fbffff0afbfffffbfaffffecfaffffddfaffffcefaffffbffaffff00000000b5ffffff6affffff1fffffffd4feffff8afeffff40fefffff6fdffffacfdffff63fdffff1afdffffd1fcffff89fcffff42fcfffffbfbffffb5fbffff70fbffff2bfbffffe7faffffa4faffff62faffff20faffffe0f9ffffa0f9ffff62f9ffff25f9ffffe8f8ffffadf8ffff73f8ffff3bf8ffff03f8ffffcdf7ffff98f7ffff64f7ffff32f7ffff01f7ffffd2f6ffffa4f6ffff78f6ffff4df6ffff24f6fffffcf5ffffd6f5ffffb1f5ffff8ef5ffff6df5ffff4ef5ffff30f5ffff14f5fffffaf4ffffe1f4ffffcaf4ffffb5f4ffffa2f4ffff91f4ffff81f4ffff74f4ffff68f4ffff5ef4ffff56f4ffff50f4ffff4bf4ffff49f4ffff48f4ffff49f4ffff4cf4ffff51f4ffff58f4ffff61f4ffff6cf4ffff78f4ffff86f4ffff96f4ffffa8f4ffffbcf4ffffd2f4ffffe9f4ffff02f5ffff1df5ffff3af5ffff58f5ffff78f5ffff9af5ffffbdf5ffffe2f5ffff09f6ffff31f6ffff5bf6ffff87f6ffffb3f6ffffe2f6ffff12f7ffff43f7ffff76f7ffffaaf7ffffdff7ffff16f8ffff4ef8ffff87f8ffffc1f8fffffdf8ffff39f9ffff77f9ffffb6f9fffff5f9ffff36faffff78faffffbafafffffefaffff42fbffff87fbffffcdfbffff13fcffff5afcffffa2fcffffeafcffff32fdffff7bfdffffc5fdffff0ffeffff59feffffa3feffffeefeffff38ffffff83ffffffceffffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900100008d0100008a0100008701000084010000810100007e0100007b0100007801000075010000720100006f0100006c010000690100006601000063010000600100005d0100005a0100005701000054010000510100004e0100004b0100004801000045010000420100003f0100003c010000390100003601000033010000300100002d0100002a0100002701000024010000210100001e0100001b0100001801000015010000120100000f0100000c01000009010000060100000301000000010000fd000000fa000000f7000000f4000000f1000000ee000000eb000000e8000000e5000000e2000000df000000dc000000d9000000d6000000d3000000d0000000cd000000ca000000c7000000c4000000c1000000be000000bb000000b8000000b5000000b2000000af000000ac000000a9000000a6000000a3000000a00000009d0000009a0000009700000094000000910000008e0000008b0000008800000085000000820000007f0000007c000000790000007600000073000000700000006d0000006a0000006700000064000000610000005e0000005b0000005800000055000000520000004f0000004c000000490000004600000043000000400000003d0000003a0000003700000034000000310000002e0000002b0000002800000025000000220000001f0000001c000000190000001600000013000000100000000d0000000a000000070000000400000001000000fefffffffbfffffff8fffffff5fffffff2ffffffefffffffecffffffe9ffffffe6ffffffe3ffffffc2010000c5010000c7010000ca010000cc010000cf010000d1010000d4010000d6010000d9010000db010000de010000e0010000e3010000e5010000e8010000ea010000ed010000ef010000f2010000f4010000f7010000f9010000fc010000fe010000010200000302000006020000080200000b0200000d020000100200001202000015020000170200001a0200000000008000000080000000800000008000000080000000802b0200002e020000300200003302000035020000380200003a0200003d0200003f020000420200004402000047020000490200004c0200004e020000510200005302000056020000580200005b0200005d020000600200006202000065020000670200006a0200006c0200006f020000710200007402000076020000790200007b0200007e020000800200008302000085020000880200008a0200008d0200008f020000920200009402000097020000990200009c0200009e020000a1020000a3020000a6020000a8020000ab020000ad020000b0020000b2020000b5020000b7020000ba020000bc020000bf020000c1020000c4020000c6020000c9020000cb020000ce020000d0020000d3020000d5020000d8020000da020000dd020000df020000e2020000e4020000e7020000e9020000ec020000ee020000f1020000f3020000f6020000f8020000fb020000fd020000000300000203000005030000070300000a0300000c0300000f030000110300001403000016030000190300001b0300001e030000200300002303000025030000280300006465746167683938366531386364667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437366677616c6c4d731b000001a12ffc5b30677061727469616cf46773746172744d731b000001a12d68b80068757074696d654d7319eb2861746574696c6573637369675840ef58627fcdebc178aa551ec1a64223d45d22488c58d699f0c26d9db6a989249248ac33233b4aa8d5196b4d5ff71046b54f700fc07c71084e08be200eb2af25d5" + }, + { + "name": "whole", + "tileId": "5m/1/41477", + "etag": "7285c700", + "startMs": 1791806400000, + "points": 144, + "partial": true, + "seq": 21, + "body": "ac6372657362356d637365711564626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be0300000000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000806465746167683732383563373030667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1310f03b0677061727469616cf56773746172744d731b000001a12ffbe60068757074696d654d7319eb32", + "signedInputSha256": "8b0f2e108c7c018c10927f7b960b3644316fd9a3a692f2eac230024589884d84", + "sig": "b48eeac0518263ea7e8115b94f495e9d86a4fde190243a54bd7450f55b7e1136c2fed27664d4a9c4d89560526f429574ac8242d7303fb8ce5500dd3e5c4ed861", + "envelope": "a361625909b2ac6372657362356d637365711564626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be0300000000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000806465746167683732383563373030667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1310f03b0677061727469616cf56773746172744d731b000001a12ffbe60068757074696d654d7319eb3261746574696c6573637369675840b48eeac0518263ea7e8115b94f495e9d86a4fde190243a54bd7450f55b7e1136c2fed27664d4a9c4d89560526f429574ac8242d7303fb8ce5500dd3e5c4ed861", + "frame": "010100000a06a361625909b2ac6372657362356d637365711564626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461590900b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be0300000000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000806465746167683732383563373030667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1310f03b0677061727469616cf56773746172744d731b000001a12ffbe60068757074696d654d7319eb3261746574696c6573637369675840b48eeac0518263ea7e8115b94f495e9d86a4fde190243a54bd7450f55b7e1136c2fed27664d4a9c4d89560526f429574ac8242d7303fb8ce5500dd3e5c4ed861" + }, + { + "name": "append 17:00", + "tileId": "5m/1/41477", + "etag": "229cca34", + "startMs": 1791824400000, + "points": 1, + "partial": true, + "seq": 22, + "body": "ac6372657362356d637365711664626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461502cf7ffff000000002cffffffc00300006465746167683232396363613334667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a131139790677061727469616cf56773746172744d731b000001a1310e8e8068757074696d654d7319eb3c", + "signedInputSha256": "31b4596a56ad6722bc4367cddf42a01c5d998cc12da5c5ed9ef5c2afc357aa97", + "sig": "547fd8fbd017964d5800bd918b35cf40ec97889659cc93ca571d0b6d0fe9ce5aca4f10c706b790026902c13e77a470b783c01d10e24949214b9902187101f422", + "envelope": "a3616258c0ac6372657362356d637365711664626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461502cf7ffff000000002cffffffc00300006465746167683232396363613334667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a131139790677061727469616cf56773746172744d731b000001a1310e8e8068757074696d654d7319eb3c61746574696c6573637369675840547fd8fbd017964d5800bd918b35cf40ec97889659cc93ca571d0b6d0fe9ce5aca4f10c706b790026902c13e77a470b783c01d10e24949214b9902187101f422", + "frame": "010100000113a3616258c0ac6372657362356d637365711664626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b06464617461502cf7ffff000000002cffffffc00300006465746167683232396363613334667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a131139790677061727469616cf56773746172744d731b000001a1310e8e8068757074696d654d7319eb3c61746574696c6573637369675840547fd8fbd017964d5800bd918b35cf40ec97889659cc93ca571d0b6d0fe9ce5aca4f10c706b790026902c13e77a470b783c01d10e24949214b9902187101f422" + }, + { + "name": "append 17:05", + "tileId": "5m/1/41477", + "etag": "0044f57f", + "startMs": 1791824700000, + "points": 2, + "partial": true, + "seq": 23, + "body": "ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201df7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683030343466353766667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb46", + "signedInputSha256": "2988782538a7ec1f2d63315375a01d829ffd9dadf242a84e4730f1eebea80f76", + "sig": "720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5", + "envelope": "a3616258d1ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201df7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683030343466353766667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb4661746574696c6573637369675840720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5", + "frame": "010100000124a3616258d1ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201df7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683030343466353766667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb4661746574696c6573637369675840720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5" + } + ], + "forged": { + "name": "forged append", + "tileId": "5m/1/41477", + "etag": "86ff94ae", + "startMs": 1791824700000, + "points": 2, + "partial": true, + "seq": 23, + "body": "ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201cf7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683836666639346165667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb46", + "signedInputSha256": "", + "sig": "720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5", + "envelope": "a3616258d1ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201cf7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683836666639346165667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb4661746574696c6573637369675840720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5", + "frame": "010100000124a3616258d1ac6372657362356d637365711764626f6f7450b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0646461746158201cf7ffff0ef7ffff000000000000000029ffffff26ffffffc3030000c50300006465746167683836666639346165667365726965738466677269645f776470765f7769626174746572795f77666c6f61645f7766737465704d731a000493e06674696c6549646a356d2f312f34313437376677616c6c4d731b000001a1311cbf50677061727469616cf56773746172744d731b000001a13113226068757074696d654d7319eb4661746574696c6573637369675840720790a10b1eb0806b4cf20594c97745ab1f4edc3d5835e7188da3bb7a88efe46e832e9b867de8d539ed15f2165123a0087a158f9d128298b0f36667b3165cf5" + }, + "laid": [ + { + "of": [ + "closed" + ], + "tileId": "5m/1/41476", + "etag": "986e18cd", + "startMs": 1791763200000, + "partial": false, + "data": "200300001103000002030000f3020000e4020000d5020000c6020000b7020000a8020000990200008a0200007b0200006c0200005d0200004e0200003f02000030020000210200001202000003020000f4010000e5010000d6010000c7010000b8010000a90100009a0100008b0100007c0100006d0100005e0100004f0100004001000031010000220100001301000004010000f5000000e6000000d7000000c8000000b9000000aa0000009b0000008c0000007d0000006e0000005f000000500000004100000032000000230000001400000005000000f6ffffffe7ffffffd8ffffffc9ffffffbaffffffabffffff9cffffff8dffffff7effffff6fffffff60ffffff51ffffff42ffffff33ffffff24ffffff15ffffff06fffffff7feffffe8feffffd9feffffcafeffffbbfeffffacfeffff9dfeffff8efeffff7ffeffff70feffff61feffff52feffff43feffff34feffff25feffff16feffff07fefffff8fdffffe9fdffffdafdffffcbfdffffbcfdffffadfdffff9efdffff8ffdffff80fdffff71fdffff62fdffff53fdffff44fdffff35fdffff26fdffff17fdffff08fdfffff9fcffffeafcffffdbfcffffccfcffffbdfcffffaefcffff9ffcffff90fcffff81fcffff72fcffff63fcffff54fcffff45fcffff36fcffff27fcffff18fcffff09fcfffffafbffffebfbffffdcfbffffcdfbffffbefbffffaffbffffa0fbffff91fbffff82fbffff73fbffff64fbffff55fbffff46fbffff37fbffff28fbffff19fbffff0afbfffffbfaffffecfaffffddfaffffcefaffffbffaffff00000000b5ffffff6affffff1fffffffd4feffff8afeffff40fefffff6fdffffacfdffff63fdffff1afdffffd1fcffff89fcffff42fcfffffbfbffffb5fbffff70fbffff2bfbffffe7faffffa4faffff62faffff20faffffe0f9ffffa0f9ffff62f9ffff25f9ffffe8f8ffffadf8ffff73f8ffff3bf8ffff03f8ffffcdf7ffff98f7ffff64f7ffff32f7ffff01f7ffffd2f6ffffa4f6ffff78f6ffff4df6ffff24f6fffffcf5ffffd6f5ffffb1f5ffff8ef5ffff6df5ffff4ef5ffff30f5ffff14f5fffffaf4ffffe1f4ffffcaf4ffffb5f4ffffa2f4ffff91f4ffff81f4ffff74f4ffff68f4ffff5ef4ffff56f4ffff50f4ffff4bf4ffff49f4ffff48f4ffff49f4ffff4cf4ffff51f4ffff58f4ffff61f4ffff6cf4ffff78f4ffff86f4ffff96f4ffffa8f4ffffbcf4ffffd2f4ffffe9f4ffff02f5ffff1df5ffff3af5ffff58f5ffff78f5ffff9af5ffffbdf5ffffe2f5ffff09f6ffff31f6ffff5bf6ffff87f6ffffb3f6ffffe2f6ffff12f7ffff43f7ffff76f7ffffaaf7ffffdff7ffff16f8ffff4ef8ffff87f8ffffc1f8fffffdf8ffff39f9ffff77f9ffffb6f9fffff5f9ffff36faffff78faffffbafafffffefaffff42fbffff87fbffffcdfbffff13fcffff5afcffffa2fcffffeafcffff32fdffff7bfdffffc5fdffff0ffeffff59feffffa3feffffeefeffff38ffffff83ffffffceffffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900100008d0100008a0100008701000084010000810100007e0100007b0100007801000075010000720100006f0100006c010000690100006601000063010000600100005d0100005a0100005701000054010000510100004e0100004b0100004801000045010000420100003f0100003c010000390100003601000033010000300100002d0100002a0100002701000024010000210100001e0100001b0100001801000015010000120100000f0100000c01000009010000060100000301000000010000fd000000fa000000f7000000f4000000f1000000ee000000eb000000e8000000e5000000e2000000df000000dc000000d9000000d6000000d3000000d0000000cd000000ca000000c7000000c4000000c1000000be000000bb000000b8000000b5000000b2000000af000000ac000000a9000000a6000000a3000000a00000009d0000009a0000009700000094000000910000008e0000008b0000008800000085000000820000007f0000007c000000790000007600000073000000700000006d0000006a0000006700000064000000610000005e0000005b0000005800000055000000520000004f0000004c000000490000004600000043000000400000003d0000003a0000003700000034000000310000002e0000002b0000002800000025000000220000001f0000001c000000190000001600000013000000100000000d0000000a000000070000000400000001000000fefffffffbfffffff8fffffff5fffffff2ffffffefffffffecffffffe9ffffffe6ffffffe3ffffffc2010000c5010000c7010000ca010000cc010000cf010000d1010000d4010000d6010000d9010000db010000de010000e0010000e3010000e5010000e8010000ea010000ed010000ef010000f2010000f4010000f7010000f9010000fc010000fe010000010200000302000006020000080200000b0200000d020000100200001202000015020000170200001a0200000000008000000080000000800000008000000080000000802b0200002e020000300200003302000035020000380200003a0200003d0200003f020000420200004402000047020000490200004c0200004e020000510200005302000056020000580200005b0200005d020000600200006202000065020000670200006a0200006c0200006f020000710200007402000076020000790200007b0200007e020000800200008302000085020000880200008a0200008d0200008f020000920200009402000097020000990200009c0200009e020000a1020000a3020000a6020000a8020000ab020000ad020000b0020000b2020000b5020000b7020000ba020000bc020000bf020000c1020000c4020000c6020000c9020000cb020000ce020000d0020000d3020000d5020000d8020000da020000dd020000df020000e2020000e4020000e7020000e9020000ec020000ee020000f1020000f3020000f6020000f8020000fb020000fd020000000300000203000005030000070300000a0300000c0300000f030000110300001403000016030000190300001b0300001e03000020030000230300002503000028030000" + }, + { + "of": [ + "whole" + ], + "tileId": "5m/1/41477", + "etag": "7285c700", + "startMs": 1791806400000, + "partial": true, + "data": "b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be030000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080" + }, + { + "of": [ + "whole", + "append 17:00", + "append 17:05" + ], + "tileId": "5m/1/41477", + "etag": "96002093", + "startMs": 1791806400000, + "partial": true, + "data": "b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff2cf7ffff1df7ffff0ef7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff2cffffff29ffffff26ffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be030000c0030000c3030000c5030000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080" + }, + { + "of": [ + "closed", + "whole", + "append 17:00", + "append 17:05" + ], + "tileId": "5m/1/41476", + "etag": "986e18cd", + "startMs": 1791763200000, + "partial": false, + "data": "200300001103000002030000f3020000e4020000d5020000c6020000b7020000a8020000990200008a0200007b0200006c0200005d0200004e0200003f02000030020000210200001202000003020000f4010000e5010000d6010000c7010000b8010000a90100009a0100008b0100007c0100006d0100005e0100004f0100004001000031010000220100001301000004010000f5000000e6000000d7000000c8000000b9000000aa0000009b0000008c0000007d0000006e0000005f000000500000004100000032000000230000001400000005000000f6ffffffe7ffffffd8ffffffc9ffffffbaffffffabffffff9cffffff8dffffff7effffff6fffffff60ffffff51ffffff42ffffff33ffffff24ffffff15ffffff06fffffff7feffffe8feffffd9feffffcafeffffbbfeffffacfeffff9dfeffff8efeffff7ffeffff70feffff61feffff52feffff43feffff34feffff25feffff16feffff07fefffff8fdffffe9fdffffdafdffffcbfdffffbcfdffffadfdffff9efdffff8ffdffff80fdffff71fdffff62fdffff53fdffff44fdffff35fdffff26fdffff17fdffff08fdfffff9fcffffeafcffffdbfcffffccfcffffbdfcffffaefcffff9ffcffff90fcffff81fcffff72fcffff63fcffff54fcffff45fcffff36fcffff27fcffff18fcffff09fcfffffafbffffebfbffffdcfbffffcdfbffffbefbffffaffbffffa0fbffff91fbffff82fbffff73fbffff64fbffff55fbffff46fbffff37fbffff28fbffff19fbffff0afbfffffbfaffffecfaffffddfaffffcefaffffbffaffff00000000b5ffffff6affffff1fffffffd4feffff8afeffff40fefffff6fdffffacfdffff63fdffff1afdffffd1fcffff89fcffff42fcfffffbfbffffb5fbffff70fbffff2bfbffffe7faffffa4faffff62faffff20faffffe0f9ffffa0f9ffff62f9ffff25f9ffffe8f8ffffadf8ffff73f8ffff3bf8ffff03f8ffffcdf7ffff98f7ffff64f7ffff32f7ffff01f7ffffd2f6ffffa4f6ffff78f6ffff4df6ffff24f6fffffcf5ffffd6f5ffffb1f5ffff8ef5ffff6df5ffff4ef5ffff30f5ffff14f5fffffaf4ffffe1f4ffffcaf4ffffb5f4ffffa2f4ffff91f4ffff81f4ffff74f4ffff68f4ffff5ef4ffff56f4ffff50f4ffff4bf4ffff49f4ffff48f4ffff49f4ffff4cf4ffff51f4ffff58f4ffff61f4ffff6cf4ffff78f4ffff86f4ffff96f4ffffa8f4ffffbcf4ffffd2f4ffffe9f4ffff02f5ffff1df5ffff3af5ffff58f5ffff78f5ffff9af5ffffbdf5ffffe2f5ffff09f6ffff31f6ffff5bf6ffff87f6ffffb3f6ffffe2f6ffff12f7ffff43f7ffff76f7ffffaaf7ffffdff7ffff16f8ffff4ef8ffff87f8ffffc1f8fffffdf8ffff39f9ffff77f9ffffb6f9fffff5f9ffff36faffff78faffffbafafffffefaffff42fbffff87fbffffcdfbffff13fcffff5afcffffa2fcffffeafcffff32fdffff7bfdffffc5fdffff0ffeffff59feffffa3feffffeefeffff38ffffff83ffffffceffffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000900100008d0100008a0100008701000084010000810100007e0100007b0100007801000075010000720100006f0100006c010000690100006601000063010000600100005d0100005a0100005701000054010000510100004e0100004b0100004801000045010000420100003f0100003c010000390100003601000033010000300100002d0100002a0100002701000024010000210100001e0100001b0100001801000015010000120100000f0100000c01000009010000060100000301000000010000fd000000fa000000f7000000f4000000f1000000ee000000eb000000e8000000e5000000e2000000df000000dc000000d9000000d6000000d3000000d0000000cd000000ca000000c7000000c4000000c1000000be000000bb000000b8000000b5000000b2000000af000000ac000000a9000000a6000000a3000000a00000009d0000009a0000009700000094000000910000008e0000008b0000008800000085000000820000007f0000007c000000790000007600000073000000700000006d0000006a0000006700000064000000610000005e0000005b0000005800000055000000520000004f0000004c000000490000004600000043000000400000003d0000003a0000003700000034000000310000002e0000002b0000002800000025000000220000001f0000001c000000190000001600000013000000100000000d0000000a000000070000000400000001000000fefffffffbfffffff8fffffff5fffffff2ffffffefffffffecffffffe9ffffffe6ffffffe3ffffffc2010000c5010000c7010000ca010000cc010000cf010000d1010000d4010000d6010000d9010000db010000de010000e0010000e3010000e5010000e8010000ea010000ed010000ef010000f2010000f4010000f7010000f9010000fc010000fe010000010200000302000006020000080200000b0200000d020000100200001202000015020000170200001a0200000000008000000080000000800000008000000080000000802b0200002e020000300200003302000035020000380200003a0200003d0200003f020000420200004402000047020000490200004c0200004e020000510200005302000056020000580200005b0200005d020000600200006202000065020000670200006a0200006c0200006f020000710200007402000076020000790200007b0200007e020000800200008302000085020000880200008a0200008d0200008f020000920200009402000097020000990200009c0200009e020000a1020000a3020000a6020000a8020000ab020000ad020000b0020000b2020000b5020000b7020000ba020000bc020000bf020000c1020000c4020000c6020000c9020000cb020000ce020000d0020000d3020000d5020000d8020000da020000dd020000df020000e2020000e4020000e7020000e9020000ec020000ee020000f1020000f3020000f6020000f8020000fb020000fd020000000300000203000005030000070300000a0300000c0300000f030000110300001403000016030000190300001b0300001e03000020030000230300002503000028030000" + }, + { + "of": [ + "closed", + "whole", + "append 17:00", + "append 17:05" + ], + "tileId": "5m/1/41477", + "etag": "96002093", + "startMs": 1791806400000, + "partial": true, + "data": "b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff2cf7ffff1df7ffff0ef7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff2cffffff29ffffff26ffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be030000c0030000c3030000c5030000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080" + }, + { + "of": [ + "append 17:00", + "whole" + ], + "tileId": "5m/1/41477", + "etag": "7285c700", + "startMs": 1791806400000, + "partial": true, + "data": "b0faffffa1faffff92faffff83faffff74faffff65faffff56faffff47faffff38faffff29faffff1afaffff0bfafffffcf9ffffedf9ffffdef9ffffcff9ffffc0f9ffffb1f9ffffa2f9ffff93f9ffff84f9ffff75f9ffff66f9ffff57f9ffff48f9ffff39f9ffff2af9ffff1bf9ffff0cf9fffffdf8ffffeef8ffffdff8ffffd0f8ffffc1f8ffffb2f8ffffa3f8ffff94f8ffff85f8ffff76f8ffff67f8ffff58f8ffff49f8ffff3af8ffff2bf8ffff1cf8ffff0df8fffffef7ffffeff7ffffe0f7ffffd1f7ffffc2f7ffffb3f7ffffa4f7ffff95f7ffff86f7ffff77f7ffff68f7ffff59f7ffff4af7ffff3bf7ffff000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080bcffffffb9ffffffb6ffffffb3ffffffb0ffffffadffffffaaffffffa7ffffffa4ffffffa1ffffff9effffff9bffffff98ffffff95ffffff92ffffff8fffffff8cffffff89ffffff86ffffff83ffffff80ffffff7dffffff7affffff77ffffff74ffffff71ffffff6effffff6bffffff68ffffff65ffffff62ffffff5fffffff5cffffff59ffffff56ffffff53ffffff50ffffff4dffffff4affffff47ffffff44ffffff41ffffff3effffff3bffffff38ffffff35ffffff32ffffff2fffffff0000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000802a0300002d0300002f030000320300003403000037030000390300003c0300003e030000410300004303000046030000480300004b0300004d030000500300005203000055030000570300005a0300005c0300005f030000610300006403000066030000690300006b0300006e030000700300007303000075030000780300007a0300007d0300007f030000820300008403000087030000890300008c0300008e030000910300009303000096030000980300009b0300009d030000a0030000a2030000a5030000a7030000aa030000ac030000af030000b1030000b4030000b6030000b9030000bb030000be030000000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080000000800000008000000080" + } + ], + "tilesGet": { + "body": "a16468617665a16a356d2f312f3431343736683938366531386364", + "envelope": "a36162581ba16468617665a16a356d2f312f343134373668393836653138636461746974696c65732e6765746373696740" + }, + "tilesEnd": { + "body": "a1616e03", + "envelope": "a3616244a1616e0361746974696c65732e656e646373696740" + }, + "notes": { + "forged": "The last append with its data changed after signing: it must not verify, and is never held or laid.", + "have": "The tiles.get have that holds the closed tile: the service answers it with the pieces alone.", + "laid": "What Lay makes of the named tiles, in that order. A trailing tile stays partial; its etag is that of the data as laid.", + "tiles": "One tile of each shape, in the order the box sends them, signed by boxKey. Each passes Tiles.Check." + }, + "have": { + "answer": [ + "whole", + "append 17:00", + "append 17:05" + ] + } +} diff --git a/go/protocol/cloudwire/tiles.go b/go/protocol/cloudwire/tiles.go new file mode 100644 index 000000000..7fc0d77e4 --- /dev/null +++ b/go/protocol/cloudwire/tiles.go @@ -0,0 +1,236 @@ +package cloudwire + +import ( + "errors" + "fmt" + "slices" + "sort" + + "github.com/srcfl/ftw/go/protocol" +) + +// Tiles: the last 48 hours of the box's 5-minute history, so a phone can draw +// it while the box cannot answer (docs/protocol-cloud.md, tiles). +// +// A tiles body is the app protocol's hist.chunk with the common header. It +// comes in three shapes, all at 5 minutes and stride 1: +// +// - closed: partial false, startMs the tile's start, every point. The same +// tile id, etag and bytes as the box serves over Noise for TileSeries. +// - whole: partial true, startMs the tile's start, every point, missing +// from the first bucket that has not closed. +// - append: partial true, startMs a later bucket, only the buckets that +// closed since the last piece of that tile. +// +// A whole tile and an append are pieces of the trailing tile. Lay puts the +// pieces a receiver holds back together. + +// MsgTilesGet asks the service for the tiles a phone lacks, as TilesGet. The +// service answers with the box's tiles envelopes, unchanged, then MsgTilesEnd. +const MsgTilesGet = "tiles.get" + +// MsgTilesEnd closes the service's answer to tiles.get. It is the service's +// own and says only how many envelopes went before it. +const MsgTilesEnd = "tiles.end" + +// The geometry FTW Cloud keeps. +const ( + // TileRes is the one resolution the box sends. + TileRes = protocol.ResN5m + // TileKeepMs is how far back tiles are kept: a tile that ended earlier + // than this before now is gone. + TileKeepMs = 48 * 3_600_000 + // TileSettleMs is how long after a bucket ends the box counts it closed. + // The history writer commits every 15 s. + TileSettleMs = 30_000 + // TileAppendsMax is how many appends follow a whole piece of the + // trailing tile before the box sends it whole again: once an hour. + TileAppendsMax = 11 + // TilePiecesMax is how many pieces of the trailing tile the service + // holds. A whole piece replaces the ones before it. + TilePiecesMax = 16 + // TilesClosedMax is how many closed tiles 48 hours can touch. + TilesClosedMax = 5 + // TileEnvelopeMax bounds one tiles envelope. A tile of the four series + // is about 2.6 KB, so FTW Cloud drops anything larger, padding with keys + // it ignores included. + TileEnvelopeMax = 4 << 10 +) + +// TileSeries are the columns the box sends, in this order, and the only ones +// a tile may carry. They are what the app asks for over Noise, so a closed +// tile is the same bytes either way. +var TileSeries = []string{ + protocol.FieldNames[protocol.FidGridW], + protocol.FieldNames[protocol.FidPvW], + protocol.FieldNames[protocol.FidBatteryW], + protocol.FieldNames[protocol.FidLoadW], +} + +// Tiles is one tile, or one piece of the trailing tile. +type Tiles struct { + Header + TileID string `cbor:"tileId"` + Etag string `cbor:"etag"` + Res protocol.Resolution `cbor:"res"` + StartMs int64 `cbor:"startMs"` + StepMs int64 `cbor:"stepMs"` + Series []string `cbor:"series"` + // Data is int32 LE, one block per series, protocol.MissingSample where + // the box has no record. + Data []byte `cbor:"data"` + Partial bool `cbor:"partial"` +} + +// TilesGet asks for tiles. Have maps tile id to etag for the closed tiles the +// phone holds; the service sends every closed tile it holds that is not +// there with that etag, and every piece of the trailing tile. +type TilesGet struct { + Have map[string]string `cbor:"have"` +} + +// TilesEnd closes the answer to tiles.get. +type TilesEnd struct { + // N is how many tiles envelopes went before it. + N int `cbor:"n"` +} + +// Chunk is the tile as the app protocol carries it, without the header. +func (t Tiles) Chunk() protocol.HistChunk { + return protocol.HistChunk{TileID: t.TileID, Etag: t.Etag, Res: t.Res, StartMs: t.StartMs, + StepMs: t.StepMs, Series: slices.Clone(t.Series), Data: slices.Clone(t.Data), Partial: t.Partial} +} + +// TileStart is the start of the tile this body belongs to. +func (t Tiles) TileStart() int64 { return protocol.TileStartFor(t.Res, t.StartMs) } + +// TileEnd is where that tile ends. +func (t Tiles) TileEnd() int64 { return t.TileStart() + protocol.ResolutionTileSpanMs[t.Res] } + +// Points is how many points each series carries. +func (t Tiles) Points() int { + if len(t.Series) == 0 { + return 0 + } + return len(t.Data) / 4 / len(t.Series) +} + +// Whole says the body starts at its tile's start: a closed tile, or a whole +// piece that replaces every piece of that tile before it. +func (t Tiles) Whole() bool { return t.StartMs == t.TileStart() } + +// ErrTile is a tiles body that breaks the shapes above. +var ErrTile = errors.New("cloudwire: not a tile") + +// Check says whether t is one of the three shapes. The service holds no +// other; the phone lays no other. +func (t Tiles) Check() error { + bad := func(why string) error { return fmt.Errorf("%w: %s", ErrTile, why) } + if t.Res != TileRes || t.StepMs != protocol.ResolutionStepMs[TileRes] { + return bad("not 5-minute steps") + } + if t.StartMs < 0 || t.StartMs%t.StepMs != 0 { + return bad("start off the 5-minute grid") + } + if t.TileID != protocol.TileIDFor(t.Res, 1, t.TileStart()) { + return bad("tile id does not name the tile its start is in") + } + if !slices.Equal(t.Series, TileSeries) { + return bad("series are not TileSeries") + } + n := t.Points() + if n < 1 || len(t.Data) != n*4*len(t.Series) { + return bad("data is not whole columns") + } + first := int((t.StartMs - t.TileStart()) / t.StepMs) + if first+n > int(protocol.PointsPerTile(t.Res)) { + return bad("data runs past the tile") + } + if !t.Partial && (!t.Whole() || n != int(protocol.PointsPerTile(t.Res))) { + return bad("a closed tile that is not the whole tile") + } + if t.Etag != protocol.EtagOf(t.Data) { + return bad("etag") + } + return nil +} + +// Piece cuts points [from, to) of each series out of a packed tile, as the +// box sends an append or a whole piece. from 0 starts at the tile's start. +// The header is the caller's. +func Piece(c protocol.HistChunk, from, to int) Tiles { + per := 0 + if len(c.Series) > 0 { + per = len(c.Data) / 4 / len(c.Series) + } + to = min(to, per) + from = min(max(from, 0), to) + data := make([]byte, 0, len(c.Series)*(to-from)*4) + for s := range c.Series { + base := s * per * 4 + data = append(data, c.Data[base+from*4:base+to*4]...) + } + return Tiles{TileID: c.TileID, Etag: protocol.EtagOf(data), Res: c.Res, + StartMs: c.StartMs + int64(from)*c.StepMs, StepMs: c.StepMs, + Series: slices.Clone(c.Series), Data: data, Partial: true} +} + +// Lay puts tiles back together, in the order the box sent them, by wallMs +// and then seq, which starts again with the box: a closed tile as it is; the +// trailing tile from its newest whole piece, with each later append laid +// over it; a later closed tile replaces an earlier one. A piece of a tile +// already closed changes nothing, and neither does a body that fails Check. The result has one chunk per tile, oldest first, +// each with every point; a trailing tile stays partial, its etag that of the +// data as laid, never one to offer as held. +func Lay(bodies []Tiles) []protocol.HistChunk { + byID := map[string]*protocol.HistChunk{} + for _, t := range bodies { + if t.Check() != nil { + continue + } + held := byID[t.TileID] + if held != nil && !held.Partial && t.Partial { + // A piece of a tile already closed. + continue + } + if !t.Partial { + // A closed tile, or a later one for the same tile: the box sends + // a closed tile again when the ledger changed under it. + c := t.Chunk() + byID[t.TileID] = &c + continue + } + per := int(protocol.PointsPerTile(t.Res)) + if held == nil || t.Whole() || !slices.Equal(held.Series, t.Series) { + c := protocol.HistChunk{TileID: t.TileID, Res: t.Res, StartMs: t.TileStart(), StepMs: t.StepMs, + Series: slices.Clone(t.Series), Data: missing(len(t.Series) * per), Partial: true} + held = &c + byID[t.TileID] = held + } + n := t.Points() + at := int((t.StartMs - t.TileStart()) / t.StepMs) + for s := range t.Series { + copy(held.Data[(s*per+at)*4:(s*per+at+n)*4], t.Data[s*n*4:(s+1)*n*4]) + } + } + out := make([]protocol.HistChunk, 0, len(byID)) + for _, c := range byID { + if c.Partial { + c.Etag = protocol.EtagOf(c.Data) + } + out = append(out, *c) + } + sort.Slice(out, func(a, b int) bool { return out[a].StartMs < out[b].StartMs }) + return out +} + +// missing is n samples of protocol.MissingSample. +func missing(n int) []byte { + b := make([]byte, n*4) + m := int32(protocol.MissingSample) + u := uint32(m) + for i := 0; i < n; i++ { + b[i*4], b[i*4+1], b[i*4+2], b[i*4+3] = byte(u), byte(u>>8), byte(u>>16), byte(u>>24) + } + return b +} diff --git a/go/protocol/cloudwire/tiles_test.go b/go/protocol/cloudwire/tiles_test.go new file mode 100644 index 000000000..23621ce23 --- /dev/null +++ b/go/protocol/cloudwire/tiles_test.go @@ -0,0 +1,423 @@ +package cloudwire + +import ( + "bytes" + "crypto/rand" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "encoding/json" + "errors" + "math" + "os" + "slices" + "testing" + + "github.com/srcfl/ftw/go/protocol" +) + +// testdata/cloud-tile-vectors.json freezes one tile of each shape, signed by +// the vector box key of cloud-vectors.json, how three pieces lay back +// together, and the bodies of tiles.get and tiles.end. srcfl/ftw-cloud checks +// and holds them; the phone verifies, decodes and lays them. +// +// go test ./cloudwire -run TestTileVectors -update + +const tileVectorsPath = "testdata/cloud-tile-vectors.json" + +// The vectors' tiles: the tile before 2026-10-12 12:00 UTC, closed, and the +// one after it, still filling at 17:15. +const ( + vectorTileStart = int64(1_791_806_400_000) // 2026-10-12 12:00 UTC + vectorTilePrev = vectorTileStart - 43_200_000 +) + +// vectorWindows is the ledger the vectors pack from: a value for every +// bucket of both tiles before 17:15, but none for the house between 03:00 +// and 03:30, and no battery for the first hour of the afternoon. +func vectorWindows() map[string][]protocol.HistorySample { + out := map[string][]protocol.HistorySample{} + for at := vectorTilePrev; at < vectorTileStart+5*3_600_000+15*60_000; at += 300_000 { + i := float64((at - vectorTilePrev) / 300_000) + out["grid_w"] = append(out["grid_w"], protocol.HistorySample{StartMs: at, W: 800 - 15*i}) + out["pv_w"] = append(out["pv_w"], protocol.HistorySample{StartMs: at, W: -math.Max(0, 3000*math.Sin(i/40))}) + if at < vectorTileStart || at >= vectorTileStart+3_600_000 { + out["battery_w"] = append(out["battery_w"], protocol.HistorySample{StartMs: at, W: 400 - 3*i}) + } + if at < vectorTilePrev+3*3_600_000 || at >= vectorTilePrev+3*3_600_000+30*60_000 { + out["load_w"] = append(out["load_w"], protocol.HistorySample{StartMs: at, W: 450 + 2.5*i}) + } + } + return out +} + +// vectorPack packs a tile as the box does: buckets from closedThrough on are +// not there yet. +func vectorPack(start, closedThrough int64) protocol.HistChunk { + windows := map[string][]protocol.HistorySample{} + for name, samples := range vectorWindows() { + for _, s := range samples { + if s.StartMs < closedThrough { + windows[name] = append(windows[name], s) + } + } + } + return protocol.PackTile(TileRes, 1, start, TileSeries, windows, closedThrough) +} + +func header(seq uint64, wall int64) Header { + return Header{Seq: seq, Boot: bytes.Repeat([]byte{0xb0}, BootBytes), UptimeMs: 60_000 + int64(seq)*10, WallMs: i64(wall)} +} + +// The vectors' bodies, in the order the box sends them. +func vectorTiles() []struct { + name string + body Tiles +} { + closed := vectorPack(vectorTilePrev, vectorTileStart+30_000) + whole := Piece(vectorPack(vectorTileStart, vectorTileStart+5*3_600_000), 0, 144) + first := Piece(vectorPack(vectorTileStart, vectorTileStart+5*3_600_000+5*60_000), 60, 61) + second := Piece(vectorPack(vectorTileStart, vectorTileStart+5*3_600_000+15*60_000), 61, 63) + closedBody := Tiles{TileID: closed.TileID, Etag: closed.Etag, Res: closed.Res, StartMs: closed.StartMs, + StepMs: closed.StepMs, Series: closed.Series, Data: closed.Data, Partial: closed.Partial} + + wall := vectorTileStart + 5*3_600_000 + closedBody.Header = header(20, vectorTileStart+30_000) + whole.Header = header(21, wall+30_000) + first.Header = header(22, wall+5*60_000+30_000) + second.Header = header(23, wall+15*60_000+30_000) + return []struct { + name string + body Tiles + }{{"closed", closedBody}, {"whole", whole}, {"append 17:00", first}, {"append 17:05", second}} +} + +func TestTheThreeShapesCheck(t *testing.T) { + for _, v := range vectorTiles() { + if err := v.body.Check(); err != nil { + t.Errorf("%s: %v", v.name, err) + } + } + closed := vectorTiles()[0].body + if closed.Partial || !closed.Whole() || closed.Points() != 144 { + t.Fatalf("closed tile: partial %v whole %v points %d", closed.Partial, closed.Whole(), closed.Points()) + } + if a := vectorTiles()[3].body; a.Whole() || a.Points() != 2 || a.StartMs != vectorTileStart+5*3_600_000+5*60_000 { + t.Fatalf("append: whole %v points %d start %d", a.Whole(), a.Points(), a.StartMs) + } +} + +// A closed tile is what the box serves over Noise for the same series: the +// same id, etag and bytes. +// The series are the registry's names, and a whole tile fits the bound +// FTW Cloud holds a tiles envelope to. +func TestTileSeriesAreTheRegistrysAndATileFitsItsBound(t *testing.T) { + if !slices.Equal(TileSeries, []string{"grid_w", "pv_w", "battery_w", "load_w"}) { + t.Fatalf("TileSeries %v", TileSeries) + } + b := vectorTiles()[0].body + env, err := Seal(nil, "home", MsgTiles, b) + if err != nil { + t.Fatal(err) + } + raw, _ := env.Encode() + if len(raw)+64+6 > TileEnvelopeMax { + t.Fatalf("a closed tile with its signature is %d bytes", len(raw)+64) + } +} + +func TestAClosedTileIsTheNoiseTile(t *testing.T) { + noise := protocol.PackTile(TileRes, 1, vectorTilePrev, TileSeries, vectorWindows(), vectorTileStart+3_600_000) + got := vectorTiles()[0].body.Chunk() + if got.TileID != noise.TileID || got.Etag != noise.Etag || !bytes.Equal(got.Data, noise.Data) || got.Partial != noise.Partial { + t.Fatalf("closed tile %s %s differs from Noise's %s %s", got.TileID, got.Etag, noise.TileID, noise.Etag) + } +} + +func TestCheckRefusesWhatIsNotATile(t *testing.T) { + good := vectorTiles()[2].body + cases := map[string]func(*Tiles){ + "etag": func(b *Tiles) { b.Etag = "00000000" }, + "another tile's id": func(b *Tiles) { b.TileID = protocol.TileIDFor(TileRes, 1, vectorTilePrev) }, + "stride 2 id": func(b *Tiles) { b.TileID = protocol.TileIDFor(TileRes, 2, vectorTileStart) }, + "hourly": func(b *Tiles) { b.Res = protocol.ResN1h }, + "step": func(b *Tiles) { b.StepMs = 600_000 }, + "off the grid": func(b *Tiles) { b.StartMs++ }, + "half a column": func(b *Tiles) { + b.Data = b.Data[:len(b.Data)-2] + b.Etag = protocol.EtagOf(b.Data) + }, + "no series": func(b *Tiles) { b.Series = nil }, + "another series": func(b *Tiles) { + b.Series = []string{"grid_w", "pv_w", "battery_w", "ev_w"} + }, + "one series less": func(b *Tiles) { + b.Series = b.Series[:3] + b.Data = b.Data[:3*b.Points()*4] + b.Etag = protocol.EtagOf(b.Data) + }, + "twice the same series": func(b *Tiles) { + b.Series = []string{"grid_w", "grid_w", "pv_w", "load_w"} + }, + "past the tile": func(b *Tiles) { + b.StartMs = vectorTileStart + 143*300_000 + b.Data = append(b.Data, b.Data...) + b.Etag = protocol.EtagOf(b.Data) + }, + "closed but a piece": func(b *Tiles) { b.Partial = false }, + } + for name, mutate := range cases { + b := good + b.Series = slices.Clone(good.Series) + b.Data = slices.Clone(good.Data) + mutate(&b) + if err := b.Check(); !errors.Is(err, ErrTile) { + t.Errorf("%s: Check() = %v, want ErrTile", name, err) + } + } +} + +func sample(c protocol.HistChunk, series string, point int) int32 { + s := slices.Index(c.Series, series) + per := len(c.Data) / 4 / len(c.Series) + return int32(binary.LittleEndian.Uint32(c.Data[(s*per+point)*4:])) +} + +// The whole piece and two appends lay back into the tile the box would pack +// at 17:15, less the bucket that has not closed. +func TestLayPutsThePiecesBackTogether(t *testing.T) { + v := vectorTiles() + laid := Lay([]Tiles{v[0].body, v[1].body, v[2].body, v[3].body}) + if len(laid) != 2 || laid[0].TileID != v[0].body.TileID || laid[1].TileID != v[1].body.TileID { + t.Fatalf("laid %d tiles", len(laid)) + } + want := vectorPack(vectorTileStart, vectorTileStart+5*3_600_000+15*60_000) + got := laid[1] + if !got.Partial || got.StartMs != vectorTileStart || !bytes.Equal(got.Data, want.Data) || got.Etag != protocol.EtagOf(want.Data) { + t.Fatalf("trailing tile does not match the box's own pack") + } + if sample(got, "battery_w", 3) != protocol.MissingSample || sample(got, "grid_w", 63) != protocol.MissingSample { + t.Fatal("a hole, or a bucket not closed yet, reads as a value") + } + if laid[0].Partial || laid[0].Etag != v[0].body.Etag { + t.Fatal("the closed tile changed") + } +} + +func TestLayKeepsTheNewestWholeAndTheClosedTile(t *testing.T) { + v := vectorTiles() + // An append, then a whole piece sent later: the whole one wins, and so + // the append before it is gone. + laid := Lay([]Tiles{v[2].body, v[1].body}) + if sample(laid[0], "grid_w", 60) != protocol.MissingSample { + t.Fatal("an append before a whole piece survived it") + } + // A piece of a tile already closed changes nothing. + piece := Piece(vectorPack(vectorTilePrev, vectorTilePrev+3_600_000), 0, 144) + laid = Lay([]Tiles{v[0].body, piece}) + if laid[0].Partial || laid[0].Etag != v[0].body.Etag { + t.Fatal("a piece reopened a closed tile") + } + // A closed tile sent again after the ledger changed replaces the first. + fixed := v[0].body + fixed.Data = slices.Clone(fixed.Data) + fixed.Data[0] ^= 1 + fixed.Etag = protocol.EtagOf(fixed.Data) + laid = Lay([]Tiles{v[0].body, fixed, piece}) + if laid[0].Partial || laid[0].Etag != fixed.Etag { + t.Fatal("a later closed tile did not replace the first") + } + // A body that fails Check is skipped. + bad := v[3].body + bad.Etag = "00000000" + laid = Lay([]Tiles{v[1].body, v[2].body, bad}) + if sample(laid[0], "grid_w", 61) != protocol.MissingSample { + t.Fatal("a body with a wrong etag was laid") + } +} + +type tileVectorFile struct { + BoxKey vectorKey `json:"boxKey"` + Series []string `json:"series"` + Tiles []tileVector `json:"tiles"` + Forged tileVector `json:"forged"` + Laid []tileVectorLaid `json:"laid"` + Get tileVectorExchange `json:"tilesGet"` + End tileVectorExchange `json:"tilesEnd"` + Notes map[string]string `json:"notes"` + Have map[string][]string `json:"have"` +} + +type tileVector struct { + Name string `json:"name"` + TileID string `json:"tileId"` + Etag string `json:"etag"` + StartMs int64 `json:"startMs"` + Points int `json:"points"` + Partial bool `json:"partial"` + Seq uint64 `json:"seq"` + Body string `json:"body"` + SignedInputSHA256 string `json:"signedInputSha256"` + Sig string `json:"sig"` + Envelope string `json:"envelope"` + Frame string `json:"frame"` +} + +type tileVectorLaid struct { + Of []string `json:"of"` + TileID string `json:"tileId"` + Etag string `json:"etag"` + StartMs int64 `json:"startMs"` + Partial bool `json:"partial"` + Data string `json:"data"` +} + +type tileVectorExchange struct { + Body string `json:"body"` + Envelope string `json:"envelope"` +} + +func TestTileVectors(t *testing.T) { + box := testKey(t, "ftw.cloud vector box") + home, boxPK := homeOf(t, box) + + var stored tileVectorFile + raw, err := os.ReadFile(tileVectorsPath) + switch { + case err == nil: + if err := json.Unmarshal(raw, &stored); err != nil { + t.Fatal(err) + } + case !*update: + t.Fatalf("%v (run with -update to create it)", err) + } + + out := tileVectorFile{ + BoxKey: vectorKey{D: hex.EncodeToString(box.D.FillBytes(make([]byte, 32))), PK: hex.EncodeToString(boxPK), Home: home}, + Series: TileSeries, + Notes: map[string]string{ + "tiles": "One tile of each shape, in the order the box sends them, signed by boxKey. Each passes Tiles.Check.", + "forged": "The last append with its data changed after signing: it must not verify, and is never held or laid.", + "laid": "What Lay makes of the named tiles, in that order. A trailing tile stays partial; its etag is that of the data as laid.", + "have": "The tiles.get have that holds the closed tile: the service answers it with the pieces alone.", + }, + } + signed := func(i int, name string, b Tiles) tileVector { + body, err := Marshal(b) + if err != nil { + t.Fatal(err) + } + input, err := SignedInput(home, MsgTiles, body) + if err != nil { + t.Fatal(err) + } + digest := sha256.Sum256(input) + v := tileVector{Name: name, TileID: b.TileID, Etag: b.Etag, StartMs: b.StartMs, Points: b.Points(), Partial: b.Partial, + Seq: b.Seq, Body: hex.EncodeToString(body), SignedInputSHA256: hex.EncodeToString(digest[:])} + var sig []byte + if i < len(stored.Tiles) && stored.Tiles[i].Name == name && stored.Tiles[i].SignedInputSHA256 == v.SignedInputSHA256 { + sig, _ = hex.DecodeString(stored.Tiles[i].Sig) + } else if *update { + if sig, err = Sign(rand.Reader, box, home, MsgTiles, body); err != nil { + t.Fatal(err) + } + } else { + t.Fatalf("tile vector %d is not %q as stored; regenerate with -update", i, name) + } + env := Envelope{T: MsgTiles, B: body, Sig: sig} + if err := Verify(&box.PublicKey, home, env); err != nil { + t.Fatalf("%s: stored signature does not verify: %v", name, err) + } + v.Sig = hex.EncodeToString(sig) + envBytes, err := env.Encode() + if err != nil { + t.Fatal(err) + } + frame, err := EncodeFrame(envBytes) + if err != nil { + t.Fatal(err) + } + v.Envelope, v.Frame = hex.EncodeToString(envBytes), hex.EncodeToString(frame) + return v + } + + tiles := vectorTiles() + var bodies []Tiles + for i, v := range tiles { + out.Tiles = append(out.Tiles, signed(i, v.name, v.body)) + bodies = append(bodies, v.body) + } + + // The forged tile: the last append's signature over other data. + last := out.Tiles[len(out.Tiles)-1] + forged := tiles[len(tiles)-1].body + forged.Data = slices.Clone(forged.Data) + forged.Data[0] ^= 1 + forged.Etag = protocol.EtagOf(forged.Data) + fb, err := Marshal(forged) + if err != nil { + t.Fatal(err) + } + sig, _ := hex.DecodeString(last.Sig) + fenv := Envelope{T: MsgTiles, B: fb, Sig: sig} + if Verify(&box.PublicKey, home, fenv) == nil { + t.Fatal("the forged tile verifies") + } + fraw, err := fenv.Encode() + if err != nil { + t.Fatal(err) + } + fframe, err := EncodeFrame(fraw) + if err != nil { + t.Fatal(err) + } + out.Forged = tileVector{Name: "forged append", TileID: forged.TileID, Etag: forged.Etag, StartMs: forged.StartMs, + Points: forged.Points(), Partial: true, Seq: forged.Seq, Body: hex.EncodeToString(fb), Sig: last.Sig, + Envelope: hex.EncodeToString(fraw), Frame: hex.EncodeToString(fframe)} + + for _, of := range [][]int{{0}, {1}, {1, 2, 3}, {0, 1, 2, 3}, {2, 1}} { + var in []Tiles + var names []string + for _, i := range of { + in = append(in, bodies[i]) + names = append(names, tiles[i].name) + } + for _, c := range Lay(in) { + out.Laid = append(out.Laid, tileVectorLaid{Of: names, TileID: c.TileID, Etag: c.Etag, StartMs: c.StartMs, + Partial: c.Partial, Data: hex.EncodeToString(c.Data)}) + } + } + + have := map[string]string{bodies[0].TileID: bodies[0].Etag} + out.Have = map[string][]string{"answer": {"whole", "append 17:00", "append 17:05"}} + for _, ex := range []struct { + dst *tileVectorExchange + t string + body any + }{{&out.Get, MsgTilesGet, TilesGet{Have: have}}, {&out.End, MsgTilesEnd, TilesEnd{N: 3}}} { + body, err := Marshal(ex.body) + if err != nil { + t.Fatal(err) + } + env, err := Unsigned(ex.t, ex.body) + if err != nil { + t.Fatal(err) + } + *ex.dst = tileVectorExchange{Body: hex.EncodeToString(body), Envelope: hex.EncodeToString(env)} + } + + got, err := json.MarshalIndent(out, "", " ") + if err != nil { + t.Fatal(err) + } + got = append(got, '\n') + if *update { + if err := os.WriteFile(tileVectorsPath, got, 0o644); err != nil { + t.Fatal(err) + } + return + } + if !bytes.Equal(got, raw) { + t.Fatalf("%s is out of date; regenerate with -update", tileVectorsPath) + } +} diff --git a/go/protocol/cloudwire/tunnel_test.go b/go/protocol/cloudwire/tunnel_test.go index d817c817d..9b23313bc 100644 --- a/go/protocol/cloudwire/tunnel_test.go +++ b/go/protocol/cloudwire/tunnel_test.go @@ -37,9 +37,10 @@ func TestTheLargestTunFitsItsFrame(t *testing.T) { } } -// The downlink grows by tun alone. -func TestTheDownlinkIsSixTypes(t *testing.T) { - want := []string{MsgChallenge, MsgHelloOK, MsgWatch, MsgResync, MsgPurged, MsgTun} +// The downlink grows by tun and x alone: the two carry ciphertext the box +// judges itself. +func TestTheDownlinkIsSevenTypes(t *testing.T) { + want := []string{MsgChallenge, MsgHelloOK, MsgWatch, MsgResync, MsgPurged, MsgTun, MsgX} if !slices.Equal(Downlink, want) { t.Fatalf("Downlink is %v, want %v", Downlink, want) } diff --git a/go/protocol/energywire/energy.go b/go/protocol/energywire/energy.go new file mode 100644 index 000000000..08c95fd89 --- /dev/null +++ b/go/protocol/energywire/energy.go @@ -0,0 +1,105 @@ +// Package energywire names the anonymous daily energy report: one closed UTC +// day of one home as six whole-kWh sums, and nothing that says whose. +// +// It travels alone through a relay (package reportwire), so the gateway never +// sees the box's address. It carries no id, version, driver, zone or size, +// because the daily fleet report (package fleetwire) carries those, and the +// two must never be joined into one home's record. +package energywire + +import ( + "bytes" + "encoding/json" + "errors" + "time" +) + +const Schema = "ftw.energy/1" + +const ( + // MaxBodyBytes bounds the report itself, before any parsing. + MaxBodyBytes = 256 + + // MaxKWh bounds one field of one day. A home on a 3 × 63 A fuse moves + // at most about 1,000 kWh a day. + MaxKWh = 1000 + + // AcceptDays is how many closed days back a report may name. A box sends + // 1 to 12 hours after the day closes, so one day would do; three leave + // room for a slow clock. + AcceptDays = 3 + + // MaxReports caps one date's reports, so junk can add bounded noise and + // no more. + MaxReports = 100_000 +) + +// Fields are the report's sums, in the order the public view lists them. +var Fields = []string{"pv_generation_kwh", "grid_import_kwh", "grid_export_kwh", "battery_charge_kwh", "battery_discharge_kwh", "vehicle_charge_kwh"} + +// Report is one home's closed UTC day. Never add an identifier. +type Report struct { + Schema string `json:"schema"` + Date string `json:"date"` + PVGeneration int64 `json:"pv_generation_kwh"` + GridImport int64 `json:"grid_import_kwh"` + GridExport int64 `json:"grid_export_kwh"` + BatteryCharge int64 `json:"battery_charge_kwh"` + BatteryDischarge int64 `json:"battery_discharge_kwh"` + VehicleCharge int64 `json:"vehicle_charge_kwh"` +} + +// Sums are the report's fields in Fields order. +func (r Report) Sums() []int64 { + return []int64{r.PVGeneration, r.GridImport, r.GridExport, r.BatteryCharge, r.BatteryDischarge, r.VehicleCharge} +} + +// Valid checks the fields a sender fills. The date is checked against the +// receiver's clock in Decode. +func (r Report) Valid() bool { + if r.Schema != Schema || !isDate(r.Date) { + return false + } + for _, v := range r.Sums() { + if v < 0 || v > MaxKWh { + return false + } + } + return true +} + +var errBad = errors.New("energy: invalid report") + +// Decode accepts only the bytes a box writes: the eight fields in their +// order, whole numbers in range, and a date in the last AcceptDays closed UTC +// days. Another order, spacing, a repeated or extra field, or a number +// written another way is refused. +func Decode(data []byte, now time.Time) (Report, error) { + var r Report + if len(data) > MaxBodyBytes || json.Unmarshal(data, &r) != nil || !r.Valid() || !Accepts(r.Date, now) { + return Report{}, errBad + } + if canonical, err := json.Marshal(r); err != nil || !bytes.Equal(canonical, data) { + return Report{}, errBad + } + return r, nil +} + +// Accepts reports whether date is one of the last AcceptDays closed UTC days. +func Accepts(date string, now time.Time) bool { + day, err := time.Parse(time.DateOnly, date) + today := now.UTC().Truncate(24 * time.Hour) + return err == nil && day.Before(today) && !day.Before(today.AddDate(0, 0, -AcceptDays)) +} + +// Closed reports whether no report may name date any more, so its totals are +// final. +func Closed(date string, now time.Time) bool { + day, err := time.Parse(time.DateOnly, date) + return err == nil && day.Before(now.UTC().Truncate(24*time.Hour).AddDate(0, 0, -AcceptDays)) +} + +func isDate(s string) bool { + t, err := time.Parse(time.DateOnly, s) + return err == nil && t.Format(time.DateOnly) == s +} diff --git a/go/protocol/energywire/energy_test.go b/go/protocol/energywire/energy_test.go new file mode 100644 index 000000000..0b6222ab4 --- /dev/null +++ b/go/protocol/energywire/energy_test.go @@ -0,0 +1,71 @@ +package energywire + +import ( + "encoding/json" + "strings" + "testing" + "time" +) + +var now = time.Date(2026, 10, 24, 9, 30, 0, 0, time.UTC) + +const valid = `{"schema":"ftw.energy/1","date":"2026-10-23","pv_generation_kwh":12,"grid_import_kwh":30,"grid_export_kwh":4,"battery_charge_kwh":9,"battery_discharge_kwh":8,"vehicle_charge_kwh":0}` + +func TestOnlySixSumsAndADateAreAccepted(t *testing.T) { + r, err := Decode([]byte(valid), now) + if err != nil || r.PVGeneration != 12 || r.GridImport != 30 || r.Date != "2026-10-23" { + t.Fatalf("%+v %v", r, err) + } + for _, bad := range []string{ + strings.Replace(valid, `"ftw.energy/1"`, `"ftw.energy/2"`, 1), + strings.Replace(valid, `2026-10-23`, `2026-10-24`, 1), // today is still open + strings.Replace(valid, `2026-10-23`, `2026-10-20`, 1), // four days back + strings.Replace(valid, `2026-10-23`, `2026-10-32`, 1), + strings.Replace(valid, `2026-10-23`, `2026-1-23`, 1), + strings.Replace(valid, `:12,`, `:12.5,`, 1), + strings.Replace(valid, `:12,`, `:1e1,`, 1), + strings.Replace(valid, `:12,`, `:-1,`, 1), + strings.Replace(valid, `:12,`, `:1001,`, 1), + strings.Replace(valid, `:12,`, `:"12",`, 1), + strings.Replace(valid, `,"vehicle_charge_kwh":0`, ``, 1), + strings.TrimSuffix(valid, "}") + `,"home":"a"}`, + strings.TrimSuffix(valid, "}") + `,"ftw_version":"v1.0.0"}`, + strings.Replace(valid, `"vehicle_charge_kwh":0`, `"vehicle_charge_kwh":0,"vehicle_charge_kwh":0`, 1), + valid + valid, valid + ` {}`, `[]`, `null`, strings.Repeat(" ", MaxBodyBytes) + valid, + } { + if _, err := Decode([]byte(bad), now); err == nil { + t.Fatalf("accepted %s", bad) + } + } +} + +func TestTheReportHasNoFieldThatNamesAHome(t *testing.T) { + var fields map[string]any + b, _ := json.Marshal(Report{}) + if err := json.Unmarshal(b, &fields); err != nil { + t.Fatal(err) + } + if len(fields) != 8 || len(Fields) != 6 { + t.Fatalf("fields %v", fields) + } + for _, name := range append([]string{"schema", "date"}, Fields...) { + if _, ok := fields[name]; !ok { + t.Fatalf("missing %s", name) + } + } +} + +func TestOpenAndClosedDays(t *testing.T) { + for date, want := range map[string][2]bool{ + "2026-10-24": {false, false}, + "2026-10-23": {true, false}, + "2026-10-21": {true, false}, + "2026-10-20": {false, true}, + "2026-09-01": {false, true}, + "2026-10-25": {false, false}, + } { + if got := [2]bool{Accepts(date, now), Closed(date, now)}; got != want { + t.Fatalf("%s: accepts, closed = %v, want %v", date, got, want) + } + } +} diff --git a/go/protocol/fleetwire/fleet.go b/go/protocol/fleetwire/fleet.go index 216be8fce..9c5e3fe82 100644 --- a/go/protocol/fleetwire/fleet.go +++ b/go/protocol/fleetwire/fleet.go @@ -10,7 +10,10 @@ import ( "strings" ) -const Schema = "ftw.fleet/1" +// Schema is the report a box sends: the shape of the box and how it ran +// lately. SchemaV1, the shape alone, is still accepted from older boxes. +const Schema = "ftw.fleet/2" +const SchemaV1 = "ftw.fleet/1" const StatsSchema = "ftw.fleet-stats/1" const Meaning = "reports, not unique boxes" const MaxBodyBytes = 4096 @@ -27,13 +30,30 @@ type Report struct { Battery string `json:"battery_kwh"` Zone string `json:"price_zone"` Age string `json:"install_age"` + // How the box ran lately, in Schema only: the slowest plan of the last + // 24 hours, how many plans fell back, the share of driver polls that + // failed, and the newest update of the last 7 days. + Solve string `json:"plan_solve,omitempty"` + Fallbacks string `json:"plan_fallbacks,omitempty"` + Errors string `json:"driver_errors,omitempty"` + Update string `json:"update,omitempty"` } +// The health buckets, each a few values wide. +var ( + SolveBuckets = []string{"none", "0-1s", "1-5s", "5s+"} + FallbackBuckets = []string{"none", "0", "1-3", "4+"} + ErrorBuckets = []string{"none", "0", "0-1%", "1%+"} + UpdateBuckets = []string{"none", "ok", "failed"} +) + var release = regexp.MustCompile(`^v[0-9]{1,4}\.[0-9]{1,4}\.[0-9]{1,4}(-beta\.[0-9]{1,4})?$`) var driver = regexp.MustCompile(`^[a-z0-9_]{1,64}$`) var zone = regexp.MustCompile(`^[A-Z0-9-]{2,16}$`) // Decode refuses extra fields, so a sender cannot add a household record. +// A SchemaV1 report has the seven shape fields; a Schema report has the +// four health fields too. func Decode(data []byte) (Report, error) { bad := errors.New("fleet: invalid report") var r Report @@ -41,12 +61,19 @@ func Decode(data []byte) (Report, error) { return r, bad } var fields map[string]json.RawMessage - if json.Unmarshal(data, &fields) != nil || len(fields) != 7 || bytes.Equal(bytes.TrimSpace(fields["drivers"]), []byte("null")) { + if json.Unmarshal(data, &fields) != nil || bytes.Equal(bytes.TrimSpace(fields["drivers"]), []byte("null")) { return r, bad } d := json.NewDecoder(bytes.NewReader(data)) d.DisallowUnknownFields() - if d.Decode(&r) != nil || r.Schema != Schema || (r.Version != "unknown" && !release.MatchString(r.Version)) { + if d.Decode(&r) != nil || (r.Version != "unknown" && !release.MatchString(r.Version)) { + return r, bad + } + switch { + case r.Schema == SchemaV1 && len(fields) == 7: + case r.Schema == Schema && len(fields) == 11 && slices.Contains(SolveBuckets, r.Solve) && slices.Contains(FallbackBuckets, r.Fallbacks) && + slices.Contains(ErrorBuckets, r.Errors) && slices.Contains(UpdateBuckets, r.Update): + default: return r, bad } channel := "stable" @@ -67,16 +94,22 @@ func Decode(data []byte) (Report, error) { return r, nil } -// Day is a day's totals. No individual report survives aggregation. +// Day is a day's totals. No individual report survives aggregation. The +// health counts take only Schema reports, and are absent on a day without +// one. type Day struct { - Date string `json:"date"` - Reports int `json:"reports"` - Versions map[string]int `json:"ftw_versions"` - Channels map[string]int `json:"channels"` - Drivers map[string]int `json:"drivers"` - Battery map[string]int `json:"battery_kwh"` - Zones map[string]int `json:"price_zones"` - Age map[string]int `json:"install_age"` + Date string `json:"date"` + Reports int `json:"reports"` + Versions map[string]int `json:"ftw_versions"` + Channels map[string]int `json:"channels"` + Drivers map[string]int `json:"drivers"` + Battery map[string]int `json:"battery_kwh"` + Zones map[string]int `json:"price_zones"` + Age map[string]int `json:"install_age"` + Solve map[string]int `json:"plan_solve,omitempty"` + Fallbacks map[string]int `json:"plan_fallbacks,omitempty"` + Errors map[string]int `json:"driver_errors,omitempty"` + Update map[string]int `json:"update,omitempty"` } type View struct { @@ -108,6 +141,17 @@ func (d *Day) Add(r Report) bool { inc(d.Battery, r.Battery) inc(d.Zones, r.Zone) inc(d.Age, r.Age) + if r.Schema == Schema { + for _, c := range []struct { + counts *map[string]int + label string + }{{&d.Solve, r.Solve}, {&d.Fallbacks, r.Fallbacks}, {&d.Errors, r.Errors}, {&d.Update, r.Update}} { + if *c.counts == nil { + *c.counts = map[string]int{} + } + inc(*c.counts, c.label) + } + } return true } diff --git a/go/protocol/fleetwire/fleet_test.go b/go/protocol/fleetwire/fleet_test.go index 568e72aee..5975a496d 100644 --- a/go/protocol/fleetwire/fleet_test.go +++ b/go/protocol/fleetwire/fleet_test.go @@ -45,3 +45,59 @@ func TestCountersHaveFixedLimitsAndNoReportSurvives(t *testing.T) { t.Fatal("identifier persisted") } } + +const validV2 = `{"schema":"ftw.fleet/2","ftw_version":"v0.146.0-beta.1","channel":"beta","drivers":["pixii","pixii_pv"],"battery_kwh":"5-15","price_zone":"SE4","install_age":"1-6m","plan_solve":"1-5s","plan_fallbacks":"0","driver_errors":"0-1%","update":"none"}` + +// A report that says how the box ran carries four health buckets, each from +// a short fixed list; one without them is the older shape, and nothing else +// passes. +func TestHealthBucketsAreAFewFixedWords(t *testing.T) { + r, err := Decode([]byte(validV2)) + if err != nil || r.Solve != "1-5s" || r.Fallbacks != "0" || r.Errors != "0-1%" || r.Update != "none" { + t.Fatalf("%+v %v", r, err) + } + for _, bad := range []string{ + strings.Replace(validV2, `"1-5s"`, `"2.3s"`, 1), + strings.Replace(validV2, `"plan_fallbacks":"0"`, `"plan_fallbacks":"7"`, 1), + strings.Replace(validV2, `"0-1%"`, `"0.4%"`, 1), + strings.Replace(validV2, `"update":"none"`, `"update":"v0.145.0"`, 1), + strings.Replace(validV2, `,"update":"none"`, ``, 1), + strings.Replace(validV2, `"plan_solve":"1-5s"`, `"plan_solve":""`, 1), + strings.Replace(validV2, `"ftw.fleet/2"`, `"ftw.fleet/1"`, 1), + strings.Replace(valid, `"ftw.fleet/1"`, `"ftw.fleet/2"`, 1), + strings.TrimSuffix(validV2, "}") + `,"home":"secret"}`, + } { + if _, err := Decode([]byte(bad)); err == nil { + t.Fatalf("accepted %s", bad) + } + } + for _, list := range [][]string{SolveBuckets, FallbackBuckets, ErrorBuckets, UpdateBuckets} { + if len(list) > 4 { + t.Fatalf("%v is wider than four buckets", list) + } + } +} + +// A day counts health only from reports that carry it, and a day without +// any has no health fields at all. +func TestADayCountsHealthOnlyFromReportsThatCarryIt(t *testing.T) { + v1, _ := Decode([]byte(valid)) + v2, _ := Decode([]byte(validV2)) + d := NewDay("2026-10-24") + d.Add(v1) + data, _ := json.Marshal(d) + if strings.Contains(string(data), "plan_solve") || strings.Contains(string(data), `"update"`) { + t.Fatalf("a day of older reports has health fields: %s", data) + } + d.Add(v2) + d.Add(v2) + if d.Reports != 3 || d.Solve["1-5s"] != 2 || d.Fallbacks["0"] != 2 || d.Errors["0-1%"] != 2 || d.Update["none"] != 2 || d.Battery["5-15"] != 3 { + t.Fatalf("%+v", d) + } + // A day stored before health existed takes it once a report brings it. + var old Day + _ = json.Unmarshal([]byte(`{"date":"2026-10-24","reports":1,"ftw_versions":{},"channels":{},"drivers":{},"battery_kwh":{},"price_zones":{},"install_age":{}}`), &old) + if !old.Add(v2) || old.Update["none"] != 1 { + t.Fatalf("%+v", old) + } +} diff --git a/go/protocol/ohttp/bhttp.go b/go/protocol/ohttp/bhttp.go new file mode 100644 index 000000000..cf6604b41 --- /dev/null +++ b/go/protocol/ohttp/bhttp.go @@ -0,0 +1,200 @@ +package ohttp + +import ( + "encoding/binary" + "errors" +) + +// Known-length Binary HTTP (RFC 9292), the message inside a request and its +// response. Indeterminate-length messages are not read: nothing here sends +// them. + +// ErrMessage is a malformed Binary HTTP message. +var ErrMessage = errors.New("ohttp: invalid binary HTTP message") + +// Field is one header or trailer line. +type Field struct { + Name, Value string +} + +// Request is a known-length request. Trailers are always empty. +type Request struct { + Method, Scheme, Authority, Path string + Fields []Field + Content []byte +} + +// Response is a known-length final response. Trailers are always empty, and +// informational responses are not carried. +type Response struct { + Status int + Fields []Field + Content []byte +} + +const ( + knownRequest = 0 + knownResponse = 1 +) + +// Marshal writes the request, then zero padding up to size bytes. Padding +// keeps every request the same length on the wire (RFC 9292, section 3.8). +func (r Request) Marshal(size int) []byte { + b := appendVarint(nil, knownRequest) + for _, s := range []string{r.Method, r.Scheme, r.Authority, r.Path} { + b = appendBytes(b, []byte(s)) + } + b = appendFields(b, r.Fields) + b = appendBytes(b, r.Content) + b = appendVarint(b, 0) + return pad(b, size) +} + +// Marshal writes the response, then zero padding up to size bytes. +func (r Response) Marshal(size int) []byte { + b := appendVarint(nil, knownResponse) + b = appendVarint(b, uint64(r.Status)) + b = appendFields(b, r.Fields) + b = appendBytes(b, r.Content) + b = appendVarint(b, 0) + return pad(b, size) +} + +// ParseRequest reads a known-length request. Sections after the control data +// may be truncated, and anything after the trailers must be zero padding. +func ParseRequest(b []byte) (Request, error) { + d := decoder{b: b} + if d.varint() != knownRequest { + return Request{}, ErrMessage + } + var r Request + r.Method, r.Scheme, r.Authority, r.Path = string(d.bytes()), string(d.bytes()), string(d.bytes()), string(d.bytes()) + if d.err != nil { + return Request{}, ErrMessage + } + r.Fields = d.fields() + r.Content = d.optional() + trailers := d.fields() + if d.err != nil || len(trailers) != 0 || !d.padding() { + return Request{}, ErrMessage + } + return r, nil +} + +// ParseResponse reads a known-length final response. +func ParseResponse(b []byte) (Response, error) { + d := decoder{b: b} + if d.varint() != knownResponse { + return Response{}, ErrMessage + } + status := d.varint() + if d.err != nil || status < 200 || status > 599 { + return Response{}, ErrMessage + } + r := Response{Status: int(status)} + r.Fields = d.fields() + r.Content = d.optional() + trailers := d.fields() + if d.err != nil || len(trailers) != 0 || !d.padding() { + return Response{}, ErrMessage + } + return r, nil +} + +func appendVarint(b []byte, v uint64) []byte { + switch { + case v < 1<<6: + return append(b, byte(v)) + case v < 1<<14: + return binary.BigEndian.AppendUint16(b, uint16(v)|0x4000) + case v < 1<<30: + return binary.BigEndian.AppendUint32(b, uint32(v)|0x8000_0000) + default: + return binary.BigEndian.AppendUint64(b, v|0xc000_0000_0000_0000) + } +} + +func appendBytes(b, v []byte) []byte { + return append(appendVarint(b, uint64(len(v))), v...) +} + +func appendFields(b []byte, fields []Field) []byte { + var lines []byte + for _, f := range fields { + lines = appendBytes(lines, []byte(f.Name)) + lines = appendBytes(lines, []byte(f.Value)) + } + return appendBytes(b, lines) +} + +func pad(b []byte, size int) []byte { + if len(b) < size { + b = append(b, make([]byte, size-len(b))...) + } + return b +} + +type decoder struct { + b []byte + err error +} + +func (d *decoder) varint() uint64 { + if d.err != nil || len(d.b) == 0 { + d.err = ErrMessage + return 0 + } + n := 1 << (d.b[0] >> 6) + if len(d.b) < n { + d.err = ErrMessage + return 0 + } + v := uint64(d.b[0] & 0x3f) + for _, c := range d.b[1:n] { + v = v<<8 | uint64(c) + } + d.b = d.b[n:] + return v +} + +func (d *decoder) bytes() []byte { + n := d.varint() + if d.err != nil || n > uint64(len(d.b)) { + d.err = ErrMessage + return nil + } + v := d.b[:n] + d.b = d.b[n:] + return v +} + +// optional reads a length-prefixed part that a truncated message leaves out. +func (d *decoder) optional() []byte { + if d.err != nil || len(d.b) == 0 { + return nil + } + return d.bytes() +} + +func (d *decoder) fields() []Field { + section := decoder{b: d.optional()} + var out []Field + for d.err == nil && len(section.b) > 0 { + name, value := section.bytes(), section.bytes() + if section.err != nil || len(name) == 0 { + d.err = ErrMessage + return nil + } + out = append(out, Field{Name: string(name), Value: string(value)}) + } + return out +} + +func (d *decoder) padding() bool { + for _, c := range d.b { + if c != 0 { + return false + } + } + return true +} diff --git a/go/protocol/ohttp/bhttp_test.go b/go/protocol/ohttp/bhttp_test.go new file mode 100644 index 000000000..f3bd9acc9 --- /dev/null +++ b/go/protocol/ohttp/bhttp_test.go @@ -0,0 +1,63 @@ +package ohttp + +import ( + "bytes" + "encoding/hex" + "testing" +) + +func TestBinaryHTTPRoundTripsWithPadding(t *testing.T) { + req := Request{Method: "POST", Scheme: "https", Authority: "cloud.ftw.energy", Path: "/v1/x", + Fields: []Field{{"content-type", "application/json"}}, Content: bytes.Repeat([]byte{'a'}, 100)} + b := req.Marshal(256) + if len(b) != 256 { + t.Fatalf("padded to %d", len(b)) + } + got, err := ParseRequest(b) + if err != nil || got.Method != req.Method || got.Path != req.Path || len(got.Fields) != 1 || got.Fields[0] != req.Fields[0] || !bytes.Equal(got.Content, req.Content) { + t.Fatalf("%+v %v", got, err) + } + // Content longer than one byte's varint, and no padding asked for. + long := Request{Method: "POST", Scheme: "https", Authority: "a", Path: "/", Content: make([]byte, 20000)} + if got, err := ParseRequest(long.Marshal(0)); err != nil || len(got.Content) != 20000 { + t.Fatalf("long content %d %v", len(got.Content), err) + } + + resp := Response{Status: 204}.Marshal(16) + if r, err := ParseResponse(resp); err != nil || r.Status != 204 || len(resp) != 16 { + t.Fatalf("response %+v %v", r, err) + } +} + +func TestBinaryHTTPRefusesMalformedMessages(t *testing.T) { + good := Request{Method: "POST", Scheme: "https", Authority: "a", Path: "/", Content: []byte("x")}.Marshal(0) + withTrailer := append(bytes.Clone(good[:len(good)-1]), 4, 1, 'a', 1, 'b') + for name, b := range map[string][]byte{ + "empty": nil, + "response": Response{Status: 200}.Marshal(0), + "indeterminate": {2}, + "cut control": good[:4], + "cut content": good[:len(good)-2], + "bad padding": append(bytes.Clone(good), 0, 1), + "trailers": withTrailer, + "long length": {0, 0x3f}, + } { + if _, err := ParseRequest(b); err == nil { + t.Fatalf("%s parsed", name) + } + } + for name, b := range map[string][]byte{ + "informational": unhexB("014064"), + "status 600": unhexB("014258"), + "request": good, + } { + if _, err := ParseResponse(b); err == nil { + t.Fatalf("%s parsed", name) + } + } +} + +func unhexB(s string) []byte { + b, _ := hex.DecodeString(s) + return b +} diff --git a/go/protocol/ohttp/ohttp.go b/go/protocol/ohttp/ohttp.go new file mode 100644 index 000000000..cb990d5c4 --- /dev/null +++ b/go/protocol/ohttp/ohttp.go @@ -0,0 +1,335 @@ +// Package ohttp is Oblivious HTTP (RFC 9458) for one suite: DHKEM(X25519, +// HKDF-SHA256), HKDF-SHA256 and AES-128-GCM, with the known-length Binary +// HTTP messages (RFC 9292) it carries. +// +// A box sends its anonymous daily energy report this way: sealed to FTW +// Cloud's gateway key and posted to an independent relay. The relay sees the +// box's address and a ciphertext of fixed size. The gateway sees the report +// and the relay's address. Neither sees both. +// +// HPKE is the standard library's crypto/hpke. This package adds only the +// framing RFC 9458 puts around it and the response key schedule, and it +// speaks one suite so that no message can pick a weaker one. +package ohttp + +import ( + "bytes" + "crypto/aes" + "crypto/cipher" + "crypto/ecdh" + "crypto/hkdf" + "crypto/hpke" + "crypto/rand" + "crypto/sha256" + "encoding/binary" + "errors" + "slices" +) + +// The one suite, by its HPKE ids. +const ( + KEMX25519 uint16 = 0x0020 + KDFHKDFSHA256 uint16 = 0x0001 + AEADAES128GCM uint16 = 0x0001 +) + +// Media types of RFC 9458, section 9. +const ( + RequestType = "message/ohttp-req" + ResponseType = "message/ohttp-res" + KeysType = "application/ohttp-keys" +) + +const ( + // keyLen is Npk and Nenc for X25519. + keyLen = 32 + // headerLen is the request header: key id, KEM, KDF and AEAD. + headerLen = 7 + // The response key schedule for AES-128-GCM: Nk 16, Nn 12, and a + // response nonce of max(Nn, Nk). + aeadKeyLen = 16 + aeadNonceLen = 12 + nonceLen = 16 + tagLen = 16 + + requestLabel = "message/bhttp request" + responseLabel = "message/bhttp response" +) + +var ( + ErrConfig = errors.New("ohttp: invalid key configuration") + ErrUnsupported = errors.New("ohttp: key configuration offers no supported suite") + ErrRequest = errors.New("ohttp: cannot decapsulate the request") + ErrResponse = errors.New("ohttp: cannot open the response") +) + +// Algorithms is one KDF and AEAD pair a key configuration offers. +type Algorithms struct { + KDF, AEAD uint16 +} + +var suite = Algorithms{KDF: KDFHKDFSHA256, AEAD: AEADAES128GCM} + +// KeyConfig is a gateway's public key and what it accepts with it +// (RFC 9458, section 3). +type KeyConfig struct { + ID uint8 + KEM uint16 + PublicKey []byte + Algorithms []Algorithms +} + +// Marshal writes the configuration as RFC 9458 section 3.1 lays it out. +func (c KeyConfig) Marshal() []byte { + b := make([]byte, 0, 3+len(c.PublicKey)+2+4*len(c.Algorithms)) + b = append(b, c.ID) + b = binary.BigEndian.AppendUint16(b, c.KEM) + b = append(b, c.PublicKey...) + b = binary.BigEndian.AppendUint16(b, uint16(4*len(c.Algorithms))) + for _, a := range c.Algorithms { + b = binary.BigEndian.AppendUint16(b, a.KDF) + b = binary.BigEndian.AppendUint16(b, a.AEAD) + } + return b +} + +// ParseKeyConfig reads exactly one configuration. Only the X25519 KEM is +// known, because a key length cannot be read for a KEM this package does not +// speak. +func ParseKeyConfig(b []byte) (KeyConfig, error) { + if len(b) < 3 { + return KeyConfig{}, ErrConfig + } + c := KeyConfig{ID: b[0], KEM: binary.BigEndian.Uint16(b[1:3])} + if c.KEM != KEMX25519 { + return KeyConfig{}, ErrUnsupported + } + b = b[3:] + if len(b) < keyLen+2 { + return KeyConfig{}, ErrConfig + } + c.PublicKey = bytes.Clone(b[:keyLen]) + n := int(binary.BigEndian.Uint16(b[keyLen:])) + b = b[keyLen+2:] + if n < 4 || n%4 != 0 || len(b) != n { + return KeyConfig{}, ErrConfig + } + for ; len(b) > 0; b = b[4:] { + c.Algorithms = append(c.Algorithms, Algorithms{KDF: binary.BigEndian.Uint16(b), AEAD: binary.BigEndian.Uint16(b[2:])}) + } + if _, err := hpke.DHKEM(ecdh.X25519()).NewPublicKey(c.PublicKey); err != nil { + return KeyConfig{}, ErrConfig + } + return c, nil +} + +// MarshalKeys writes application/ohttp-keys: each configuration after its +// two-byte length. +func MarshalKeys(configs ...KeyConfig) []byte { + var b []byte + for _, c := range configs { + m := c.Marshal() + b = binary.BigEndian.AppendUint16(b, uint16(len(m))) + b = append(b, m...) + } + return b +} + +// ParseKeys reads application/ohttp-keys. A configuration for another KEM is +// skipped, as RFC 9458 section 3.2 allows; a malformed one fails the whole +// list. +func ParseKeys(b []byte) ([]KeyConfig, error) { + var out []KeyConfig + for len(b) > 0 { + if len(b) < 2 { + return nil, ErrConfig + } + n := int(binary.BigEndian.Uint16(b)) + if len(b) < 2+n { + return nil, ErrConfig + } + c, err := ParseKeyConfig(b[2 : 2+n]) + switch { + case errors.Is(err, ErrUnsupported): + case err != nil: + return nil, err + default: + out = append(out, c) + } + b = b[2+n:] + } + return out, nil +} + +func header(id uint8) []byte { + h := []byte{id} + h = binary.BigEndian.AppendUint16(h, KEMX25519) + h = binary.BigEndian.AppendUint16(h, suite.KDF) + return binary.BigEndian.AppendUint16(h, suite.AEAD) +} + +func info(hdr []byte) []byte { + return append(append([]byte(requestLabel), 0), hdr...) +} + +// ClientContext opens the one response to one request. +type ClientContext struct { + enc []byte + sender *hpke.Sender +} + +// Encapsulate seals msg to the gateway that published cfg (RFC 9458, +// section 4.3). A configuration that does not offer this package's suite is +// refused, never downgraded. +func Encapsulate(cfg KeyConfig, msg []byte) ([]byte, *ClientContext, error) { + if cfg.KEM != KEMX25519 || !slices.Contains(cfg.Algorithms, suite) { + return nil, nil, ErrUnsupported + } + pk, err := hpke.DHKEM(ecdh.X25519()).NewPublicKey(cfg.PublicKey) + if err != nil { + return nil, nil, ErrConfig + } + hdr := header(cfg.ID) + enc, sender, err := hpke.NewSender(pk, hpke.HKDFSHA256(), hpke.AES128GCM(), info(hdr)) + if err != nil { + return nil, nil, err + } + ct, err := sender.Seal(nil, msg) + if err != nil { + return nil, nil, err + } + out := slices.Concat(hdr, enc, ct) + return out, &ClientContext{enc: enc, sender: sender}, nil +} + +// Open reads the gateway's answer (RFC 9458, section 4.4). +func (c *ClientContext) Open(encResp []byte) ([]byte, error) { + secret, err := c.sender.Export(responseLabel, nonceLen) + if err != nil { + return nil, ErrResponse + } + return openResponse(secret, c.enc, encResp) +} + +// Key is one private key a gateway decapsulates with, under its key id. +type Key struct { + ID uint8 + Private *ecdh.PrivateKey +} + +// Gateway decapsulates requests sealed to its keys. +type Gateway struct { + keys map[uint8]hpke.PrivateKey + configs []KeyConfig +} + +// NewGateway holds X25519 keys under distinct ids. Each is published with +// this package's one suite. +func NewGateway(keys ...Key) (*Gateway, error) { + g := &Gateway{keys: map[uint8]hpke.PrivateKey{}} + for _, k := range keys { + if k.Private == nil || k.Private.Curve() != ecdh.X25519() { + return nil, ErrConfig + } + if _, dup := g.keys[k.ID]; dup { + return nil, ErrConfig + } + priv, err := hpke.NewDHKEMPrivateKey(k.Private) + if err != nil { + return nil, ErrConfig + } + g.keys[k.ID] = priv + g.configs = append(g.configs, KeyConfig{ID: k.ID, KEM: KEMX25519, PublicKey: k.Private.PublicKey().Bytes(), Algorithms: []Algorithms{suite}}) + } + if len(g.keys) == 0 { + return nil, ErrConfig + } + return g, nil +} + +// Configs are the gateway's key configurations, in the order given. +func (g *Gateway) Configs() []KeyConfig { return slices.Clone(g.configs) } + +// ServerContext seals the one response to one request. +type ServerContext struct { + enc, secret []byte +} + +// Decapsulate opens a request (RFC 9458, section 4.3). Every failure is the +// same error, so an answer says nothing about which check refused it. +func (g *Gateway) Decapsulate(req []byte) ([]byte, *ServerContext, error) { + if len(req) < headerLen+keyLen+tagLen { + return nil, nil, ErrRequest + } + key, ok := g.keys[req[0]] + if !ok || !bytes.Equal(req[:headerLen], header(req[0])) { + return nil, nil, ErrRequest + } + hdr, enc, ct := req[:headerLen], req[headerLen:headerLen+keyLen], req[headerLen+keyLen:] + r, err := hpke.NewRecipient(enc, key, hpke.HKDFSHA256(), hpke.AES128GCM(), info(hdr)) + if err != nil { + return nil, nil, ErrRequest + } + msg, err := r.Open(nil, ct) + if err != nil { + return nil, nil, ErrRequest + } + secret, err := r.Export(responseLabel, nonceLen) + if err != nil { + return nil, nil, ErrRequest + } + return msg, &ServerContext{enc: bytes.Clone(enc), secret: secret}, nil +} + +// Seal encapsulates the response under a fresh nonce (RFC 9458, section 4.4). +func (s *ServerContext) Seal(resp []byte) ([]byte, error) { + nonce := make([]byte, nonceLen) + if _, err := rand.Read(nonce); err != nil { + return nil, err + } + return sealResponse(s.secret, s.enc, nonce, resp) +} + +func responseAEAD(secret, enc, nonce []byte) (cipher.AEAD, []byte, error) { + prk, err := hkdf.Extract(sha256.New, secret, slices.Concat(enc, nonce)) + if err != nil { + return nil, nil, err + } + key, err := hkdf.Expand(sha256.New, prk, "key", aeadKeyLen) + if err != nil { + return nil, nil, err + } + iv, err := hkdf.Expand(sha256.New, prk, "nonce", aeadNonceLen) + if err != nil { + return nil, nil, err + } + block, err := aes.NewCipher(key) + if err != nil { + return nil, nil, err + } + aead, err := cipher.NewGCM(block) + return aead, iv, err +} + +func sealResponse(secret, enc, nonce, resp []byte) ([]byte, error) { + aead, iv, err := responseAEAD(secret, enc, nonce) + if err != nil { + return nil, err + } + return aead.Seal(bytes.Clone(nonce), iv, resp, nil), nil +} + +func openResponse(secret, enc, encResp []byte) ([]byte, error) { + if len(encResp) < nonceLen+tagLen { + return nil, ErrResponse + } + aead, iv, err := responseAEAD(secret, enc, encResp[:nonceLen]) + if err != nil { + return nil, ErrResponse + } + msg, err := aead.Open(nil, iv, encResp[nonceLen:], nil) + if err != nil { + return nil, ErrResponse + } + return msg, nil +} diff --git a/go/protocol/ohttp/ohttp_test.go b/go/protocol/ohttp/ohttp_test.go new file mode 100644 index 000000000..12f12386d --- /dev/null +++ b/go/protocol/ohttp/ohttp_test.go @@ -0,0 +1,247 @@ +package ohttp + +import ( + "bytes" + "crypto/ecdh" + "encoding/hex" + "errors" + "strings" + "testing" +) + +// RFC 9458, Appendix A. +const ( + rfcSecretKey = "3c168975674b2fa8e465970b79c8dcf09f1c741626480bd4c6162fc5b6a98e1a" + rfcKeyConfig = "01002031e1f05a740102115220e9af918f738674aec95f54db6e04eb705aae8e79815500080001000100010003" + rfcRequest = "00034745540568747470730b6578616d706c652e636f6d012f" + rfcInfo = "6d6573736167652f626874747020726571756573740001002000010001" + rfcEncRequest = "010020000100014b28f881333e7c164ffc499ad9796f877f4e1051ee6d31bad19dec96c208b4726374e469135906992e1268c594d2a10c695d858c40a026e7965e7d86b83dd440b2c0185204b4d63525" + rfcResponse = "0140c8" + rfcSecret = "62d87a6ba569ee81014c2641f52bea36" + rfcNonce = "c789e7151fcba46158ca84b04464910d" + rfcEncResp = "c789e7151fcba46158ca84b04464910d86f9013e404feea014e7be4a441f234f857fbd" +) + +func unhex(t *testing.T, s string) []byte { + t.Helper() + b, err := hex.DecodeString(s) + if err != nil { + t.Fatal(err) + } + return b +} + +func rfcGateway(t *testing.T) *Gateway { + t.Helper() + priv, err := ecdh.X25519().NewPrivateKey(unhex(t, rfcSecretKey)) + if err != nil { + t.Fatal(err) + } + g, err := NewGateway(Key{ID: 1, Private: priv}) + if err != nil { + t.Fatal(err) + } + return g +} + +func TestRFC9458KeyConfiguration(t *testing.T) { + cfg, err := ParseKeyConfig(unhex(t, rfcKeyConfig)) + if err != nil { + t.Fatal(err) + } + if cfg.ID != 1 || cfg.KEM != KEMX25519 || len(cfg.Algorithms) != 2 || + cfg.Algorithms[0] != (Algorithms{KDFHKDFSHA256, AEADAES128GCM}) || cfg.Algorithms[1] != (Algorithms{KDFHKDFSHA256, 0x0003}) { + t.Fatalf("parsed %+v", cfg) + } + if got := hex.EncodeToString(cfg.Marshal()); got != rfcKeyConfig { + t.Fatalf("marshal %s", got) + } + if got := rfcGateway(t).Configs()[0]; !bytes.Equal(got.PublicKey, cfg.PublicKey) || got.ID != 1 { + t.Fatalf("the secret key does not give the published key: %+v", got) + } + if got := hex.EncodeToString(info(header(1))); got != rfcInfo { + t.Fatalf("info %s", got) + } +} + +// The gateway opens the RFC's request, exports its secret, and seals the +// RFC's response byte for byte under the RFC's nonce. +func TestRFC9458GatewayVectors(t *testing.T) { + msg, ctx, err := rfcGateway(t).Decapsulate(unhex(t, rfcEncRequest)) + if err != nil { + t.Fatal(err) + } + if hex.EncodeToString(msg) != rfcRequest { + t.Fatalf("request %x", msg) + } + if hex.EncodeToString(ctx.secret) != rfcSecret { + t.Fatalf("secret %x", ctx.secret) + } + resp, err := sealResponse(ctx.secret, ctx.enc, unhex(t, rfcNonce), unhex(t, rfcResponse)) + if err != nil { + t.Fatal(err) + } + if hex.EncodeToString(resp) != rfcEncResp { + t.Fatalf("response %x", resp) + } + r, err := ParseRequest(msg) + if err != nil || r.Method != "GET" || r.Scheme != "https" || r.Authority != "example.com" || r.Path != "/" || len(r.Fields) != 0 || len(r.Content) != 0 { + t.Fatalf("binary request %+v %v", r, err) + } +} + +// The client half: its request header is the RFC's, and its response key +// schedule opens the RFC's encapsulated response. +func TestRFC9458ClientVectors(t *testing.T) { + enc := unhex(t, rfcEncRequest) + if got := hex.EncodeToString(header(1)); got != hex.EncodeToString(enc[:headerLen]) { + t.Fatalf("header %s", got) + } + resp, err := openResponse(unhex(t, rfcSecret), enc[headerLen:headerLen+keyLen], unhex(t, rfcEncResp)) + if err != nil { + t.Fatal(err) + } + if hex.EncodeToString(resp) != rfcResponse { + t.Fatalf("response %x", resp) + } + r, err := ParseResponse(resp) + if err != nil || r.Status != 200 { + t.Fatalf("binary response %+v %v", r, err) + } + + cfg, err := ParseKeyConfig(unhex(t, rfcKeyConfig)) + if err != nil { + t.Fatal(err) + } + req, cctx, err := Encapsulate(cfg, unhex(t, rfcRequest)) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(req[:headerLen], enc[:headerLen]) || len(req) != len(enc) { + t.Fatalf("request %x", req) + } + msg, sctx, err := rfcGateway(t).Decapsulate(req) + if err != nil || hex.EncodeToString(msg) != rfcRequest { + t.Fatalf("gateway read %x %v", msg, err) + } + sealed, err := sctx.Seal(unhex(t, rfcResponse)) + if err != nil { + t.Fatal(err) + } + got, err := cctx.Open(sealed) + if err != nil || hex.EncodeToString(got) != rfcResponse { + t.Fatalf("client read %x %v", got, err) + } +} + +func TestTheGatewayRefusesWhatItCannotOpen(t *testing.T) { + g := rfcGateway(t) + good := unhex(t, rfcEncRequest) + flip := func(i int) []byte { + b := bytes.Clone(good) + b[i] ^= 1 + return b + } + for name, req := range map[string][]byte{ + "empty": nil, + "short": good[:headerLen+keyLen], + "key id": flip(0), + "kem": flip(2), + "kdf": flip(4), + "aead": flip(6), + "enc": flip(headerLen), + "ciphertext": flip(len(good) - 1), + "truncated": good[:len(good)-1], + "extra bytes": append(bytes.Clone(good), 0), + } { + if _, _, err := g.Decapsulate(req); !errors.Is(err, ErrRequest) { + t.Fatalf("%s: %v", name, err) + } + } +} + +// A request sealed to one key cannot be read by a gateway holding another, +// even under the same key id. A box encrypts only to the configuration its +// release pins, so a key swapped in at the service reads nothing. +func TestASwappedKeyReadsNothing(t *testing.T) { + cfg, err := ParseKeyConfig(unhex(t, rfcKeyConfig)) + if err != nil { + t.Fatal(err) + } + req, _, err := Encapsulate(cfg, []byte("report")) + if err != nil { + t.Fatal(err) + } + other, err := ecdh.X25519().GenerateKey(nil) + if err != nil { + t.Fatal(err) + } + g, err := NewGateway(Key{ID: 1, Private: other}) + if err != nil { + t.Fatal(err) + } + if _, _, err := g.Decapsulate(req); !errors.Is(err, ErrRequest) { + t.Fatalf("a different key opened the request: %v", err) + } +} + +func TestTheClientRefusesOtherSuitesAndForgedResponses(t *testing.T) { + cfg, err := ParseKeyConfig(unhex(t, rfcKeyConfig)) + if err != nil { + t.Fatal(err) + } + only := cfg + only.Algorithms = []Algorithms{{KDFHKDFSHA256, 0x0003}} + if _, _, err := Encapsulate(only, nil); !errors.Is(err, ErrUnsupported) { + t.Fatalf("ChaCha20 only: %v", err) + } + _, cctx, err := Encapsulate(cfg, []byte("x")) + if err != nil { + t.Fatal(err) + } + for _, forged := range [][]byte{nil, make([]byte, nonceLen+tagLen), unhex(t, rfcEncResp)} { + if _, err := cctx.Open(forged); !errors.Is(err, ErrResponse) { + t.Fatalf("opened %x: %v", forged, err) + } + } +} + +func TestKeyConfigurationsAreReadStrictly(t *testing.T) { + good := unhex(t, rfcKeyConfig) + for name, b := range map[string][]byte{ + "empty": nil, + "trailing byte": append(bytes.Clone(good), 0), + "short key": append(bytes.Clone(good[:3+31]), good[3+32:]...), + "no algorithms": append(bytes.Clone(good[:3+32]), 0, 0), + "odd length": append(bytes.Clone(good[:3+32]), 0, 6, 0, 1, 0, 1, 0, 1), + } { + if _, err := ParseKeyConfig(b); err == nil { + t.Fatalf("%s parsed", name) + } + } + p256 := bytes.Clone(good) + p256[2] = 0x10 + if _, err := ParseKeyConfig(p256); !errors.Is(err, ErrUnsupported) { + t.Fatalf("P-256: %v", err) + } + + cfg, _ := ParseKeyConfig(good) + list := MarshalKeys(KeyConfig{ID: 9, KEM: 0x0010, PublicKey: make([]byte, 65), Algorithms: []Algorithms{{1, 1}}}, cfg) + got, err := ParseKeys(list) + if err != nil || len(got) != 1 || !bytes.Equal(got[0].Marshal(), good) { + t.Fatalf("keys %+v %v", got, err) + } + if _, err := ParseKeys(list[:len(list)-1]); err == nil { + t.Fatal("a cut list parsed") + } + if _, err := NewGateway(); err == nil { + t.Fatal("a gateway without keys") + } + k, _ := ecdh.X25519().GenerateKey(nil) + if _, err := NewGateway(Key{ID: 1, Private: k}, Key{ID: 1, Private: k}); err == nil { + t.Fatal("two keys under one id") + } + if !strings.HasPrefix(RequestType, "message/") || KeysType != "application/ohttp-keys" { + t.Fatal("media types") + } +} diff --git a/go/protocol/reportwire/report.go b/go/protocol/reportwire/report.go new file mode 100644 index 000000000..a69557be4 --- /dev/null +++ b/go/protocol/reportwire/report.go @@ -0,0 +1,112 @@ +// Package reportwire is how the box's anonymous daily reports reach FTW +// Cloud: each report alone, as one padded inner request sealed with Oblivious +// HTTP (package ohttp) and posted to an independent relay. The gateway reads +// the report's schema and hands it to that report's decoder: the fleet report +// (package fleetwire) or the energy report (package energywire). +// +// Two reports, never one message. With a fleet of a few boxes a driver set +// and a price zone can point at one home, so energy sent beside them would +// belong to that home again. Each report leaves at its own random time. +package reportwire + +import ( + "encoding/base64" + "encoding/json" + "time" + + "github.com/srcfl/ftw/go/protocol/ohttp" +) + +// The one inner request. The gateway answers no other. +const ( + Method = "POST" + Scheme = "https" + Authority = "cloud.ftw.energy" + Path = "/v1/report" +) + +const ( + // RequestSize is every padded inner request and ResponseSize every padded + // inner response, so the relay sees one length whatever the report, its + // kind or the answer. The largest fleet report fits. + RequestSize = 1536 + ResponseSize = 16 +) + +// IDBytes is a report id's length. The box draws one at random for each +// report and keeps it if it ever sends that report again, so the gateway +// counts a report once however often it arrives (RFC 9458, section 6.5). +const IDBytes = 16 + +// Report is what one inner request carries besides its fixed parts: the +// report's random id, the UTC date the box sent it, and the report itself. +// The id names one report, never a box. +type Report struct { + ID []byte + Day string + Body []byte +} + +const ( + idField = "report-id" + dayField = "report-day" +) + +// Request is the padded inner request carrying one report. ok is false for +// an id or a day of the wrong form, and when the report does not fit, so a +// report never goes out at a length of its own. +func Request(r Report) (msg []byte, ok bool) { + if len(r.ID) != IDBytes || !isDate(r.Day) { + return nil, false + } + msg = ohttp.Request{Method: Method, Scheme: Scheme, Authority: Authority, Path: Path, + Fields: []ohttp.Field{{Name: idField, Value: base64.RawURLEncoding.EncodeToString(r.ID)}, {Name: dayField, Value: r.Day}}, + Content: r.Body}.Marshal(RequestSize) + return msg, len(msg) == RequestSize +} + +// Read opens an inner request, or returns false for anything but the one +// padded request with an id and a day. +func Read(msg []byte) (Report, bool) { + req, err := ohttp.ParseRequest(msg) + if err != nil || len(msg) != RequestSize || req.Method != Method || req.Scheme != Scheme || + req.Authority != Authority || req.Path != Path || len(req.Fields) != 2 || + req.Fields[0].Name != idField || req.Fields[1].Name != dayField { + return Report{}, false + } + id, err := base64.RawURLEncoding.DecodeString(req.Fields[0].Value) + if err != nil || len(id) != IDBytes || !isDate(req.Fields[1].Value) { + return Report{}, false + } + return Report{ID: id, Day: req.Fields[1].Value, Body: req.Content}, true +} + +// Fresh reports whether a report sent on day still counts at now: sent today +// or yesterday, UTC. An older one is refused, so a captured request cannot be +// posted again on a later day, when its id is no longer remembered. +func Fresh(day string, now time.Time) bool { + today := now.UTC().Format(time.DateOnly) + return day == today || day == now.UTC().AddDate(0, 0, -1).Format(time.DateOnly) +} + +func isDate(s string) bool { + t, err := time.Parse(time.DateOnly, s) + return err == nil && t.Format(time.DateOnly) == s +} + +// Schema reads only a report's schema, to choose its decoder. The decoder +// still checks every byte. +func Schema(body []byte) string { + var head struct { + Schema string `json:"schema"` + } + if json.Unmarshal(body, &head) != nil { + return "" + } + return head.Schema +} + +// Response is the padded inner answer. +func Response(status int) []byte { + return ohttp.Response{Status: status}.Marshal(ResponseSize) +} diff --git a/go/protocol/reportwire/report_test.go b/go/protocol/reportwire/report_test.go new file mode 100644 index 000000000..02c0ed802 --- /dev/null +++ b/go/protocol/reportwire/report_test.go @@ -0,0 +1,96 @@ +package reportwire + +import ( + "encoding/json" + "strings" + "testing" + "time" + + "github.com/srcfl/ftw/go/protocol/energywire" + "github.com/srcfl/ftw/go/protocol/fleetwire" + "github.com/srcfl/ftw/go/protocol/ohttp" +) + +// The largest report of either kind fills the same padded request, so the +// relay cannot tell a fleet report from an energy report, or one box's from +// another's, by its length. +func TestEveryReportHasOneLength(t *testing.T) { + drivers := make([]string, 16) + for i := range drivers { + drivers[i] = strings.Repeat(string(rune('a'+i)), 64) + } + fleet, _ := json.Marshal(fleetwire.Report{Schema: fleetwire.Schema, Version: "v1234.1234.1234-beta.1234", Channel: "unknown", + Drivers: drivers, Battery: "30+", Zone: strings.Repeat("Z", 16), Age: "unknown", + Solve: "none", Fallbacks: "none", Errors: "none", Update: "failed"}) + energy, _ := json.Marshal(energywire.Report{Schema: energywire.Schema, Date: "2026-10-23", PVGeneration: 1000, GridImport: 1000, + GridExport: 1000, BatteryCharge: 1000, BatteryDischarge: 1000, VehicleCharge: 1000}) + small, _ := json.Marshal(energywire.Report{Schema: energywire.Schema, Date: "2026-10-23"}) + id := []byte("0123456789abcdef") + for _, body := range [][]byte{fleet, energy, small} { + msg, ok := Request(Report{ID: id, Day: "2026-10-24", Body: body}) + if !ok || len(msg) != RequestSize { + t.Fatalf("%d-byte report padded to %d", len(body), len(msg)) + } + got, ok := Read(msg) + if !ok || string(got.Body) != string(body) || string(got.ID) != string(id) || got.Day != "2026-10-24" || Schema(got.Body) == "" { + t.Fatalf("read back %+v", got) + } + } + if Schema(fleet) != fleetwire.Schema || Schema(energy) != energywire.Schema || Schema([]byte("x")) != "" { + t.Fatal("schema") + } + for name, r := range map[string]Report{ + "oversized": {ID: id, Day: "2026-10-24", Body: make([]byte, RequestSize)}, + "short id": {ID: id[:15], Day: "2026-10-24", Body: small}, + "no day": {ID: id, Body: small}, + "loose day": {ID: id, Day: "2026-10-4", Body: small}, + } { + if _, ok := Request(r); ok { + t.Fatalf("%s was padded", name) + } + } + if len(Response(204)) != ResponseSize { + t.Fatal("response padding") + } +} + +func TestOnlyTheOneInnerRequestIsRead(t *testing.T) { + fields := []ohttp.Field{{Name: "report-id", Value: "MDEyMzQ1Njc4OWFiY2RlZg"}, {Name: "report-day", Value: "2026-10-24"}} + good := ohttp.Request{Method: Method, Scheme: Scheme, Authority: Authority, Path: Path, Fields: fields, Content: []byte("{}")} + if _, ok := Read(good.Marshal(RequestSize)); !ok { + t.Fatal("the one request was refused") + } + other := func(f func(*ohttp.Request)) []byte { + r := good + f(&r) + return r.Marshal(RequestSize) + } + for name, msg := range map[string][]byte{ + "another path": other(func(r *ohttp.Request) { r.Path = "/v1/energy/report" }), + "another method": other(func(r *ohttp.Request) { r.Method = "PUT" }), + "another host": other(func(r *ohttp.Request) { r.Authority = "example.com" }), + "plain http": other(func(r *ohttp.Request) { r.Scheme = "http" }), + "another header": other(func(r *ohttp.Request) { r.Fields = append(r.Fields, ohttp.Field{Name: "x-home", Value: "1"}) }), + "no id": other(func(r *ohttp.Request) { r.Fields = r.Fields[1:] }), + "short id": other(func(r *ohttp.Request) { r.Fields = []ohttp.Field{{Name: "report-id", Value: "MDEy"}, fields[1]} }), + "bad day": other(func(r *ohttp.Request) { r.Fields = []ohttp.Field{fields[0], {Name: "report-day", Value: "yesterday"}} }), + "swapped": other(func(r *ohttp.Request) { r.Fields = []ohttp.Field{fields[1], fields[0]} }), + "not padded": good.Marshal(0), + "padded more": good.Marshal(RequestSize + 1), + "a response": Response(200), + } { + if _, ok := Read(msg); ok { + t.Fatalf("%s was read", name) + } + } +} + +// A report counts on the day it was sent and the next, UTC, and no later. +func TestOnlyAFreshReportCounts(t *testing.T) { + now := time.Date(2026, 10, 24, 0, 30, 0, 0, time.UTC) + for day, want := range map[string]bool{"2026-10-24": true, "2026-10-23": true, "2026-10-22": false, "2026-10-25": false, "x": false} { + if Fresh(day, now) != want { + t.Fatalf("%s: fresh %v", day, !want) + } + } +} diff --git a/go/test/site/README.md b/go/test/site/README.md new file mode 100644 index 000000000..3289356fb --- /dev/null +++ b/go/test/site/README.md @@ -0,0 +1,100 @@ +# Site test: FTW against a simulated house + +`TestSite` runs the real `ftw` binary against a house from +[srcfl/device-simulator](https://github.com/srcfl/device-simulator): its +headless run (`simulator run`, docs/HEADLESS_RUN.md there) builds a Sungrow +hybrid whose battery follows FTW's EMS writes after 2 s, an SDM630 site +meter behind an IEC gG fuse on each phase, and OCPP 1.6 chargers with the +car on them. FTW talks to it over Modbus and OCPP as on a real site. Each +scenario starts a fresh house and a fresh `state.db`, drives the house and +FTW, and judges the run against fixed criteria from what flowed through the +fuse (the house's record) and what FTW said (its API, once a second). FTW +runs `planner_self` with the Core planner and SE3 prices. + +The house lives in device-simulator; this directory keeps only what is +FTW's: the scenarios, FTW's configuration, the criteria and the pinned +simulator commit in [`devsim.version`](devsim.version). + +## Run it + +```bash +make siterig # the short set, two lanes +make siterig-full # every scenario +make siterig SITERIG_SET=s1-25,s9-25 # named scenarios +make siterig-coverage CATALOG=../reports/felkatalog/catalog.yaml +make siterig-judge DIR=/tmp/ftw-site-123 # judge a recorded run again +``` + +| Variable | Meaning | +|---|---| +| `FTW_SITE` | `short`, `full`, or names. Unset skips the test. | +| `FTW_SITE_LANES` | scenarios at once (`SITERIG_LANES`, default 2), each lane on its own 100 ports | +| `FTW_SITE_PORT_BASE` | first port on 127.0.0.1 (default 18700): the house on base+0..59, FTW on base+90 and +91 | +| `FTW_SITE_SHARD` | `k/n`: shard k of n, for several machines | +| `FTW_SITE_OUT` | run directories, `summary.md` and `summary.json` (default: a new temporary directory) | +| `FTW_SITE_DEVSIM_SRC` | a device-simulator checkout at the pinned commit; without it the test clones the pin into the user cache | +| `FTW_SITE_FTW_BIN`, `FTW_SITE_DEVSIM_BIN` | prebuilt binaries | +| `FTW_SITE_CATALOG` | the fault catalog, for `TestCatalogCoverage` | +| `FTW_SITE_JUDGE` | a results directory: `TestJudge` judges its runs again with the criteria as they are now, and rewrites their `result.json` and the summary | + +Each run directory keeps `house/plant.jsonl` (the house once a second, with +every event and mark), `house/ocpp-.jsonl` (every OCPP frame the +charger read or wrote), `ftw.jsonl` (FTW's `/api/status` and +`/api/loadpoints` once a second), `ftw.log`, `house.log`, `config.yaml`, +`house.json` and `result.json`. + +A run whose record has a hole over 15 s fails as "host paused": a sleeping +laptop stretches every wait and proves nothing. + +## Where it runs + +Not in GitHub Actions. `siterig-runner` ([runner/](runner/README.md)) runs +on a machine of its own, watches `master` and open pull requests, and posts +the result on the exact commit as the status `siterig/short` (pull requests +that touch control, loadpoint, OCPP or telemetry code), `siterig/full` +(each new `master` commit) or `siterig/random`. A beta needs +`siterig/full` on its commit. + +## Known failures + +A criterion named in a scenario's `known` map fails on master for the +finding it names. The run passes while it keeps failing, and fails once it +passes: a fix deletes its line in the same pull request. + +## The fault catalog + +Each scenario names the entries of FTW's field fault catalog it covers and +its sources (`provenance` in [catalog_test.go](catalog_test.go)). +`make siterig-coverage` lists, per area, which faults have a scenario. + +## Scenarios + +"Over the limit" is the worst phase above the fuse − 1 A, measured through +the fuse. A device's response time counts as 2 s, the meter and control +tick as 2 s more. + +| Scenario | House and steps | Criteria | Short | +|---|---|---|---| +| `s0-25` Idle charger | 25 A, 3 A per phase, a charger with no car, heartbeat 60 s, 300 s | Home use `measured` in 95 % of samples; no OCPP disconnect | yes | +| `s0b-25` | as s0, Core's heartbeat | as s0 | | +| `s1-16`, `s1-25` Charge now | 3 A per phase, Charge now 16 A for 180 s | the room the fuse gives (25 A: 11 040 W) within 60 s; held 95 %; fuse holds; on 25 A no `fuse_limit` | yes | +| `s2-16` Charge now during export | 16 A, 1 A house, 7 kW PV, full battery: 9.1 A export per phase | car starts within 60 s (**known**: export counted as import); fuse holds | | +| `s3-16`, `s3-25` Load step | car and a 3 kW battery hold charging, then 13 A (16 A) or 21 A (25 A) per phase for 180 s | fuse holds; over the limit at most 7 s after the step | yes | +| `s4-25`, `s4b-25` Saved unlock | `surplus_unlock_bat_soc` 0.9 saved, not armed; Charge now 3p, and bare (s4b) | no stops; no offer drops to 0 W; full power over the last minute | | +| `s5-25`, `s5s-25` Surplus only | 25 A, 9.3 kW surplus, `auto`; s5 without steps, s5s with Easee's | no offer above 16 A; fuse holds; only 6–16 A; ≥ 4 140 W 90 % of the time (**known** in both) | | +| `s5t-25` Surplus with a goal | as s5s with a goal, 3.8 kW of export, a charger that switches phases | no offer above 16 A; fuse holds; ≥ 6 A on one phase 90 % of the time | | +| `s6-25` Inverter offline | 1.5 kW PV, the inverter silent for 300 s | Home use shown and marked estimated throughout; measured before and after; never estimated while every driver reads | | +| `s7-25` Restart | Charge now, 15 min, Core down 20 s, 420 s after | the day's Home use is kept; Home use back within 30 s; Charge now never below 4 140 W; full power within 420 s | | +| `s7s-25` Short restart | as s7, restarted after 60 s | as s7 without the day total, which needs the long run (10 % rule) | yes | +| `s8-60-25`, `s8-120-25` Slow charger | 15 A per phase, battery at its 5 % floor, profiles applied 60 or 120 s late | fuse holds; over the limit at most the delay + 7 s | 60 s | +| `s9-25` 3p to 1p | 10 A per phase, a 32 A charger that switches phases; Charge now 7 360 W on 3p, after 90 s on 1p | fuse holds; over the limit at most 3 s before and after the switch | | +| `s9r-25` 1p to 3p beside a loaded L2 | house 2/20/2 A, the 32 A charger; Charge now 7 360 W on 1p, after 90 s on 3p | as s9 | | +| `s10-25` Two chargers | `carport` 60 s late; cut 6 A and raise 16 A together | fuse holds; over the limit at most 3 s after the step | | +| `s11-25` Pause, then restart | a charger applying 5 s late; Charge now 16 A, 0 W for 25 s, then 6 A | an offer within 60 s of the restart; over the limit at most 3 s; fuse holds | | +| `s12-20` One-phase night | 3 × 20 A, house 1.6/1.7/2.3 A, a charger applying 40 s late, the field's step table on `auto`, a plan for 6.6 kWh by the first quarter hour 40 min out | fuse holds; planned charging never orders one phase; no order cut while every phase has room, unless the plan paused; 90 % of the plan by the ready time | | +| `s13-25` Socket reconnect | Charge now 16 A, the charger's connection gone for 10 s | Charge now goes on after the charger returns (**known**: #1788); no unplug seen | | +| `s15-25` One-phase charger, Charge now | 25 A, 3 A per phase, an Easee wired on L1 (6-16 A, steps to 3 680 W, no phase mode); Charge now on the phases `/api/loadpoints` reports, as the app sends it, 180 s | 16 A on L1 within 60 s; held 95 %; fuse holds; no `fuse_limit` | yes | +| `s15b-25`, `s15c-25` | as s15, phase mode `1p` and `auto` | as s15 | | +| `s15bat-16` One-phase charger, battery rising | 16 A, 3 A per phase, the s15 charger, battery at 50 % following its EMS 8 s late; PV 2.1 kW rising every 30 s (+2.5 kW, then 1.5 kW) so Core charges the battery in steps; Charge now 6 s after the second step, while the raise is on its way | the battery order rose over 1 150 W in the 20 s before (setup check); car charging within 10 s; fuse holds; over the limit at most 3 s | | +| `s15p-25` One-phase charger, planned | as s15, a plan for 600 Wh by the first quarter hour 15 min out | fuse holds; 90 % of the plan by the ready time; never called not following | | +| `s14-25` One-phase car | an 11 kW charger with a one-phase car, a plan it cannot meet, 20 min | 3.4–3.8 kW over the last 10 min; never called not following | | diff --git a/go/test/site/catalog_test.go b/go/test/site/catalog_test.go new file mode 100644 index 000000000..ab7a55199 --- /dev/null +++ b/go/test/site/catalog_test.go @@ -0,0 +1,171 @@ +//go:build unix + +package site + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +// provenance says where each scenario comes from: the entries of FTW's +// field fault catalog it covers (reports/felkatalog/catalog.yaml in the FTW +// stack workspace) and its sources. +var provenance = map[string]struct { + catalog []string + sources []string +}{ + "s0-25": {[]string{"home-070", "ocpp-011", "ocpp-013", "ui-052"}, []string{"https://github.com/srcfl/ftw/pull/1779", "https://github.com/srcfl/ftw/pull/1790"}}, + "s0b-25": {[]string{"home-070", "ocpp-011", "ocpp-013", "ui-052"}, []string{"https://github.com/srcfl/ftw/pull/1790"}}, + "s1-16": {nil, []string{"simulator test before beta 0.155.0, 21 Oct 2026"}}, + "s1-25": {nil, []string{"simulator test before beta 0.155.0, 21 Oct 2026"}}, + "s2-16": {[]string{"ev-078"}, []string{"https://github.com/srcfl/ftw/pull/1782"}}, + "s3-16": {[]string{"ev-100"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s3-25": {[]string{"ev-100"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s4-25": {[]string{"ev-093"}, []string{"https://github.com/srcfl/ftw/pull/1774"}}, + "s4b-25": {[]string{"ev-093"}, []string{"https://github.com/srcfl/ftw/pull/1774"}}, + "s5-25": {[]string{"ev-089", "ev-110"}, []string{"https://github.com/srcfl/ftw/pull/1774"}}, + "s5s-25": {[]string{"ev-089", "ev-102"}, []string{"https://github.com/srcfl/ftw/pull/1791"}}, + "s6-25": {[]string{"home-061", "ocpp-010", "tel-035", "tel-037", "tel-038", "ui-052"}, []string{"https://github.com/srcfl/ftw/pull/1780"}}, + "s7-25": {[]string{"dev-100", "ev-085", "ev-091", "home-061", "home-074", "hw-061"}, []string{"https://github.com/srcfl/ftw/pull/1778", "https://github.com/srcfl/ftw/pull/1787"}}, + "s7s-25": {[]string{"ev-085", "ev-091"}, []string{"https://github.com/srcfl/ftw/pull/1778"}}, + "s8-60-25": {[]string{"dev-076", "ev-099", "ev-107"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s8-120-25": {[]string{"dev-076", "ev-099", "ev-107"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s9-25": {[]string{"ev-094"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s9r-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1800 (probe: a held 1p order raised to 3p beside L2 at 20 A)"}}, + "s10-25": {[]string{"dev-076", "ev-105"}, []string{"https://github.com/srcfl/ftw/pull/1776"}}, + "s11-25": {[]string{"ev-104"}, []string{"https://github.com/srcfl/ftw/pull/1763", "https://github.com/srcfl/ftw/pull/1793"}}, + "s5t-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1800 (probe: surplus-only with a goal)", "https://github.com/srcfl/ftw/pull/1799"}}, + "s12-20": {[]string{"ev-097"}, []string{"https://github.com/srcfl/ftw/pull/1799", "field report, 22 Oct 2026, v0.155.0-beta.1"}}, + "s13-25": {[]string{"ocpp-009", "dev-119"}, []string{"https://github.com/srcfl/ftw/issues/1788"}}, + "s14-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1800 (help1022 car1p, fix D)"}}, + "s15-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1804", "release blocker 22 Oct 2026: a one-phase charger gets 0 W under Charge now"}}, + "s15b-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1804", "release blocker 22 Oct 2026: a one-phase charger gets 0 W under Charge now (phase mode 1p)"}}, + "s15c-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1804", "release blocker 22 Oct 2026: a one-phase charger gets 0 W under Charge now (phase mode auto)"}}, + "s15bat-16": {nil, []string{"https://github.com/srcfl/ftw/pull/1805", "release blocker 22 Oct 2026: a one-phase charger never starts while Core raises its battery"}}, + "s15p-25": {nil, []string{"https://github.com/srcfl/ftw/pull/1804", "release blocker 22 Oct 2026: a one-phase charger, planned charging"}}, +} + +func TestEveryScenarioSaysWhereItComesFrom(t *testing.T) { + for _, sc := range scenarios { + p, ok := provenance[sc.name] + if !ok || len(p.catalog)+len(p.sources) == 0 { + t.Errorf("%s has no catalog entry or source", sc.name) + } + } + for name := range provenance { + found := false + for _, sc := range scenarios { + found = found || sc.name == name + } + if !found { + t.Errorf("provenance for %s, which is no scenario", name) + } + } +} + +type catalogEntry struct { + ID string `yaml:"id"` + Kind string `yaml:"kind"` + Area string `yaml:"area"` + Title string `yaml:"title"` + Status string `yaml:"status"` + Sources []string `yaml:"sources"` +} + +// TestCatalogCoverage reads the fault catalog at FTW_SITE_CATALOG and writes +// coverage.md: per area, which faults have a scenario here and which do not. +// It fails when a scenario names an entry the catalog does not have. +func TestCatalogCoverage(t *testing.T) { + path := os.Getenv("FTW_SITE_CATALOG") + if path == "" { + t.Skip("set FTW_SITE_CATALOG to the fault catalog (catalog.yaml)") + } + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + var cat struct { + Entries []catalogEntry `yaml:"entries"` + } + if err := yaml.Unmarshal(b, &cat); err != nil { + t.Fatal(err) + } + byID := map[string]catalogEntry{} + for _, e := range cat.Entries { + byID[e.ID] = e + } + covered := map[string][]string{} + for name, p := range provenance { + for _, id := range p.catalog { + if _, ok := byID[id]; !ok { + t.Errorf("%s names catalog entry %s, which the catalog does not have", name, id) + } + covered[id] = append(covered[id], name) + } + } + type tally struct{ faults, open, covered, openCovered int } + areas := map[string]*tally{} + var md strings.Builder + md.WriteString("# Site-test coverage of the fault catalog\n\n") + md.WriteString("| Entry | Status | Title | Scenarios |\n|---|---|---|---|\n") + ids := make([]string, 0, len(byID)) + for id := range byID { + ids = append(ids, id) + } + sort.Strings(ids) + for _, id := range ids { + e := byID[id] + if e.Kind != "fault" { + if len(covered[id]) > 0 { + sort.Strings(covered[id]) + fmt.Fprintf(&md, "| %s | %s | %s | %s |\n", id, e.Kind, e.Title, strings.Join(covered[id], ", ")) + } + continue + } + a := areas[e.Area] + if a == nil { + a = &tally{} + areas[e.Area] = a + } + open := e.Status == "open" || e.Status == "partly_fixed" || e.Status == "fix_unmerged" + a.faults++ + if open { + a.open++ + } + if len(covered[id]) > 0 { + a.covered++ + if open { + a.openCovered++ + } + sort.Strings(covered[id]) + fmt.Fprintf(&md, "| %s | %s | %s | %s |\n", id, e.Status, e.Title, strings.Join(covered[id], ", ")) + } + } + md.WriteString("\n| Area | Faults | With a scenario | Open | Open with a scenario |\n|---|---:|---:|---:|---:|\n") + names := make([]string, 0, len(areas)) + for a := range areas { + names = append(names, a) + } + sort.Strings(names) + for _, a := range names { + x := areas[a] + fmt.Fprintf(&md, "| %s | %d | %d | %d | %d |\n", a, x.faults, x.covered, x.open, x.openCovered) + } + out := os.Getenv("FTW_SITE_OUT") + if out == "" { + // No place to keep it: the log is the report. + t.Logf("%d catalog entries, %d with a scenario\n\n%s", len(byID), len(covered), md.String()) + return + } + dst := filepath.Join(out, "coverage.md") + if err := os.WriteFile(dst, []byte(md.String()), 0o644); err != nil { + t.Fatal(err) + } + t.Logf("%d catalog entries, %d with a scenario; wrote %s", len(byID), len(covered), dst) +} diff --git a/go/test/site/devsim.version b/go/test/site/devsim.version new file mode 100644 index 000000000..c0ca38fa8 --- /dev/null +++ b/go/test/site/devsim.version @@ -0,0 +1 @@ +a3030d3ff2d4b547d0aa596022ca85daa69904eb diff --git a/go/test/site/evaluate_test.go b/go/test/site/evaluate_test.go new file mode 100644 index 000000000..83b34ff20 --- /dev/null +++ b/go/test/site/evaluate_test.go @@ -0,0 +1,682 @@ +//go:build unix + +package site + +import ( + "bufio" + "encoding/json" + "math" + "os" + "path/filepath" + "sort" + "strings" + "time" +) + +const ( + volt = 230.0 + maxW = 11040.0 // the rig charger, 16 A on three phases + minW = 4140.0 // the loadpoints' minimum, 6 A on three phases +) + +// ---- what the house and the recorder wrote ---- + +// plantLine is a line of the house's plant.jsonl: a sample a second, an +// event or a mark. +type plantLine struct { + T float64 `json:"t"` + Mark string `json:"mark"` + Event json.RawMessage `json:"event"` + PhaseA [3]float64 `json:"phase_a"` // through each fuse, + into the site + Fuse *plantFuse `json:"fuse"` + Devices map[string]plantDevice `json:"devices"` +} + +type plantFuse struct { + Damage [3]float64 `json:"damage"` + Blown [3]bool `json:"blown"` +} + +type plantDevice struct { + W float64 `json:"w"` // a charger's draw + BatteryW float64 `json:"battery_w"` // an inverter's battery, + charging + SoCPct float64 `json:"soc_pct"` + EMSCmd float64 `json:"ems_cmd"` // 170 charge, 187 discharge, 204 stop + EMSSetW float64 `json:"ems_setpoint_w"` + Connected bool `json:"connected"` +} + +type ftwStatus struct { + Drivers map[string]struct { + Status string `json:"status"` + } `json:"drivers"` + LoadQuality *string `json:"load_quality"` + LoadReason *string `json:"load_reason"` + LoadW *float64 `json:"load_w"` + Energy struct { + Today struct { + LoadWh *float64 `json:"load_wh"` + HistoryQuality *string `json:"history_quality"` + } `json:"today"` + } `json:"energy"` +} + +type ftwLoadpoint struct { + ID string `json:"id"` + Plugged bool `json:"plugged_in"` + Phases int `json:"phases"` + Charger struct { + Known bool `json:"known"` + } `json:"charger"` + CommandedReason string `json:"commanded_reason"` + ControlFeedback struct { + Status string `json:"status"` // following, not_following, limited... + Severity string `json:"severity"` + Reason string `json:"reason"` + } `json:"control_feedback"` +} + +type ftwLine struct { + T float64 `json:"t"` + Status *ftwStatus `json:"status"` + LPs *struct { + Loadpoints []ftwLoadpoint `json:"loadpoints"` + } `json:"lps"` +} + +// sample is one second: the house's truth and the last FTW reading at or +// before it. +type sample struct { + T float64 + PhaseA [3]float64 + Damage [3]float64 + Blown [3]bool + CarW []float64 // by loadpoint, in lpIDs order + BatW float64 + Bat plantDevice // the inverter + Status *ftwStatus + LPs []ftwLoadpoint +} + +type frame struct { + T float64 `json:"t"` + Dir string `json:"dir"` + Frame json.RawMessage `json:"frame"` +} + +// houseEventLine is a change the house made, as its record has it. +type houseEventLine struct { + T float64 + Event houseEvent +} + +// runData is one finished scenario's record. +type runData struct { + dir string + samples []*sample + marks map[string]float64 + allMarks []markLine // every mark, in order, also repeated ones + events []houseEventLine + frames []frame // garage's charger + meta map[string]any +} + +type markLine struct { + T float64 + Label string +} + +func loadRun(dir string, meta map[string]any) (*runData, error) { + d := &runData{dir: dir, marks: map[string]float64{}, meta: meta} + var ftw []ftwLine + if err := readLines(filepath.Join(dir, "ftw.jsonl"), func(b []byte) { + var l ftwLine + if json.Unmarshal(b, &l) == nil { + ftw = append(ftw, l) + } + }); err != nil { + return nil, err + } + next := 0 + var last *ftwLine + err := readLines(filepath.Join(dir, "house", "plant.jsonl"), func(b []byte) { + var l plantLine + if json.Unmarshal(b, &l) != nil { + return + } + switch { + case l.Mark != "": + if _, ok := d.marks[l.Mark]; !ok { + d.marks[l.Mark] = l.T + } + d.allMarks = append(d.allMarks, markLine{l.T, l.Mark}) + return + case l.Event != nil: + var ev houseEvent + if json.Unmarshal(l.Event, &ev) == nil { + d.events = append(d.events, houseEventLine{l.T, ev}) + } + return + case l.Fuse == nil: + return + } + for next < len(ftw) && ftw[next].T <= l.T { + last = &ftw[next] + next++ + } + s := &sample{T: l.T, PhaseA: l.PhaseA, Damage: l.Fuse.Damage, Blown: l.Fuse.Blown, + BatW: l.Devices["inverter"].BatteryW, Bat: l.Devices["inverter"]} + for _, id := range lpIDs { + if dev, ok := l.Devices[id]; ok { + s.CarW = append(s.CarW, dev.W) + } + } + // A reading older than 3 s is no reading: FTW was down or slow. + if last != nil && l.T-last.T <= 3 { + s.Status = last.Status + if last.LPs != nil { + s.LPs = last.LPs.Loadpoints + } + } + d.samples = append(d.samples, s) + }) + if err != nil { + return nil, err + } + d.frames = d.loadFrames(0) + return d, nil +} + +// loadFrames is the OCPP frames charger i read and wrote, from its own log. +func (d *runData) loadFrames(i int) []frame { + var out []frame + readLines(filepath.Join(d.dir, "house", "ocpp-"+lpIDs[i]+".jsonl"), func(b []byte) { + var f frame + if json.Unmarshal(b, &f) == nil { + out = append(out, f) + } + }) + return out +} + +func readLines(path string, fn func([]byte)) error { + f, err := os.Open(path) + if err != nil { + if os.IsNotExist(err) { + return nil + } + return err + } + defer f.Close() + sc := bufio.NewScanner(f) + sc.Buffer(make([]byte, 1<<20), 64<<20) + for sc.Scan() { + fn(sc.Bytes()) + } + return sc.Err() +} + +func (d *runData) metaF(k string) float64 { + v, _ := d.meta[k].(float64) + if i, ok := d.meta[k].(int); ok { + v = float64(i) + } + return v +} + +// longestGap is the longest time between two samples, and when it began. +func (d *runData) longestGap() (gap, at float64) { + for i := 1; i < len(d.samples); i++ { + if g := d.samples[i].T - d.samples[i-1].T; g > gap { + gap, at = g, d.samples[i-1].T-d.samples[0].T + } + } + return gap, at +} + +// ---- per-sample readings ---- + +func carW(r *sample, i int) float64 { + if i < len(r.CarW) { + return r.CarW[i] + } + return 0 +} + +// phaseI is the current through each phase's fuse, + into the site. +func phaseI(r *sample) [3]float64 { return r.PhaseA } + +func maxAbsI(r *sample) float64 { + m := 0.0 + for _, i := range r.PhaseA { + m = math.Max(m, math.Abs(i)) + } + return m +} + +func damage(r *sample) float64 { return math.Max(r.Damage[0], math.Max(r.Damage[1], r.Damage[2])) } + +func blown(r *sample) bool { return r.Blown[0] || r.Blown[1] || r.Blown[2] } + +func firstLP(r *sample) ftwLoadpoint { + if len(r.LPs) > 0 { + return r.LPs[0] + } + return ftwLoadpoint{} +} + +func reasons(r *sample) []string { + lp := firstLP(r) + set := map[string]bool{} + for _, k := range []string{lp.CommandedReason, lp.ControlFeedback.Reason} { + if k != "" { + set[k] = true + } + } + out := make([]string, 0, len(set)) + for k := range set { + out = append(out, k) + } + return out +} + +func status(r *sample) ftwStatus { + if r.Status != nil { + return *r.Status + } + return ftwStatus{} +} + +func str(p *string) string { + if p == nil { + return "null" + } + return *p +} + +func allRead(r *sample) bool { + st := status(r) + if len(st.Drivers) == 0 { + return false + } + for _, d := range st.Drivers { + if d.Status != "ok" { + return false + } + } + return true +} + +func batSigned(r *sample) float64 { return r.BatW } + +// ---- windows ---- + +func (d *runData) window(t0, t1 float64) []*sample { + var out []*sample + for _, r := range d.samples { + if t0 <= r.T && r.T < t1 { + out = append(out, r) + } + } + return out +} + +// firstT is the seconds from t0 to the first sample from t0 on that +// satisfies pred, or nil. +func (d *runData) firstT(t0 float64, pred func(*sample) bool) *float64 { + for _, r := range d.samples { + if r.T >= t0 && pred(r) { + v := r.T - t0 + return &v + } + } + return nil +} + +// runsOver is the longest contiguous stretch and the total seconds with the +// worst phase above limit. +func (d *runData) runsOver(t0, t1, limit float64) (longest, total float64) { + var curStart, prevT *float64 + for _, r := range d.window(t0, t1) { + t := r.T + if maxAbsI(r) > limit { + if curStart == nil { + curStart = &t + } + if prevT != nil { + total += t - *prevT + } + } else { + if curStart != nil { + longest = math.Max(longest, t-*curStart) + } + curStart = nil + } + prevT = &t + } + if curStart != nil && prevT != nil { + longest = math.Max(longest, *prevT-*curStart) + } + return round(longest, 1), round(total, 1) +} + +// i2Over is the seconds the worst phase's I² filter (tau 60 s) is above +// limit, and the filter's peak. +func (d *runData) i2Over(t0, t1, limit float64) (over, peak float64) { + const tau = 60.0 + var y, prev float64 + started := false + for _, r := range d.window(t0, t1) { + x := math.Pow(maxAbsI(r), 2) + if !started { + y, started = x, true + } else { + dt := r.T - prev + y += (1 - math.Exp(-dt/tau)) * (x - y) + if math.Sqrt(y) > limit { + over += dt + } + } + prev = r.T + peak = math.Max(peak, math.Sqrt(y)) + } + return round(over, 1), round(peak, 2) +} + +func mean(xs []float64) float64 { + if len(xs) == 0 { + return 0 + } + s := 0.0 + for _, x := range xs { + s += x + } + return s / float64(len(xs)) +} + +func meanOf(ws []*sample, f func(*sample) float64) float64 { + xs := make([]float64, len(ws)) + for i, r := range ws { + xs[i] = f(r) + } + return mean(xs) +} + +func maxOf(ws []*sample, f func(*sample) float64) float64 { + m := 0.0 + for _, r := range ws { + m = math.Max(m, f(r)) + } + return m +} + +func round(x float64, n int) float64 { + p := math.Pow(10, float64(n)) + return math.Round(x*p) / p +} + +func roundp(p *float64, n int) *float64 { + if p == nil { + return nil + } + v := round(*p, n) + return &v +} + +// counter counts strings and lists the commonest first. +type counter map[string]int + +func (c counter) top(n int) map[string]int { + type kv struct { + k string + v int + } + var all []kv + for k, v := range c { + all = append(all, kv{k, v}) + } + sort.Slice(all, func(i, j int) bool { + if all[i].v != all[j].v { + return all[i].v > all[j].v + } + return all[i].k < all[j].k + }) + out := map[string]int{} + for i, x := range all { + if i >= n { + break + } + out[x.k] = x.v + } + return out +} + +// ---- OCPP profiles FTW sent ---- + +type profile struct { + T float64 `json:"t"` + Conn int `json:"conn"` + Purpose string `json:"purpose"` + Unit string `json:"unit"` + Limit float64 `json:"limit"` + Phases *float64 `json:"phases"` +} + +func (p profile) phasesOr3() float64 { + if p.Phases == nil || *p.Phases == 0 { + return 3 + } + return *p.Phases +} + +func (p profile) amps() bool { return strings.HasPrefix(strings.ToUpper(p.Unit), "A") } + +func (p profile) offerW() float64 { + if p.amps() { + return p.Limit * volt * p.phasesOr3() + } + return p.Limit +} + +// perPhaseA is the offer in amperes per phase. +func (p profile) perPhaseA() float64 { + if p.amps() { + return p.Limit + } + return p.Limit / (volt * p.phasesOr3()) +} + +// profiles lists the SetChargingProfile CALLs FTW sent in [t0, t1), timed +// from t0. +func profiles(frames []frame, t0, t1 float64) []profile { + var out []profile + for _, f := range frames { + if f.T < t0 || f.T >= t1 || f.Dir != "cs->cp" { + continue + } + var arr []json.RawMessage + if json.Unmarshal(f.Frame, &arr) != nil || len(arr) < 4 { + continue + } + var typ int + var action string + json.Unmarshal(arr[0], &typ) + json.Unmarshal(arr[2], &action) + if typ != 2 || action != "SetChargingProfile" { + continue + } + var req struct { + ConnectorID int `json:"connectorId"` + Profiles struct { + Purpose string `json:"chargingProfilePurpose"` + Schedule struct { + Unit string `json:"chargingRateUnit"` + Periods []struct { + Limit float64 `json:"limit"` + Phases *float64 `json:"numberPhases"` + } `json:"chargingSchedulePeriod"` + } `json:"chargingSchedule"` + } `json:"csChargingProfiles"` + } + json.Unmarshal(arr[3], &req) + p := profile{T: round(f.T-t0, 1), Conn: req.ConnectorID, Purpose: req.Profiles.Purpose, Unit: req.Profiles.Schedule.Unit} + if per := req.Profiles.Schedule.Periods; len(per) > 0 { + p.Limit, p.Phases = per[0].Limit, per[0].Phases + } + out = append(out, p) + } + return out +} + +func connector1(ps []profile) []profile { + var out []profile + for _, p := range ps { + if p.Conn == 1 { + out = append(out, p) + } + } + return out +} + +// ---- verdicts ---- + +// metrics are a run's numbers, written to result.json. +type metrics map[string]any + +// criterion is one pass condition. A scenario's known map names the +// criteria that fail on master and why. +type criterion struct { + Name string `json:"name"` + OK bool `json:"ok"` +} + +func crit(name string, ok bool) criterion { return criterion{Name: name, OK: ok} } + +// ---- shared summaries ---- + +func (d *runData) carSummary(t0, t1, targetW float64) metrics { + w := d.window(t0, t1) + ts := d.firstT(t0, func(r *sample) bool { return carW(r, 0) >= 0.95*minW }) + tm := d.firstT(t0, func(r *sample) bool { return carW(r, 0) >= 0.98*targetW }) + var hold *float64 + if tm != nil { + var after []*sample + for _, r := range w { + if r.T >= t0+*tm { + after = append(after, r) + } + } + n := 0 + for _, r := range after { + if carW(r, 0) >= 0.95*targetW { + n++ + } + } + h := round(float64(n)/math.Max(1, float64(len(after))), 3) + hold = &h + } + drops, on := 0, false + for _, r := range w { + c := carW(r, 0) + if c >= 0.9*minW { + on = true + } else if c < 500 && on { + drops++ + on = false + } + } + rs := counter{} + for _, r := range w { + for _, x := range reasons(r) { + rs[x]++ + } + } + fuseLimit := 0 + for k, v := range rs { + if strings.Contains(k, "fuse") { + fuseLimit += v + } + } + var last60 []float64 + for _, r := range w { + if r.T >= t1-60 { + last60 = append(last60, carW(r, 0)) + } + } + var minAfter *float64 + if ts != nil { + m := math.Inf(1) + for _, r := range w { + if r.T >= t0+*ts { + m = math.Min(m, carW(r, 0)) + } + } + m = math.Round(m) + minAfter = &m + } + return metrics{ + "time_to_start_s": roundp(ts, 1), + "time_to_target_s": roundp(tm, 1), + "hold_fraction": hold, + "stops_after_start": drops, + "car_w_last60_mean": math.Round(mean(last60)), + "car_w_min_after_start": minAfter, + "fuse_limit_samples": fuseLimit, + "reasons": rs.top(6), + "max_abs_i": round(maxOf(w, maxAbsI), 2), + "fuse_damage_max": round(maxOf(w, damage), 4), + "fuse_blown": anyBlown(w), + } +} + +func anyBlown(ws []*sample) bool { + for _, r := range ws { + if blown(r) { + return true + } + } + return false +} + +func le(p *float64, limit float64) bool { return p != nil && *p <= limit } + +func (d *runData) overage(t0, t1, fuse float64) metrics { + w := d.window(t0, t1) + m := metrics{"max_i": round(maxOf(w, maxAbsI), 2)} + m["over_limit_longest_s"], m["over_limit_total_s"] = d.runsOver(t0, t1, fuse-1) + m["over_fuse_longest_s"], _ = d.runsOver(t0, t1, fuse) + m["fuse_damage_max"] = round(maxOf(w, damage), 4) + m["fuse_blown"] = anyBlown(w) + return m +} + +func (d *runData) phaseMean(t0, t1 float64) [3]float64 { + w := d.window(t0, t1) + var out [3]float64 + for k := 0; k < 3; k++ { + out[k] = round(meanOf(w, func(r *sample) float64 { return math.Abs(phaseI(r)[k]) }), 1) + } + return out +} + +// disconnects lists "OCPP charger disconnected" lines in FTW's log between +// t0 and t1, in seconds from t0. +func (d *runData) disconnects(t0, t1 float64) []float64 { + out := []float64{} + readLines(filepath.Join(d.dir, "ftw.log"), func(b []byte) { + line := string(b) + if !strings.Contains(line, "OCPP charger disconnected") || !strings.HasPrefix(line, "time=") { + return + } + ts, _, _ := strings.Cut(strings.TrimPrefix(line, "time="), " ") + t, err := time.Parse(time.RFC3339Nano, ts) + if err != nil { + return + } + at := float64(t.UnixMicro()) / 1e6 + if t0 <= at && at <= t1 { + out = append(out, round(at-t0, 1)) + } + }) + return out +} diff --git a/go/test/site/procattr_linux_test.go b/go/test/site/procattr_linux_test.go new file mode 100644 index 000000000..7fa4fff10 --- /dev/null +++ b/go/test/site/procattr_linux_test.go @@ -0,0 +1,9 @@ +package site + +import "syscall" + +// childProcAttr puts a child in its own process group and kills it if the +// test binary dies, as when go test's -timeout fires. +func childProcAttr() *syscall.SysProcAttr { + return &syscall.SysProcAttr{Setpgid: true, Pdeathsig: syscall.SIGKILL} +} diff --git a/go/test/site/procattr_other_test.go b/go/test/site/procattr_other_test.go new file mode 100644 index 000000000..c38c4e548 --- /dev/null +++ b/go/test/site/procattr_other_test.go @@ -0,0 +1,11 @@ +//go:build unix && !linux + +package site + +import "syscall" + +// childProcAttr puts a child in its own process group. Without Pdeathsig a +// child outlives a test binary that dies; stop it by its run directory. +func childProcAttr() *syscall.SysProcAttr { + return &syscall.SysProcAttr{Setpgid: true} +} diff --git a/go/test/site/random_test.go b/go/test/site/random_test.go new file mode 100644 index 000000000..aeaa2b10d --- /dev/null +++ b/go/test/site/random_test.go @@ -0,0 +1,549 @@ +//go:build unix + +package site + +import ( + "encoding/json" + "fmt" + "math" + "math/rand" + "os" + "path/filepath" + "sort" + "strings" + "time" +) + +// A random case draws FTW's side from a seed: the fuse, the house, the +// loadpoint's settings and FTW's requests (Charge now, a pause, a plan, a +// Core restart, an import ceiling) at random times. The house draws its own +// timeline from the same seed (device-simulator's random block): load steps +// and spikes, PV clouds, a battery at its floor, a car taking less, slow +// chargers, silent devices, reconnects, a late or glitched meter, signed +// currents. FTW's invariants are checked every second. The house writes +// the timeline it drew, so a failing seed can be replayed and kept. + +// randomDurationS is how long a random case runs. +const randomDurationS = 1200.0 + +// randomTolerated are invariants a seed may break for a finding still open +// on master. A fix deletes its line here. +var randomTolerated = map[string]string{ + "no discharge asked at the battery's floor": "ev-099: the fuse guard drives the battery below its floor; fix unmerged", +} + +// ftwAction is one of FTW's requests in a random case. +type ftwAction struct { + AtS float64 `json:"at_s"` + Do string `json:"do"` // charge_now, pause, clear, plan, restart, ceiling + W float64 `json:"w,omitempty"` + PhaseMode string `json:"phase_mode,omitempty"` + DownS float64 `json:"down_s,omitempty"` + CeilingW float64 `json:"ceiling_w,omitempty"` + SoCPct float64 `json:"soc_pct,omitempty"` + GoalPct float64 `json:"goal_pct,omitempty"` + ReadyMin float64 `json:"ready_min,omitempty"` +} + +// randomPlan is everything a random case does, so it can be saved and run +// again: House is set once the house has drawn its timeline. +type randomPlan struct { + Seed int64 `json:"seed"` + FuseA float64 `json:"fuse_a"` + HouseA [3]float64 `json:"house_a"` + PVW float64 `json:"pv_w"` + SoCPct float64 `json:"soc_pct"` + LPExtra map[string]any `json:"lp_extra,omitempty"` + Actions []ftwAction `json:"actions"` + DurationS float64 `json:"duration_s"` + House *houseSpec `json:"house,omitempty"` + Found *foundNote `json:"found,omitempty"` +} + +// foundNote says what a saved case failed and when it was found. +type foundNote struct { + Invariant string `json:"invariant"` + On string `json:"on"` // the date + Commit string `json:"commit,omitempty"` +} + +func drawPlan(seed int64) randomPlan { + rng := rand.New(rand.NewSource(seed)) + pick := func(xs ...float64) float64 { return xs[rng.Intn(len(xs))] } + between := func(lo, hi float64) float64 { return math.Round((lo+rng.Float64()*(hi-lo))*10) / 10 } + p := randomPlan{Seed: seed, DurationS: randomDurationS, + FuseA: pick(16, 20, 25), PVW: pick(0, 0, 2000, 5000, 8000), SoCPct: pick(6, 15, 40, 70, 95)} + for k := range p.HouseA { + p.HouseA[k] = between(0.5, p.FuseA/2) + } + p.LPExtra = map[string]any{} + switch rng.Intn(3) { + case 1: + p.LPExtra["phase_mode"] = "auto" + case 2: + p.LPExtra["phase_mode"] = "3p" + } + if rng.Intn(2) == 0 { + p.LPExtra["allowed_steps_w"] = easeeSteps + } + for t := 40 + between(0, 60); t < p.DurationS-60; t += between(60, 300) { + a := ftwAction{AtS: t} + switch rng.Intn(10) { + case 0, 1, 2: + a.Do, a.W, a.PhaseMode = "charge_now", pick(4140, 7360, 11040), "3p" + case 3: + a.Do, a.W, a.PhaseMode = "charge_now", pick(1380, 3680), "1p" + case 4: + a.Do = "pause" + case 5: + a.Do = "clear" + case 6: + a.Do, a.SoCPct, a.GoalPct, a.ReadyMin = "plan", between(20, 50), between(60, 90), pick(30, 60, 240) + case 7: + a.Do, a.DownS = "restart", between(5, 30) + case 8: + a.Do, a.CeilingW = "ceiling", pick(0, 3000, 5000, 8000) + default: + continue + } + p.Actions = append(p.Actions, a) + } + return p +} + +// randomScenario runs a plan drawn from seed, or a saved one. +func randomScenario(p randomPlan) scenario { + name, dir := fmt.Sprintf("r%d", p.Seed), fmt.Sprintf("random-%d", p.Seed) + sc := scenario{name: name, dir: dir, minutes: p.DurationS/60 + 1, + run: func(c *runCtx) (map[string]any, error) { return runPlan(c, p) }, + judge: judgeInvariants, + tolerate: randomTolerated} + if p.Found != nil { + sc.name, sc.dir = "found-"+fmt.Sprint(p.Seed), "found-"+fmt.Sprint(p.Seed) + sc.known = map[string]string{p.Found.Invariant: fmt.Sprintf("found by random seed %d on %s; triage it", p.Seed, p.Found.On)} + } + return sc +} + +func runPlan(c *runCtx, p randomPlan) (map[string]any, error) { + loads := [3]float64{p.HouseA[0] * volt, p.HouseA[1] * volt, p.HouseA[2] * volt} + spec := stackSpec{fuseA: p.FuseA, loadsW: loads, pvW: p.PVW, socPct: p.SoCPct, lpExtra: p.LPExtra} + if p.House != nil { + spec.house = p.House + } else { + spec.random = &houseRandom{Seed: p.Seed, DurationS: p.DurationS, GapS: 45} + } + plan, _ := json.MarshalIndent(p, "", " ") + err := with(c, spec, func(st *stack) error { + os.WriteFile(filepath.Join(st.rundir, "plan.json"), plan, 0o644) + if err := st.plug(0); err != nil { + return err + } + t0 := time.Now() + st.mark("start") + for _, a := range p.Actions { + sleepS(time.Until(t0.Add(time.Duration(a.AtS * float64(time.Second)))).Seconds()) + st.mark("ftw:" + a.Do) + var err error + switch a.Do { + case "charge_now": + err = st.chargeNow("garage", 0, a.PhaseMode, a.W) + case "pause": + err = st.chargeNow("garage", 0, "3p", -1) + case "clear": + _, err = call("DELETE", st.ftw+"/api/loadpoints/garage/manual_hold", nil, nil) + case "plan": + _, err = st.planTo("garage", a.SoCPct, a.GoalPct, time.Duration(a.ReadyMin*float64(time.Minute))) + case "restart": + st.stopFTW() + sleepS(a.DownS) + err = st.startFTW() + st.mark("ftw:up") + case "ceiling": + err = st.post("/api/peak_import_ceiling", map[string]any{"peak_import_ceiling_w": a.CeilingW}, nil) + } + if err != nil { + st.mark("ftw:error") + } + } + sleepS(time.Until(t0.Add(time.Duration(p.DurationS * float64(time.Second)))).Seconds()) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "random", "seed": p.Seed, "fuse": p.FuseA}, err +} + +// ---- invariants, every second ---- + +// houseTiming is what the house picked for the charger, from the spec it +// ran (timeline.json for a random house). +type houseTiming struct { + commandS, startupS float64 + floorPct float64 +} + +func (d *runData) timing() houseTiming { + h := houseTiming{floorPct: 5} + for _, name := range []string{"timeline.json", "../house.json"} { + b, err := os.ReadFile(filepath.Join(d.dir, "house", name)) + if err != nil { + continue + } + var spec houseSpec + if json.Unmarshal(b, &spec) != nil { + continue + } + for _, dev := range spec.Devices { + if dev.ID == "garage" { + if dev.CommandResponseS != nil { + h.commandS = *dev.CommandResponseS + } + if dev.StartupDelayS != nil { + h.startupS = *dev.StartupDelayS + } + } + if dev.ID == "inverter" && dev.MinSoCPct != nil { + h.floorPct = *dev.MinSoCPct + } + } + break + } + return h +} + +// eventWindow is when the house changed on its own, and how long FTW may +// take to answer it. +type eventWindow struct{ from, to float64 } + +func judgeInvariants(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + limit := fuse - 1 + h := d.timing() + t0, t1 := d.marks["start"], d.marks["end"] + run := d.window(t0, t1) + m := metrics{"charger_command_s": h.commandS, "charger_startup_s": h.startupS, "events": len(d.events)} + + // Windows in which an over-limit second is the house's doing. + var windows []eventWindow + faultUntil := map[float64]float64{} // a fault's start and end + for _, ev := range d.events { + batteryUsable := true + if r := d.at(ev.T); r != nil && r.Bat.SoCPct <= h.floorPct+1 { + batteryUsable = false + } + // The fastest lever: the battery answers in 2 s, the charger in its + // own time, never under 2 s; then the meter and tick, 2 s, and 3 s. + lever := math.Max(2, h.commandS) + if batteryUsable { + lever = 2 + } + allow := lever + 2 + 3 + switch ev.Event.Do { + case "meter_late", "silence", "offline", "reconnect": + allow += ev.Event.ForS + faultUntil[ev.T] = ev.T + ev.Event.ForS + case "meter_glitch": + faultUntil[ev.T] = ev.T + 5 + } + windows = append(windows, eventWindow{ev.T, ev.T + allow}) + } + for _, mk := range d.allMarks { + if mk.Label == "ftw:restart" { + windows = append(windows, eventWindow{mk.T, mk.T + 60}) + } + } + explained := func(t float64) (float64, bool) { + best, ok := 0.0, false + for _, w := range windows { + if t >= w.from && t <= w.to { + best, ok = math.Max(best, w.to), true + } + } + return best, ok + } + + // 1, 2: the fuse holds; over the limit only as long as an outside step + // allows, and at most 3 s after FTW's own raise. + blownAny := anyBlown(run) + unexplained, longestOwn := 0.0, 0.0 + var firstBad float64 + var start *sample + for i, r := range run { + over := maxAbsI(r) > limit + if over && start == nil { + start = r + } + if (!over || i == len(run)-1) && start != nil { + end := r.T + if to, ok := explained(start.T); ok { + if end > to { + unexplained = math.Max(unexplained, end-to) + if firstBad == 0 { + firstBad = to + } + } + } else if end-start.T > 3 { + longestOwn = math.Max(longestOwn, end-start.T) + if firstBad == 0 { + firstBad = start.T + 3 + } + } + start = nil + } + } + m["over_limit_after_steps_s"], m["over_limit_own_s"] = round(unexplained, 1), round(longestOwn, 1) + + // 3: with every device reading and no fault, Home use is measured. + inFault := func(t float64) bool { + for from, to := range faultUntil { + if t >= from && t <= to+30 { + return true + } + } + return false + } + restartAt := []float64{t0} + for _, mk := range d.allMarks { + if mk.Label == "ftw:up" { + restartAt = append(restartAt, mk.T) + } + } + nearRestart := func(t float64) bool { + for _, r := range restartAt { + if t >= r && t <= r+60 { + return true + } + } + return false + } + first := map[string]float64{} + note := func(name string, t float64) { + if _, ok := first[name]; !ok { + first[name] = t + } + } + if firstBad > 0 { + note("limit", firstBad) + } + for _, r := range run { + if blown(r) { + note("fuse", r.T) + break + } + } + dishonest, run3 := 0.0, 0.0 + for i, r := range run { + bad := r.Status != nil && allRead(r) && !inFault(r.T) && !nearRestart(r.T) && str(status(r).LoadQuality) != "measured" + if bad && i > 0 { + run3 += r.T - run[i-1].T + dishonest = math.Max(dishonest, run3) + if dishonest > 10 { + note("home use", r.T) + } + } else { + run3 = 0 + } + } + m["home_use_estimated_while_reading_s"] = round(dishonest, 1) + + // 4: during Charge now the car charges when there is room for it. + var holds []eventWindow + var holdFrom float64 + for _, mk := range d.allMarks { + switch mk.Label { + case "ftw:charge_now": + if holdFrom == 0 { + holdFrom = mk.T + } + case "ftw:pause", "ftw:clear", "ftw:plan", "end": + if holdFrom > 0 { + holds = append(holds, eventWindow{holdFrom, mk.T}) + holdFrom = 0 + } + } + } + grace := 60 + h.commandS + h.startupS + 10 + idle, run4 := 0.0, 0.0 + for _, hw := range holds { + run4 = 0 + for i, r := range d.window(hw.from+grace, hw.to) { + roomA := limit - (maxAbsI(r) - carW(r, 0)/(3*volt)) + bad := carW(r, 0) < 0.5*1380 && roomA >= 7 && !inFault(r.T) && !nearRestart(r.T) + if bad && i > 0 { + run4++ + idle = math.Max(idle, run4) + if idle > 30 { + note("charge now", r.T) + } + } else { + run4 = 0 + } + } + } + m["charge_now_idle_with_room_s"] = idle + + // 5: no order above the charger's 16 A per phase. + over := 0 + for _, p := range connector1(profiles(d.frames, t0, t1)) { + if p.Limit > 0 && p.perPhaseA() > 16+1e-6 { + over++ + note("rating", t0+p.T) + } + } + m["offers_over_rating"] = over + + // 6: FTW does not ask the battery to discharge at its floor. + floorS := 0.0 + for _, r := range run { + if r.Bat.EMSCmd == 187 && r.Bat.EMSSetW > 50 && r.Bat.SoCPct <= h.floorPct+0.5 { + floorS++ + if floorS > 10 { + note("floor", r.T) + } + } + } + m["discharge_at_floor_s"] = floorS + + // 7: a stale meter means no raise. + raises := 0 + prs := connector1(profiles(d.frames, t0, t1)) + for _, ev := range d.events { + if ev.Event.Do != "meter_late" { + continue + } + for i := 1; i < len(prs); i++ { + at := t0 + prs[i].T + if at > ev.T+2 && at <= ev.T+ev.Event.ForS && prs[i].offerW() > prs[i-1].offerW()+1 { + raises++ + note("stale meter", at) + } + } + } + m["raises_on_a_stale_meter"] = raises + // The first second an invariant broke, from the start, per invariant: + // a kept seed is cut a minute after the one it failed on. + firsts := map[string]float64{} + for k, t := range first { + firsts[k] = round(t-t0, 1) + } + m["first_violation_s"] = firsts + return m, []criterion{ + crit("fuse holds", !blownAny), + crit("over the limit no longer than an outside step allows", unexplained == 0), + crit("over the limit at most 3 s after FTW's own raise", longestOwn == 0), + crit("Home use measured while every device reads", dishonest <= 10), + crit("Charge now charges when there is room", idle <= 30), + crit("no offer above the charger's rating", over == 0), + crit("no discharge asked at the battery's floor", floorS <= 10), + crit("no raise on a stale meter", raises == 0), + } +} + +// ---- seeds, replays and saved cases ---- + +// randomSet parses FTW_SITE=random:: (or random:). +func randomSet(which string) ([]scenario, bool) { + if !strings.HasPrefix(which, "random:") { + return nil, false + } + var first, count int64 = 0, 1 + parts := strings.Split(strings.TrimPrefix(which, "random:"), ":") + fmt.Sscan(parts[0], &first) + if len(parts) > 1 { + fmt.Sscan(parts[1], &count) + } + var out []scenario + for s := first; s < first+count; s++ { + out = append(out, randomScenario(drawPlan(s))) + } + return out, true +} + +// foundScenarios are saved failing seeds (testdata/found/*.json): fixed +// cases in the full set, known until someone triages them. +func foundScenarios(root string) []scenario { + files, _ := filepath.Glob(filepath.Join(root, "go", "test", "site", "testdata", "found", "*.json")) + sort.Strings(files) + var out []scenario + for _, f := range files { + b, err := os.ReadFile(f) + if err != nil { + continue + } + var p randomPlan + if json.Unmarshal(b, &p) == nil && p.House != nil && p.Found != nil { + out = append(out, randomScenario(p)) + } + } + return out +} + +// invariantKey ties each criterion to its first_violation_s entry. +var invariantKey = map[string]string{ + "fuse holds": "fuse", + "over the limit no longer than an outside step allows": "limit", + "over the limit at most 3 s after FTW's own raise": "limit", + "Home use measured while every device reads": "home use", + "Charge now charges when there is room": "charge now", + "no offer above the charger's rating": "rating", + "no discharge asked at the battery's floor": "floor", + "no raise on a stale meter": "stale meter", +} + +// keepFailure writes a failing random run as a case to keep: its plan with +// the house's timeline, both cut a minute after the first violation. +func keepFailure(rundir, out string, o outcome) string { + b, err := os.ReadFile(filepath.Join(rundir, "plan.json")) + if err != nil { + return "" + } + var p randomPlan + if json.Unmarshal(b, &p) != nil { + return "" + } + var h houseSpec + if hb, err := os.ReadFile(filepath.Join(rundir, "house", "timeline.json")); err == nil && json.Unmarshal(hb, &h) == nil { + p.House = &h + } else if p.House == nil { + return "" + } + failed := "" + for _, c := range o.Criteria { + if _, tolerated := randomTolerated[c.Name]; !c.OK && !tolerated { + failed = c.Name + break + } + } + var res struct { + Metrics map[string]any `json:"metrics"` + } + if rb, err := os.ReadFile(filepath.Join(rundir, "result.json")); err == nil && json.Unmarshal(rb, &res) == nil { + firsts, _ := res.Metrics["first_violation_s"].(map[string]any) + at, ok := firsts[invariantKey[failed]].(float64) + if ok && at+60 < p.DurationS { + p.DurationS = math.Ceil(at + 60) + var evs []houseEvent + for _, ev := range p.House.Timeline { + if ev.AtS <= p.DurationS { + evs = append(evs, ev) + } + } + p.House.Timeline = evs + var acts []ftwAction + for _, a := range p.Actions { + if a.AtS <= p.DurationS { + acts = append(acts, a) + } + } + p.Actions = acts + } + } + p.Found = &foundNote{Invariant: failed, On: time.Now().Format("2006-01-02")} + dst := filepath.Join(out, "found", fmt.Sprintf("%d.json", p.Seed)) + os.MkdirAll(filepath.Dir(dst), 0o755) + nb, _ := json.MarshalIndent(p, "", " ") + if os.WriteFile(dst, nb, 0o644) != nil { + return "" + } + return dst +} diff --git a/go/test/site/rig_test.go b/go/test/site/rig_test.go new file mode 100644 index 000000000..35184ffb2 --- /dev/null +++ b/go/test/site/rig_test.go @@ -0,0 +1,413 @@ +//go:build unix + +// Package site runs the real FTW binary against srcfl/device-simulator: a +// Sungrow hybrid over Modbus, an SDM630 site meter behind an IEC gG fuse on +// each phase, and OCPP 1.6 chargers. Each scenario builds a fresh house and +// a fresh state.db, drives it, records the simulator and FTW once a second +// and judges the run. See README.md for the scenarios and how to run them. +package site + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "sort" + "strconv" + "strings" + "sync" + "testing" + "time" +) + +// rigEnv is what every scenario shares. +type rigEnv struct { + root string // repository root: drivers/ and web/ + ftwBin string + devsimBin string + out string +} + +func repoRoot(t *testing.T) string { + _, file, _, _ := runtime.Caller(0) + dir := filepath.Dir(file) + for i := 0; i < 6; i++ { + if _, err := os.Stat(filepath.Join(dir, "Makefile")); err == nil { + return dir + } + dir = filepath.Dir(dir) + } + t.Fatal("could not locate the repository root") + return "" +} + +// devsimCommit is the device-simulator commit the rig runs, from +// devsim.version. +func devsimCommit(t *testing.T, root string) string { + b, err := os.ReadFile(filepath.Join(root, "go", "test", "site", "devsim.version")) + if err != nil { + t.Fatal(err) + } + c := strings.TrimSpace(string(b)) + if len(c) != 40 { + t.Fatalf("devsim.version: want a full commit hash, got %q", c) + } + return c +} + +func mustRun(t *testing.T, dir string, name string, args ...string) { + t.Helper() + cmd := exec.Command(name, args...) + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("%s %s: %v\n%s", name, strings.Join(args, " "), err, out) + } +} + +// buildDevsim builds the pinned device-simulator, from FTW_SITE_DEVSIM_SRC +// when set (it must be at the pinned commit) or from a clone in the cache. +func buildDevsim(t *testing.T, root, bin string) { + commit := devsimCommit(t, root) + src := os.Getenv("FTW_SITE_DEVSIM_SRC") + if src == "" { + cache, err := os.UserCacheDir() + if err != nil { + t.Fatal(err) + } + src = filepath.Join(cache, "ftw-site", "device-simulator-"+commit[:12]) + if _, err := os.Stat(filepath.Join(src, "go.mod")); err != nil { + os.RemoveAll(src) + mustRun(t, "", "git", "init", "-q", src) + mustRun(t, src, "git", "fetch", "-q", "--depth=1", "https://github.com/srcfl/device-simulator", commit) + mustRun(t, src, "git", "checkout", "-q", "FETCH_HEAD") + } + } + cmd := exec.Command("git", "rev-parse", "HEAD") + cmd.Dir = src + head, err := cmd.Output() + if err != nil || strings.TrimSpace(string(head)) != commit { + t.Fatalf("device-simulator in %s is at %s, not the pinned %s", src, strings.TrimSpace(string(head)), commit) + } + cmd = exec.Command("go", "build", "-o", bin, "./cmd/simulator/") + cmd.Dir = src + cmd.Env = append(os.Environ(), "CGO_ENABLED=0") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("build device-simulator: %v\n%s", err, out) + } +} + +func setup(t *testing.T) *rigEnv { + root := repoRoot(t) + for _, f := range []string{"sdm630.lua", "sungrow.lua"} { + if _, err := os.Stat(filepath.Join(root, "drivers", f)); err != nil { + t.Fatalf("drivers/%s missing: run make drivers first", f) + } + } + out := os.Getenv("FTW_SITE_OUT") + if out == "" { + var err error + if out, err = os.MkdirTemp("", "ftw-site-"); err != nil { + t.Fatal(err) + } + } + env := &rigEnv{root: root, out: out, + ftwBin: os.Getenv("FTW_SITE_FTW_BIN"), devsimBin: os.Getenv("FTW_SITE_DEVSIM_BIN")} + bin := filepath.Join(out, "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + if env.ftwBin == "" { + env.ftwBin = filepath.Join(bin, "ftw") + mustRun(t, filepath.Join(root, "go"), "go", "build", "-o", env.ftwBin, "./cmd/ftw") + } + if env.devsimBin == "" { + env.devsimBin = filepath.Join(bin, "devsim") + buildDevsim(t, root, env.devsimBin) + } + t.Logf("results in %s", out) + return env +} + +// selected resolves FTW_SITE: "short" (the pull-request set), "full" (every +// scenario and every saved random case), "random::", or +// a comma-separated list of names. +func selected(t *testing.T, which string) []scenario { + if rs, ok := randomSet(which); ok { + return rs + } + var out []scenario + switch which { + case "full", "all": + for _, sc := range scenarios { + if sc.name != "s7s-25" { // the full set runs the long s7 + out = append(out, sc) + } + } + out = append(out, foundScenarios(repoRoot(t))...) + case "short": + for _, sc := range scenarios { + if sc.short { + out = append(out, sc) + } + } + default: + byName := map[string]scenario{} + for _, sc := range scenarios { + byName[sc.name] = sc + } + for _, n := range strings.Split(which, ",") { + sc, ok := byName[strings.TrimSpace(n)] + if !ok { + t.Fatalf("unknown scenario %q", n) + } + out = append(out, sc) + } + } + return out +} + +// shard picks shard k of n (FTW_SITE_SHARD "k/n"). +func shard(t *testing.T, scs []scenario, spec string) []scenario { + if spec == "" { + return scs + } + k, n := 0, 0 + if _, err := fmt.Sscanf(spec, "%d/%d", &k, &n); err != nil || n < 1 || k < 1 || k > n { + t.Fatalf("FTW_SITE_SHARD %q: want k/n", spec) + } + return split(scs, n)[k-1] +} + +// split deals scenarios into n groups of about equal time: longest first, +// each to the group with the least time so far. +func split(scs []scenario, n int) [][]scenario { + sorted := append([]scenario(nil), scs...) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].minutes > sorted[j].minutes }) + load := make([]float64, n) + out := make([][]scenario, n) + for _, sc := range sorted { + best := 0 + for i := range load { + if load[i] < load[best] { + best = i + } + } + load[best] += sc.minutes + out[best] = append(out[best], sc) + } + return out +} + +// outcome is one scenario's line in summary.json. +type outcome struct { + Name string `json:"name"` + Dir string `json:"dir"` + Verdict string `json:"verdict"` // pass, known, fail + DurationS float64 `json:"duration_s"` + Criteria []criterion `json:"criteria"` + Known []string `json:"known,omitempty"` + Failures []string `json:"failures,omitempty"` + Error string `json:"error,omitempty"` +} + +// TestSite runs the scenarios FTW_SITE names. FTW_SITE_LANES runs that many +// at once on ports 100 apart from FTW_SITE_PORT_BASE (default 18700); +// FTW_SITE_SHARD splits a set over machines; FTW_SITE_OUT keeps the run +// directories. +func TestSite(t *testing.T) { + which := os.Getenv("FTW_SITE") + if which == "" { + t.Skip("set FTW_SITE=short, full or a list of scenarios; see go/test/site/README.md") + } + scs := shard(t, selected(t, which), os.Getenv("FTW_SITE_SHARD")) + base := 18700 + if s := os.Getenv("FTW_SITE_PORT_BASE"); s != "" { + var err error + if base, err = strconv.Atoi(s); err != nil { + t.Fatalf("FTW_SITE_PORT_BASE: %v", err) + } + } + lanes := 1 + if s := os.Getenv("FTW_SITE_LANES"); s != "" { + var err error + if lanes, err = strconv.Atoi(s); err != nil || lanes < 1 { + t.Fatalf("FTW_SITE_LANES %q", s) + } + } + env := setup(t) + var mu sync.Mutex + var outcomes []outcome + t.Run("lanes", func(t *testing.T) { + for k, group := range split(scs, lanes) { + k, group := k, group + t.Run(fmt.Sprintf("lane%d", k+1), func(t *testing.T) { + if lanes > 1 { + t.Parallel() + } + for _, sc := range group { + sc := sc + t.Run(sc.name, func(t *testing.T) { + o := runScenario(env, lane(base+100*k), sc) + mu.Lock() + outcomes = append(outcomes, o) + mu.Unlock() + if strings.HasPrefix(sc.dir, "random-") && o.Verdict == "fail" && o.Error == "" { + if kept := keepFailure(filepath.Join(env.out, sc.dir), env.out, o); kept != "" { + t.Logf("kept as %s; copy it to go/test/site/testdata/found/ to run it in the full set", kept) + } + } + t.Logf("%s in %.0f s: %v", o.Verdict, o.DurationS, o.Criteria) + for _, k := range o.Known { + t.Logf("fails as known: %s", k) + } + if o.Error != "" { + t.Fatal(o.Error) + } + for _, f := range o.Failures { + t.Error(f) + } + }) + } + }) + } + }) + sort.Slice(outcomes, func(i, j int) bool { return outcomes[i].Name < outcomes[j].Name }) + writeSummary(t, env.out, outcomes) +} + +func runScenario(env *rigEnv, l lane, sc scenario) outcome { + o := outcome{Name: sc.name, Dir: sc.dir, Verdict: "fail"} + rundir := filepath.Join(env.out, sc.dir) + t0 := time.Now() + meta, err := sc.run(&runCtx{env: env, lane: l, rundir: rundir}) + o.DurationS = time.Since(t0).Round(time.Second).Seconds() + if meta != nil { + b, _ := json.MarshalIndent(meta, "", " ") + os.WriteFile(filepath.Join(rundir, "meta.json"), b, 0o644) + } + if err != nil { + o.Error = err.Error() + return o + } + return judgeRun(sc, rundir, meta, o) +} + +// judgeRun judges a recorded run against the scenario's criteria and its +// known and tolerated lines, and writes result.json. +func judgeRun(sc scenario, rundir string, meta map[string]any, o outcome) outcome { + // Numbers go through JSON, as a rerun from the run directory sees them. + var m map[string]any + b, _ := json.Marshal(meta) + json.Unmarshal(b, &m) + d, err := loadRun(rundir, m) + if err != nil { + o.Error = err.Error() + return o + } + // The criteria count seconds. A host that sleeps or stalls (a laptop + // lid) stretches the run and leaves holes in the record. A Core restart + // costs a sample or two at most: FTW refuses at once or times out in 5 s. + if gap, at := d.longestGap(); gap > 15 { + o.Error = fmt.Sprintf("the recorder missed %.0f s at %.0f s into the run: the host paused, so the timing proves nothing", gap, at) + return o + } + numbers, criteria := sc.judge(d) + o.Criteria = criteria + for _, c := range criteria { + if finding, ok := sc.tolerate[c.Name]; ok { + if !c.OK { + o.Known = append(o.Known, fmt.Sprintf("%s: %s (tolerated)", c.Name, finding)) + } + continue + } + finding, known := sc.known[c.Name] + switch { + case known && c.OK: + o.Failures = append(o.Failures, fmt.Sprintf("%q passes now: delete its known line (%s)", c.Name, finding)) + case known: + o.Known = append(o.Known, fmt.Sprintf("%s: %s", c.Name, finding)) + case !c.OK: + o.Failures = append(o.Failures, fmt.Sprintf("%q fails", c.Name)) + } + } + for name := range sc.known { + found := false + for _, c := range criteria { + found = found || c.Name == name + } + if !found { + o.Failures = append(o.Failures, fmt.Sprintf("known line names no criterion: %q", name)) + } + } + switch { + case len(o.Failures) > 0: + o.Verdict = "fail" + case len(o.Known) > 0: + o.Verdict = "known" + default: + o.Verdict = "pass" + } + res, _ := json.MarshalIndent(map[string]any{"verdict": o.Verdict, "criteria": criteria, "known": o.Known, + "failures": o.Failures, "duration_s": o.DurationS, "metrics": numbers, "meta": meta, "samples": len(d.samples)}, "", " ") + os.WriteFile(filepath.Join(rundir, "result.json"), res, 0o644) + return o +} + +// TestJudge judges the recorded runs in FTW_SITE_JUDGE again with the +// criteria as they are now, without running anything: a changed criterion +// or known line is checked against a long run in seconds. +func TestJudge(t *testing.T) { + dir := os.Getenv("FTW_SITE_JUDGE") + if dir == "" { + t.Skip("set FTW_SITE_JUDGE to a results directory of an earlier run") + } + var outs []outcome + for _, sc := range selected(t, "full") { + rundir := filepath.Join(dir, sc.dir) + b, err := os.ReadFile(filepath.Join(rundir, "meta.json")) + if err != nil { + continue + } + var meta map[string]any + if err := json.Unmarshal(b, &meta); err != nil { + t.Fatalf("%s: %v", rundir, err) + } + o := outcome{Name: sc.name, Dir: sc.dir, Verdict: "fail"} + if b, err := os.ReadFile(filepath.Join(rundir, "result.json")); err == nil { + var prev struct { + DurationS float64 `json:"duration_s"` + } + json.Unmarshal(b, &prev) + o.DurationS = prev.DurationS + } + o = judgeRun(sc, rundir, meta, o) + t.Logf("%s: %s %v", sc.name, o.Verdict, append(o.Failures, o.Known...)) + outs = append(outs, o) + } + if len(outs) == 0 { + t.Fatalf("no recorded run of a scenario in %s", dir) + } + writeSummary(t, dir, outs) +} + +func writeSummary(t *testing.T, out string, outs []outcome) { + if len(outs) == 0 { + return + } + b, _ := json.MarshalIndent(outs, "", " ") + os.WriteFile(filepath.Join(out, "summary.json"), b, 0o644) + var md strings.Builder + md.WriteString("| Scenario | Verdict | Time | Failed or known |\n|---|---|---|---|\n") + for _, o := range outs { + notes := append(append([]string{}, o.Failures...), o.Known...) + if o.Error != "" { + notes = append([]string{o.Error}, notes...) + } + fmt.Fprintf(&md, "| %s | %s | %s | %s |\n", o.Name, o.Verdict, + (time.Duration(o.DurationS) * time.Second).String(), strings.ReplaceAll(strings.Join(notes, "; "), "|", "/")) + } + os.WriteFile(filepath.Join(out, "summary.md"), []byte(md.String()), 0o644) + t.Logf("summary:\n%s", md.String()) +} diff --git a/go/test/site/rig_unit_test.go b/go/test/site/rig_unit_test.go new file mode 100644 index 000000000..7e921c2ad --- /dev/null +++ b/go/test/site/rig_unit_test.go @@ -0,0 +1,141 @@ +//go:build unix + +package site + +import ( + "encoding/json" + "testing" +) + +// These run in every go test, without a simulator. + +func TestShardsCoverEveryScenarioOnce(t *testing.T) { + for _, set := range []string{"short", "full"} { + all := selected(t, set) + for n := 1; n <= 7; n++ { + seen := map[string]int{} + for k := 1; k <= n; k++ { + for _, sc := range shard(t, all, fmtShard(k, n)) { + seen[sc.name]++ + } + } + for _, sc := range all { + if seen[sc.name] != 1 { + t.Fatalf("%s in %d shards: %s runs %d times", set, n, sc.name, seen[sc.name]) + } + } + } + } +} + +func fmtShard(k, n int) string { return string(rune('0'+k)) + "/" + string(rune('0'+n)) } + +func TestFullSetRunsTheLongRestart(t *testing.T) { + names := map[string]bool{} + for _, sc := range selected(t, "full") { + names[sc.name] = true + } + if !names["s7-25"] || names["s7s-25"] || len(names) != len(scenarios)-1 { + t.Fatalf("full set %v", names) + } +} + +// A known line must name a criterion its judge reports. +func TestKnownLinesNameCriteria(t *testing.T) { + for _, sc := range scenarios { + _, criteria := sc.judge(&runData{marks: map[string]float64{}, meta: map[string]any{"fuse": 25.0}}) + have := map[string]bool{} + for _, c := range criteria { + have[c.Name] = true + } + for name := range sc.known { + if !have[name] { + t.Errorf("%s: known line names no criterion: %q", sc.name, name) + } + } + } + _, criteria := judgeInvariants(&runData{marks: map[string]float64{}, meta: map[string]any{"fuse": 25.0}}) + have := map[string]bool{} + for _, c := range criteria { + have[c.Name] = true + if invariantKey[c.Name] == "" { + t.Errorf("invariant %q has no first_violation_s key", c.Name) + } + } + for name := range randomTolerated { + if !have[name] { + t.Errorf("tolerated line names no invariant: %q", name) + } + } +} + +func TestRunsOver(t *testing.T) { + d := &runData{} + for i, a := range []float64{20, 25, 26, 26, 20, 25, 20} { + d.samples = append(d.samples, &sample{T: float64(i), PhaseA: [3]float64{a, 0, 0}}) + } + // As the Python rig counted: a run lasts until the first sample under + // the limit, and each sample over it adds the time since the last one. + longest, total := d.runsOver(0, 10, 24) + if longest != 3 || total != 4 { + t.Fatalf("longest %v total %v, want 3 and 4", longest, total) + } +} + +func TestRandomPlansAreSeeded(t *testing.T) { + a, b, c := drawPlan(3), drawPlan(3), drawPlan(4) + ja, _ := json.Marshal(a) + jb, _ := json.Marshal(b) + jc, _ := json.Marshal(c) + if string(ja) != string(jb) || string(ja) == string(jc) { + t.Fatal("seed 3 twice differs, or seed 4 is the same") + } + for seed := int64(1); seed <= 30; seed++ { + p := drawPlan(seed) + for _, a := range p.Actions { + if a.AtS < 40 || a.AtS > p.DurationS-60 { + t.Fatalf("seed %d: %s at %.0f s", seed, a.Do, a.AtS) + } + } + for _, x := range p.HouseA { + if x > p.FuseA/2 { + t.Fatalf("seed %d: house %.1f A on %.0f A", seed, x, p.FuseA) + } + } + } +} + +// An over-limit stretch right after an outside step is the house's; the +// same stretch with no step before it is FTW's own and fails after 3 s. +func TestInvariantsTellStepsFromOwnRaises(t *testing.T) { + mk := func(withStep bool) *runData { + d := &runData{marks: map[string]float64{"start": 0, "end": 100}, meta: map[string]any{"fuse": 25.0}} + for i := 0; i <= 100; i++ { + a := 10.0 + if i >= 50 && i < 56 { + a = 26 // 6 s over the 24 A limit + } + d.samples = append(d.samples, &sample{T: float64(i), PhaseA: [3]float64{a, a, a}, Bat: plantDevice{SoCPct: 50}}) + } + if withStep { + d.events = append(d.events, houseEventLine{T: 49.5, Event: houseEvent{Do: "loads"}}) + } + return d + } + failed := func(cs []criterion) []string { + var out []string + for _, c := range cs { + if !c.OK { + out = append(out, c.Name) + } + } + return out + } + if _, cs := judgeInvariants(mk(true)); len(failed(cs)) != 0 { + t.Fatalf("after a step: %v", failed(cs)) + } + _, cs := judgeInvariants(mk(false)) + if f := failed(cs); len(f) != 1 || f[0] != "over the limit at most 3 s after FTW's own raise" { + t.Fatalf("without a step: %v", f) + } +} diff --git a/go/test/site/runner/README.md b/go/test/site/runner/README.md new file mode 100644 index 000000000..346d7747c --- /dev/null +++ b/go/test/site/runner/README.md @@ -0,0 +1,78 @@ +# siterig-runner + +Runs FTW's site test (`go/test/site`) on a machine of its own and publishes +the result as a GitHub commit status on the exact commit it tested: + +| Status | When | What runs | +|---|---|---| +| `siterig/full` | each new commit on `master` | every scenario | +| `siterig/short` | the head of each open pull request from this repository that touches `go/internal/{control,loadpoint,ocpp,telemetry}` or `go/test/site` | the short set | +| `siterig/random` | once a night on `master` | that night's random seeds | + +A commit that already has the status is not run again. To run one again, +post a new `pending` status or remove the result and wait a day. No GitHub +Actions workflow runs the rig; `native-release.yml` only reads +`siterig/full`. + +The description says how it went, for example `16 pass, 3 known, 0 fail of +19 in 24m`. The run directories stay in `~/siterig/results` for 14 days. + +## The machine + +Any Mac or Linux host with Go, git, make and the `gh` CLI that can stay +awake. A laptop works if it does not sleep: a sleeping host stretches the +rig's timing, and the rig then fails the run as "host paused". Not the home +energy box: the rig needs the CPU, and the box is production. + +The rig uses 100 ports per lane from `-port-base` (default 18700) on +127.0.0.1. Leave them free. + +Pull requests from forks are skipped unless `-allow-forks` is set: the rig +builds and runs the pull request's code on this machine. + +## Install + +```bash +gh auth login # an account that may write commit statuses on srcfl/ftw +mkdir -p ~/siterig/bin +git clone https://github.com/srcfl/ftw ~/siterig/src +cd ~/siterig/src/go && go build -o ~/siterig/bin/siterig-runner ./test/site/runner +~/siterig/bin/siterig-runner -once # one pass in the foreground +``` + +macOS (launchd): + +```bash +sed "s/USER/$USER/g" ~/siterig/src/go/test/site/runner/energy.ftw.siterig-runner.plist \ + > ~/Library/LaunchAgents/energy.ftw.siterig-runner.plist +launchctl load ~/Library/LaunchAgents/energy.ftw.siterig-runner.plist +``` + +Linux (systemd): edit `siterig-runner.service`, copy it to +`/etc/systemd/system/` and run `sudo systemctl enable --now siterig-runner`. + +The log is `~/siterig/runner.log` (launchd) or `journalctl -u +siterig-runner` (systemd). Rebuild the binary after a change to this +directory; the scenarios themselves always come from the commit under test. + +## How long it takes + +On an M5 MacBook (10 cores), 22 October 2026: the 23 scenarios of that +morning took 72 minutes in three lanes. With the one-phase charger cases the +set adds up to about 270 minutes, so three lanes need about 90 and four +about 70; `s12-20` alone takes up to 56, so more than four do not help. +Each lane is one `ftw` and one `simulator run`, both mostly idle. + +## Flags + +| Flag | Default | | +|---|---|---| +| `-repo` | `srcfl/ftw` | | +| `-work` | `~/siterig` | clone, worktrees, results, `last-random` | +| `-every` | `5m` | how often to look for new commits | +| `-lanes` | `3` | scenarios at once | +| `-port-base` | `18700` | first port | +| `-random-at`, `-random-seeds` | `2`, `10` | local hour and count of the nightly random seeds; `-random-at -1` turns them off | +| `-drafts` | `true` | also run draft pull requests | +| `-allow-forks` | `false` | also run forks' pull requests | +| `-once` | | one pass, then exit | diff --git a/go/test/site/runner/energy.ftw.siterig-runner.plist b/go/test/site/runner/energy.ftw.siterig-runner.plist new file mode 100644 index 000000000..10f4604c6 --- /dev/null +++ b/go/test/site/runner/energy.ftw.siterig-runner.plist @@ -0,0 +1,24 @@ + + + + + + Labelenergy.ftw.siterig-runner + ProgramArguments + + /Users/USER/siterig/bin/siterig-runner + -work/Users/USER/siterig + -lanes3 + + EnvironmentVariables + + PATH/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin + HOME/Users/USER + + RunAtLoad + KeepAlive + StandardOutPath/Users/USER/siterig/runner.log + StandardErrorPath/Users/USER/siterig/runner.log + + diff --git a/go/test/site/runner/main.go b/go/test/site/runner/main.go new file mode 100644 index 000000000..6878257c9 --- /dev/null +++ b/go/test/site/runner/main.go @@ -0,0 +1,232 @@ +// siterig-runner runs FTW's site test (go/test/site) on a machine of its own +// and publishes the result as a GitHub commit status on the exact commit: +// siterig/full on each new master commit, siterig/short on the head of each +// open pull request that touches control, loadpoint, OCPP or telemetry code, +// and siterig/random for a night's random seeds. native-release.yml reads +// siterig/full before it publishes a beta. See README.md. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "log" + "os" + "os/exec" + "os/signal" + "path/filepath" + "strings" + "syscall" + "time" +) + +func main() { + var cfg config + flag.StringVar(&cfg.Repo, "repo", "srcfl/ftw", "GitHub repository") + work := flag.String("work", filepath.Join(os.Getenv("HOME"), "siterig"), "clone, worktrees and results") + every := flag.Duration("every", 5*time.Minute, "how often to look for new commits") + lanes := flag.Int("lanes", 3, "scenarios run at once") + portBase := flag.Int("port-base", 18700, "first port; lanes use 100 each") + keep := flag.Duration("keep", 14*24*time.Hour, "how long to keep results") + flag.BoolVar(&cfg.AllowForks, "allow-forks", false, "also run pull requests from forks (their code runs here)") + flag.BoolVar(&cfg.Drafts, "drafts", true, "also run draft pull requests") + flag.IntVar(&cfg.RandomAt, "random-at", 2, "local hour of the nightly random seeds; -1 never") + flag.IntVar(&cfg.RandomSeeds, "random-seeds", 10, "random seeds a night") + once := flag.Bool("once", false, "run what is due and exit") + flag.Parse() + + r := &runner{cfg: cfg, gh: ghCLI{repo: cfg.Repo}, work: *work, lanes: *lanes, portBase: *portBase, keep: *keep} + if err := r.prepare(); err != nil { + log.Fatal(err) + } + stop := make(chan os.Signal, 1) + signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM) + for { + r.tick() + if *once { + return + } + select { + case <-stop: + return + case <-time.After(*every): + } + } +} + +type runner struct { + cfg config + gh github + work string + lanes int + portBase int + keep time.Duration +} + +func (r *runner) clone() string { return filepath.Join(r.work, "ftw") } +func (r *runner) results() string { return filepath.Join(r.work, "results") } + +// prepare clones the repository once. +func (r *runner) prepare() error { + if err := os.MkdirAll(r.results(), 0o755); err != nil { + return err + } + if _, err := os.Stat(filepath.Join(r.clone(), ".git")); err == nil { + return nil + } + return run(r.work, "git", "clone", "--quiet", "https://github.com/"+r.cfg.Repo, r.clone()) +} + +// tick runs everything that is due, one job at a time. +func (r *runner) tick() { + stateFile := filepath.Join(r.work, "last-random") + last, _ := os.ReadFile(stateFile) + jobs, err := plan(r.gh, r.cfg, time.Now(), strings.TrimSpace(string(last))) + if err != nil { + log.Printf("plan: %v", err) + return + } + for _, j := range jobs { + log.Printf("start %s", j) + state, desc := r.runJob(j) + log.Printf("done %s: %s, %s", j, state, desc) + if err := r.gh.SetStatus(j.SHA, j.Context, state, desc); err != nil { + log.Printf("status: %v", err) + } + if j.Context == ctxRandom { + os.WriteFile(stateFile, []byte(time.Now().Format("2006-01-02")), 0o644) + } + } + r.prune() +} + +// runJob checks out the commit in a worktree of its own and runs the rig. +func (r *runner) runJob(j job) (state, desc string) { + if err := r.gh.SetStatus(j.SHA, j.Context, "pending", "running on "+hostname()); err != nil { + log.Printf("status: %v", err) + } + stamp := time.Now().Format("20060102-150405") + out := filepath.Join(r.results(), fmt.Sprintf("%s-%s-%s", stamp, short(j.SHA), strings.TrimPrefix(j.Context, "siterig/"))) + wt := filepath.Join(r.work, "wt-"+short(j.SHA)+"-"+stamp) + if err := os.MkdirAll(out, 0o755); err != nil { + return "error", err.Error() + } + defer func() { + run(r.clone(), "git", "worktree", "remove", "--force", wt) + run(r.clone(), "git", "worktree", "prune") + }() + fetch := []string{"fetch", "--quiet", "origin", "master"} + if j.PR > 0 { + fetch = append(fetch, fmt.Sprintf("refs/pull/%d/head", j.PR)) + } + if err := run(r.clone(), "git", fetch...); err != nil { + return "error", "git fetch failed" + } + if err := run(r.clone(), "git", "worktree", "add", "--detach", wt, j.SHA); err != nil { + return "error", "commit not found" + } + log1, _ := os.Create(filepath.Join(out, "rig.log")) + defer log1.Close() + mk := exec.Command("make", "drivers") + mk.Dir, mk.Stdout, mk.Stderr = wt, log1, log1 + if err := mk.Run(); err != nil { + return "error", "make drivers failed" + } + t0 := time.Now() + test := exec.Command("go", "test", "-count=1", "-timeout", "300m", "-run", "TestSite$", "-v", "./test/site/") + test.Dir, test.Stdout, test.Stderr = filepath.Join(wt, "go"), log1, log1 + test.Env = append(os.Environ(), "FTW_SITE="+j.Set, fmt.Sprintf("FTW_SITE_LANES=%d", r.lanes), + fmt.Sprintf("FTW_SITE_PORT_BASE=%d", r.portBase), "FTW_SITE_OUT="+out) + test.Run() // the summary, not the exit code, says what passed + summary, _ := os.ReadFile(filepath.Join(out, "summary.json")) + state, desc = describe(summary, time.Since(t0)) + if state == "success" && test.ProcessState != nil && !test.ProcessState.Success() { + state, desc = "failure", "the rig failed outside the scenarios: "+desc + } + return state, desc +} + +// prune drops results older than keep. +func (r *runner) prune() { + entries, _ := os.ReadDir(r.results()) + for _, e := range entries { + if info, err := e.Info(); err == nil && time.Since(info.ModTime()) > r.keep { + os.RemoveAll(filepath.Join(r.results(), e.Name())) + } + } +} + +func run(dir, name string, args ...string) error { + cmd := exec.Command(name, args...) + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + return fmt.Errorf("%s %s: %v: %s", name, strings.Join(args, " "), err, strings.TrimSpace(string(out))) + } + return nil +} + +func hostname() string { + h, _ := os.Hostname() + return h +} + +// ghCLI asks GitHub through the gh CLI, as the machine's logged-in user. +type ghCLI struct{ repo string } + +func (g ghCLI) api(out any, args ...string) error { + b, err := exec.Command("gh", append([]string{"api"}, args...)...).Output() + if err != nil { + return fmt.Errorf("gh api %s: %w", strings.Join(args, " "), err) + } + if out == nil { + return nil + } + return json.Unmarshal(b, out) +} + +func (g ghCLI) MasterSHA() (string, error) { + var c struct { + SHA string `json:"sha"` + } + err := g.api(&c, "repos/"+g.repo+"/commits/master") + return c.SHA, err +} + +func (g ghCLI) Pulls() ([]pull, error) { + var ps []pull + err := g.api(&ps, "repos/"+g.repo+"/pulls?state=open&per_page=100") + return ps, err +} + +func (g ghCLI) Files(pr int) ([]string, error) { + var fs []struct { + Filename string `json:"filename"` + } + if err := g.api(&fs, fmt.Sprintf("repos/%s/pulls/%d/files?per_page=100", g.repo, pr)); err != nil { + return nil, err + } + out := make([]string, len(fs)) + for i, f := range fs { + out[i] = f.Filename + } + return out, nil +} + +func (g ghCLI) Contexts(sha string) (map[string]bool, error) { + var ss []struct { + Context string `json:"context"` + } + if err := g.api(&ss, "repos/"+g.repo+"/commits/"+sha+"/statuses?per_page=100"); err != nil { + return nil, err + } + have := map[string]bool{} + for _, s := range ss { + have[s.Context] = true + } + return have, nil +} + +func (g ghCLI) SetStatus(sha, context, state, desc string) error { + return g.api(nil, "-X", "POST", "repos/"+g.repo+"/statuses/"+sha, + "-f", "state="+state, "-f", "context="+context, "-f", "description="+desc) +} diff --git a/go/test/site/runner/runner.go b/go/test/site/runner/runner.go new file mode 100644 index 000000000..1a0def0c0 --- /dev/null +++ b/go/test/site/runner/runner.go @@ -0,0 +1,173 @@ +package main + +import ( + "encoding/json" + "fmt" + "strings" + "time" +) + +// Status contexts the runner publishes on a commit. +const ( + ctxShort = "siterig/short" + ctxFull = "siterig/full" + ctxRandom = "siterig/random" +) + +// watchedPaths are the parts of the tree whose changes get the short set on +// a pull request. +var watchedPaths = []string{ + "go/internal/control/", "go/internal/loadpoint/", "go/internal/ocpp/", "go/internal/telemetry/", + "go/test/site/", +} + +// job is one run of the rig on one commit. +type job struct { + Context string // which status it publishes + SHA string + Set string // FTW_SITE: short, full or random:: + PR int // 0 for master +} + +func (j job) String() string { + if j.PR > 0 { + return fmt.Sprintf("%s on #%d %s", j.Context, j.PR, short(j.SHA)) + } + return fmt.Sprintf("%s on master %s", j.Context, short(j.SHA)) +} + +func short(sha string) string { + if len(sha) > 8 { + return sha[:8] + } + return sha +} + +// pull is the part of a pull request the runner needs. +type pull struct { + Number int `json:"number"` + Draft bool `json:"draft"` + Head struct { + SHA string `json:"sha"` + Repo struct { + FullName string `json:"full_name"` + } `json:"repo"` + } `json:"head"` +} + +// github is what the runner asks GitHub; gh api serves it in production. +type github interface { + MasterSHA() (string, error) + Pulls() ([]pull, error) + Files(pr int) ([]string, error) + Contexts(sha string) (map[string]bool, error) // status contexts already on a commit + SetStatus(sha, context, state, description string) error +} + +// config is the runner's flags. +type config struct { + Repo string + AllowForks bool // a fork's code runs on this machine; off by default + Drafts bool + RandomAt int // local hour of the nightly random run; -1 never + RandomSeeds int +} + +// plan lists what to run now, master first. A commit that already carries +// the context, in any state, is not run again. Random seeds run once a day +// (lastRandom is the day they last ran) on whatever master is then. +func plan(gh github, cfg config, now time.Time, lastRandom string) ([]job, error) { + var jobs []job + master, err := gh.MasterSHA() + if err != nil { + return nil, err + } + have, err := gh.Contexts(master) + if err != nil { + return nil, err + } + if !have[ctxFull] { + jobs = append(jobs, job{Context: ctxFull, SHA: master, Set: "full"}) + } + pulls, err := gh.Pulls() + if err != nil { + return nil, err + } + for _, p := range pulls { + if p.Draft && !cfg.Drafts { + continue + } + if p.Head.Repo.FullName != cfg.Repo && !cfg.AllowForks { + continue + } + have, err := gh.Contexts(p.Head.SHA) + if err != nil { + return nil, err + } + if have[ctxShort] { + continue + } + files, err := gh.Files(p.Number) + if err != nil { + return nil, err + } + if touchesWatched(files) { + jobs = append(jobs, job{Context: ctxShort, SHA: p.Head.SHA, Set: "short", PR: p.Number}) + } + } + day := now.Format("2006-01-02") + if cfg.RandomAt >= 0 && cfg.RandomSeeds > 0 && now.Hour() >= cfg.RandomAt && lastRandom != day { + // One seed range per day, so each night tries new timelines. + first := int(now.Unix()/86400) * cfg.RandomSeeds + jobs = append(jobs, job{Context: ctxRandom, SHA: master, Set: fmt.Sprintf("random:%d:%d", first, cfg.RandomSeeds)}) + } + return jobs, nil +} + +func touchesWatched(files []string) bool { + for _, f := range files { + for _, p := range watchedPaths { + if strings.HasPrefix(f, p) { + return true + } + } + } + return false +} + +// outcome is one line of the rig's summary.json. +type outcome struct { + Name string `json:"name"` + Verdict string `json:"verdict"` // pass, known, fail + DurationS float64 `json:"duration_s"` + Error string `json:"error"` +} + +// describe turns summary.json into a status: success when nothing failed, +// with counts that fit GitHub's 140 characters. +func describe(summary []byte, took time.Duration) (state, desc string) { + var outs []outcome + if err := json.Unmarshal(summary, &outs); err != nil || len(outs) == 0 { + return "error", "no summary: the rig did not finish (see the runner's log)" + } + n := map[string]int{} + var failed []string + for _, o := range outs { + n[o.Verdict]++ + if o.Verdict == "fail" { + failed = append(failed, o.Name) + } + } + state = "success" + if len(failed) > 0 { + state = "failure" + } + desc = fmt.Sprintf("%d pass, %d known, %d fail of %d in %s", n["pass"], n["known"], n["fail"], len(outs), took.Round(time.Minute)) + if len(failed) > 0 { + desc += ": " + strings.Join(failed, " ") + } + if len(desc) > 140 { + desc = desc[:137] + "..." + } + return state, desc +} diff --git a/go/test/site/runner/runner_test.go b/go/test/site/runner/runner_test.go new file mode 100644 index 000000000..d05a1dd36 --- /dev/null +++ b/go/test/site/runner/runner_test.go @@ -0,0 +1,94 @@ +package main + +import ( + "strings" + "testing" + "time" +) + +type fakeGitHub struct { + master string + pulls []pull + files map[int][]string + contexts map[string]map[string]bool +} + +func (f fakeGitHub) MasterSHA() (string, error) { return f.master, nil } +func (f fakeGitHub) Pulls() ([]pull, error) { return f.pulls, nil } +func (f fakeGitHub) Files(pr int) ([]string, error) { return f.files[pr], nil } +func (f fakeGitHub) Contexts(sha string) (map[string]bool, error) { + return f.contexts[sha], nil +} +func (f fakeGitHub) SetStatus(sha, context, state, description string) error { return nil } + +func pr(n int, sha, repo string, draft bool) pull { + p := pull{Number: n, Draft: draft} + p.Head.SHA, p.Head.Repo.FullName = sha, repo + return p +} + +func TestPlanRunsWhatIsDue(t *testing.T) { + gh := fakeGitHub{ + master: "m1", + pulls: []pull{ + pr(1, "a1", "srcfl/ftw", false), // control code: short set + pr(2, "b1", "srcfl/ftw", false), // docs only: nothing + pr(3, "c1", "someone/ftw", false), // a fork: never by default + pr(4, "d1", "srcfl/ftw", false), // already has its status + }, + files: map[int][]string{ + 1: {"go/internal/control/dispatch.go"}, 2: {"docs/x.md"}, + 3: {"go/internal/ocpp/handlers.go"}, 4: {"go/internal/loadpoint/x.go"}, + }, + contexts: map[string]map[string]bool{"d1": {ctxShort: true}}, + } + cfg := config{Repo: "srcfl/ftw", Drafts: true, RandomAt: 2, RandomSeeds: 10} + night := time.Date(2026, 10, 23, 3, 0, 0, 0, time.Local) + jobs, err := plan(gh, cfg, night, "") + if err != nil { + t.Fatal(err) + } + var got []string + for _, j := range jobs { + got = append(got, j.String()+" "+j.Set) + } + want := []string{"siterig/full on master m1 full", "siterig/short on #1 a1 short", "siterig/random on master m1 random:"} + if len(got) != len(want) { + t.Fatalf("jobs %v, want %v", got, want) + } + for i := range want { + if !strings.HasPrefix(got[i], want[i]) { + t.Fatalf("job %d: %q, want %q", i, got[i], want[i]) + } + } + + // Master already tested, random already run today, before the hour. + gh.contexts["m1"] = map[string]bool{ctxFull: true} + jobs, _ = plan(gh, cfg, night, "2026-10-23") + if len(jobs) != 1 || jobs[0].PR != 1 { + t.Fatalf("second look: %v", jobs) + } + cfg.AllowForks = true + jobs, _ = plan(gh, cfg, night.Add(-2*time.Hour), "") + if len(jobs) != 2 || jobs[1].PR != 3 { + t.Fatalf("with forks allowed, before the random hour: %v", jobs) + } +} + +func TestDescribe(t *testing.T) { + state, desc := describe([]byte(`[{"name":"s1-25","verdict":"pass"},{"name":"s2-16","verdict":"known"},{"name":"s9-25","verdict":"fail"}]`), 24*time.Minute) + if state != "failure" || desc != "1 pass, 1 known, 1 fail of 3 in 24m0s: s9-25" { + t.Fatalf("%s %q", state, desc) + } + state, _ = describe([]byte(`[{"name":"s1-25","verdict":"pass"},{"name":"s2-16","verdict":"known"}]`), time.Minute) + if state != "success" { + t.Fatalf("known only: %s", state) + } + if state, _ := describe(nil, time.Minute); state != "error" { + t.Fatalf("no summary: %s", state) + } + long := `[` + strings.Repeat(`{"name":"a-very-long-scenario-name","verdict":"fail"},`, 9) + `{"name":"x","verdict":"fail"}]` + if _, desc := describe([]byte(long), time.Minute); len(desc) > 140 { + t.Fatalf("%d characters", len(desc)) + } +} diff --git a/go/test/site/runner/siterig-runner.service b/go/test/site/runner/siterig-runner.service new file mode 100644 index 000000000..f287a597b --- /dev/null +++ b/go/test/site/runner/siterig-runner.service @@ -0,0 +1,18 @@ +# systemd unit for siterig-runner (README.md). Replace USER and the paths, +# then: sudo cp siterig-runner.service /etc/systemd/system/ && +# sudo systemctl enable --now siterig-runner +[Unit] +Description=FTW site test runner (siterig/short, siterig/full commit statuses) +After=network-online.target +Wants=network-online.target + +[Service] +User=USER +Environment=HOME=/home/USER +Environment=PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin +ExecStart=/home/USER/siterig/bin/siterig-runner -work /home/USER/siterig -lanes 3 +Restart=always +RestartSec=60 + +[Install] +WantedBy=multi-user.target diff --git a/go/test/site/scenarios_test.go b/go/test/site/scenarios_test.go new file mode 100644 index 000000000..4c42c9c68 --- /dev/null +++ b/go/test/site/scenarios_test.go @@ -0,0 +1,1026 @@ +//go:build unix + +package site + +import ( + "encoding/json" + "fmt" + "math" + "os" + "path/filepath" + "time" +) + +// scenario is one house and what happens in it. known names the criteria +// that fail on master and the finding behind each: such a criterion must +// keep failing, and once a fix makes it pass the run fails until the fix +// deletes its line. Criteria come from reports/simulatorprov-20261021. +type scenario struct { + name string // what FTW_SITE names it by + dir string // its run directory + minutes float64 // its run time on a GitHub runner, for sharding + short bool // in the pull-request set + run func(c *runCtx) (map[string]any, error) + judge func(d *runData) (metrics, []criterion) + known map[string]string + // tolerate names criteria that may fail for a known finding but need + // not: random cases, where a seed may or may not reach it. + tolerate map[string]string +} + +var scenarios = []scenario{ + {name: "s0-25", dir: "s0-idle-charger-25A", minutes: 5.1, short: true, + run: func(c *runCtx) (map[string]any, error) { return s0(c, 25, 60) }, judge: judgeS0}, + {name: "s0b-25", dir: "s0b-idle-charger-default-hb-25A", minutes: 5.1, + run: func(c *runCtx) (map[string]any, error) { return s0(c, 25, 0) }, judge: judgeS0}, + {name: "s1-16", dir: "s1-chargenow-16A", minutes: 3.6, short: true, + run: func(c *runCtx) (map[string]any, error) { return s1(c, 16) }, judge: judgeS1}, + {name: "s1-25", dir: "s1-chargenow-25A", minutes: 3.6, short: true, + run: func(c *runCtx) (map[string]any, error) { return s1(c, 25) }, judge: judgeS1}, + {name: "s2-16", dir: "s2-export-16A", minutes: 3.1, + run: func(c *runCtx) (map[string]any, error) { return s2(c, 16) }, judge: judgeS2, + known: map[string]string{"car starts within 60 s": "export counted as import (review fuse finding 1), next beta"}}, + {name: "s3-16", dir: "s3-step-16A", minutes: 6.7, short: true, + run: func(c *runCtx) (map[string]any, error) { return s3(c, 16) }, judge: judgeS3}, + {name: "s3-25", dir: "s3-step-25A", minutes: 6.7, short: true, + run: func(c *runCtx) (map[string]any, error) { return s3(c, 25) }, judge: judgeS3}, + {name: "s4-25", dir: "s4-unlock-25A", minutes: 3.6, + run: func(c *runCtx) (map[string]any, error) { return s4(c, 25, "3p") }, judge: judgeS4}, + {name: "s4b-25", dir: "s4b-unlock-nophase-25A", minutes: 3.6, + run: func(c *runCtx) (map[string]any, error) { return s4(c, 25, "") }, judge: judgeS4}, + {name: "s5-25", dir: "s5-surplus-auto-25A", minutes: 5.2, + run: func(c *runCtx) (map[string]any, error) { return s5(c, 25, nil) }, judge: judgeS5, + known: map[string]string{"charges at least the minimum 90 % of the time": "surplus without allowed_steps_w never starts, loadpoint/controller.go surplus3PhaseSteps; next beta"}}, + {name: "s5s-25", dir: "s5s-surplus-auto-steps-25A", minutes: 5.2, + run: func(c *runCtx) (map[string]any, error) { return s5(c, 25, easeeSteps) }, judge: judgeS5, + known: map[string]string{"charges at least the minimum 90 % of the time": "phase mode auto stays on 1p under 9 kW surplus (R1); next beta"}}, + {name: "s6-25", dir: "s6-inverter-offline-25A", minutes: 7.7, + run: func(c *runCtx) (map[string]any, error) { return s6(c, 25) }, judge: judgeS6}, + {name: "s7-25", dir: "s7-restart-25A", minutes: 23, + run: func(c *runCtx) (map[string]any, error) { return s7(c, 25, false) }, judge: judgeS7}, + // The full s7 runs 15 min before the restart, because the day's Home + // use may have at most 10 % unknown time. The short one restarts after + // a minute and leaves the day total out. + {name: "s7s-25", dir: "s7s-restart-short-25A", minutes: 9, short: true, + run: func(c *runCtx) (map[string]any, error) { return s7(c, 25, true) }, judge: judgeS7}, + {name: "s8-60-25", dir: "s8-slow60-25A", minutes: 7.5, short: true, + run: func(c *runCtx) (map[string]any, error) { return s8(c, 25, 60) }, judge: judgeS8}, + {name: "s8-120-25", dir: "s8-slow120-25A", minutes: 12.5, + run: func(c *runCtx) (map[string]any, error) { return s8(c, 25, 120) }, judge: judgeS8}, + {name: "s9-25", dir: "s9-phase-switch-25A", minutes: 4.7, + run: func(c *runCtx) (map[string]any, error) { return s9(c, 25, 7360) }, judge: judgeS9}, + {name: "s10-25", dir: "s10-two-chargers-25A", minutes: 7.9, + run: func(c *runCtx) (map[string]any, error) { return s10(c, 25, 60) }, judge: judgeS10}, + {name: "s11-25", dir: "s11-pause-restart-25A", minutes: 5.1, + run: func(c *runCtx) (map[string]any, error) { return s11(c, 25) }, judge: judgeS11}, + {name: "s9r-25", dir: "s9r-house-on-l2-25A", minutes: 5, + run: func(c *runCtx) (map[string]any, error) { return s9r(c, 25, 7360) }, judge: judgeS9}, + {name: "s5t-25", dir: "s5t-surplus-goal-25A", minutes: 6, + run: func(c *runCtx) (map[string]any, error) { return s5t(c, 25) }, judge: judgeS5t}, + {name: "s12-20", dir: "s12-one-phase-night-20A", minutes: 57, + run: func(c *runCtx) (map[string]any, error) { return s12(c, 20) }, judge: judgeS12}, + {name: "s13-25", dir: "s13-socket-reconnect-25A", minutes: 6, + run: func(c *runCtx) (map[string]any, error) { return s13(c, 25) }, judge: judgeS13, + known: map[string]string{"Charge now goes on after the charger returns": "a socket reconnect turns Charge now into a 0 W pause; srcfl/ftw#1788"}}, + {name: "s15-25", dir: "s15-one-phase-charger-chargenow-25A", minutes: 4.6, short: true, + run: func(c *runCtx) (map[string]any, error) { return s15(c, 25) }, judge: judgeS15}, + {name: "s15b-25", dir: "s15b-one-phase-charger-1p-25A", minutes: 4.6, + run: func(c *runCtx) (map[string]any, error) { return s15mode(c, 25, "1p") }, judge: judgeS15}, + {name: "s15c-25", dir: "s15c-one-phase-charger-auto-25A", minutes: 4.6, + run: func(c *runCtx) (map[string]any, error) { return s15mode(c, 25, "auto") }, judge: judgeS15}, + {name: "s15bat-16", dir: "s15bat-one-phase-charger-battery-rising-16A", minutes: 5.5, + run: func(c *runCtx) (map[string]any, error) { return s15bat(c, 16) }, judge: judgeS15bat}, + {name: "s15p-25", dir: "s15p-one-phase-charger-plan-25A", minutes: 32, + run: func(c *runCtx) (map[string]any, error) { return s15p(c, 25) }, judge: judgeS15p}, + {name: "s14-25", dir: "s14-one-phase-car-25A", minutes: 22, + run: func(c *runCtx) (map[string]any, error) { return s14(c, 25) }, judge: judgeS14}, +} + +// runCtx is where one scenario runs. +type runCtx struct { + env *rigEnv + lane lane + rundir string +} + +func (c *runCtx) stack(spec stackSpec) *stack { return newStack(c.env, c.lane, c.rundir, spec) } + +func houseA(a float64) [3]float64 { return [3]float64{a * volt, a * volt, a * volt} } + +// ---- what a client does ---- + +type loadpointView struct { + ID string `json:"id"` + Plugged bool `json:"plugged_in"` + Charger struct { + Known bool `json:"known"` + } `json:"charger"` +} + +func (st *stack) loadpoint(id string) (loadpointView, error) { + var j struct { + Loadpoints []loadpointView `json:"loadpoints"` + } + if err := st.get("/api/loadpoints", &j); err != nil { + return loadpointView{}, err + } + for _, x := range j.Loadpoints { + if x.ID == id { + return x, nil + } + } + if len(j.Loadpoints) > 0 { + return j.Loadpoints[0], nil + } + return loadpointView{}, fmt.Errorf("no loadpoints") +} + +// chargeNow is what the web, PWA and iOS clients send. An empty phaseMode +// is a bare API call. powerW 0 means amps on three phases, below 0 a pause. +func (st *stack) chargeNow(lpID string, amps float64, phaseMode string, powerW float64) error { + switch { + case powerW < 0: // a pause + powerW = 0 + case powerW == 0: + powerW = amps * volt * 3 + } + body := map[string]any{"power_w": powerW, "hold_s": 0} + if phaseMode != "" { + body["phase_mode"] = phaseMode + } + return st.post("/api/loadpoints/"+lpID+"/manual_hold", body, nil) +} + +// chargerOnline waits until charger i talks to FTW. +func (st *stack) chargerOnline(i int) error { + serial := st.serial(i) + ok := waitUntil(90*time.Second, time.Second, func() bool { + var s struct { + Drivers map[string]json.RawMessage `json:"drivers"` + } + if st.get("/api/status", &s) != nil { + return false + } + _, ok := s.Drivers[serial] + return ok + }) + if !ok { + return fmt.Errorf("charger %s never connected to FTW", serial) + } + return nil +} + +// plug plugs a car into charger i once the charger is online, and waits +// for FTW to see it. +func (st *stack) plug(i int) error { + if err := st.chargerOnline(i); err != nil { + return err + } + time.Sleep(2 * time.Second) + label := "plug" + if i > 0 { + label = fmt.Sprintf("plug%d", i) + } + st.mark(label) + on := true + if err := st.event(houseEvent{Do: "plug", Device: lpIDs[i], On: &on}); err != nil { + return err + } + ok := waitUntil(90*time.Second, time.Second, func() bool { + lp, err := st.loadpoint(lpIDs[i]) + return err == nil && lp.Plugged && lp.Charger.Known + }) + if !ok { + return fmt.Errorf("loadpoint %s never saw the car plugged in", lpIDs[i]) + } + return nil +} + +func sleepS(s float64) { time.Sleep(time.Duration(s * float64(time.Second))) } + +// with brings a stack up, runs body and always takes the stack down. +func with(c *runCtx, spec stackSpec, body func(st *stack) error) error { + st := c.stack(spec) + defer st.down() + if err := st.up(); err != nil { + return err + } + if err := body(st); err != nil { + return err + } + return st.markErr +} + +// ---- 0. Adopted OCPP charger, no car: is Home use measured? ---- + +func s0(c *runCtx, fuse float64, heartbeatS int) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50, heartbeatS: heartbeatS}, func(st *stack) error { + if err := st.chargerOnline(0); err != nil { + return err + } + st.mark("connected") + // Several heartbeats: #1779 keeps the idle 0 W fresh with them. + sleepS(300) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s0", "fuse": fuse, "house_a": 3, "heartbeat_s": heartbeatS}, err +} + +// ---- 1. Charge now, 3 A house load per phase ---- + +func s1(c *runCtx, fuse float64) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(180) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s1", "fuse": fuse, "house_a": 3}, err +} + +// ---- 2. Charge now during PV export of 9 A per phase, 16 A fuse ---- + +func s2(c *runCtx, fuse float64) (map[string]any, error) { + // Full battery so it cannot soak up the export. PV 7 kW, house 1 A: + // 6.3 kW export, 9.1 A per phase out of the site. + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(1), pvW: 7000, socPct: 100}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(150) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s2", "fuse": fuse, "house_a": 1, "pv_w": 7000, "soc": 100}, err +} + +// ---- 3. House load steps to near the fuse while car and battery charge ---- + +func s3(c *runCtx, fuse float64) (map[string]any, error) { + stepA := map[float64]float64{16: 13, 25: 21}[fuse] + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 40}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("battery_hold") + if err := st.post("/api/battery/manual_hold", map[string]any{"direction": "charge", "power_w": 3000, "hold_s": 1800}, nil); err != nil { + return err + } + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(120) + st.mark("step") + if err := st.setLoadsA(stepA); err != nil { + return err + } + sleepS(180) + st.mark("step_back") + if err := st.setLoadsA(3); err != nil { + return err + } + sleepS(60) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s3", "fuse": fuse, "house_a": 3, "step_a": stepA, "battery_hold_w": 3000, "charger_delay_s": 0}, err +} + +// ---- 4. surplus_unlock_bat_soc saved but not armed, Charge now ---- + +func s4(c *runCtx, fuse float64, phaseMode string) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(20) + st.mark("schedule") + // Only the unlock, as in #1774's R2 test: a SoC goal would make the + // plan charge at once and hide the fault. + var resp json.RawMessage + if err := st.post("/api/loadpoints/garage/target", map[string]any{"schedule": map[string]any{ + "soc": 0, "time_of_day_min": 7 * 60, "time_of_day_min_utc": 5 * 60, + "recurring": false, "days": 0, "surplus_unlock_bat_soc": 0.9}}, &resp); err != nil { + return err + } + os.WriteFile(filepath.Join(st.rundir, "target-response.json"), resp, 0o644) + sleepS(10) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, phaseMode, 0); err != nil { + return err + } + sleepS(180) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s4", "fuse": fuse, "house_a": 3, "unlock": 0.9, "soc": 50, "phase_mode": phaseMode}, err +} + +// ---- 5. Surplus only, phase mode auto, ~9 kW surplus ---- + +// easeeSteps are Easee's whole-amp offers at 230 V, as in #1774's tests: +// 6-16 A on one phase, then 6-16 A on three. +var easeeSteps = []float64{0, 1380, 1610, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680, + 4140, 4830, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11040} + +func s5(c *runCtx, fuse float64, steps []float64) (map[string]any, error) { + extra := map[string]any{"surplus_only": true, "phase_mode": "auto"} + if steps != nil { + extra["allowed_steps_w"] = steps + } + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(1), pvW: 10000, socPct: 100, lpExtra: extra}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + st.mark("plugged") + sleepS(300) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s5", "fuse": fuse, "house_a": 1, "pv_w": 10000, "soc": 100, "phase_mode": "auto", "steps": steps != nil}, err +} + +// ---- 6. PV inverter offline for 5 minutes ---- + +func s6(c *runCtx, fuse float64) (map[string]any, error) { + // A car plugged in, so that case 0 does not interfere. + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), pvW: 1500, socPct: 50}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(60) + st.mark("offline") + if err := st.event(houseEvent{Do: "offline", Device: "inverter"}); err != nil { + return err + } + sleepS(300) + st.mark("online") + if err := st.event(houseEvent{Do: "online", Device: "inverter"}); err != nil { + return err + } + sleepS(90) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s6", "fuse": fuse, "house_a": 3, "pv_w": 1500, "offline_s": 300}, err +} + +// ---- 7. Core restart of ~20 s while charging ---- + +func s7(c *runCtx, fuse float64, short bool) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + if short { + sleepS(60) + } else { + // A new box has minutes of history; unknown time may be at most + // 10 % of it (history_dashboard.go). Wait until the day total is + // known, then long enough that the restart's ~110 s of unknown + // time stays below the rule, as on a box that has run since morning. + if waitUntil(900*time.Second, 5*time.Second, func() bool { + var s ftwStatus + return st.get("/api/status", &s) == nil && s.Energy.Today.LoadWh != nil + }) { + st.mark("today_known") + } + sleepS(900) + } + st.mark("ftw_stop") + st.stopFTW() + sleepS(20) + st.mark("ftw_start") + if err := st.startFTW(); err != nil { + return err + } + st.mark("ftw_up") + // Long enough to see whether the car gets back to full power. + sleepS(420) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s7", "fuse": fuse, "house_a": 3, "down_s": 20, "short": short}, err +} + +// ---- 8. Slow charger: profiles apply 60 s or 120 s late ---- + +func s8(c *runCtx, fuse, delayS float64) (map[string]any, error) { + // Battery at the rig's 5 % floor: it cannot cover the house. At 8 % FTW + // forced it to discharge for the fuse and the meter showed room that + // vanished when it ran empty, which hid the stacking (see the report). + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(15), socPct: 5, chargers: []charger{{delayS: delayS}}}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + // The charger draws its maximum until FTW's first profile lands, + // delayS after plug-in. Start from there, at 0 W. + st.mark("plug_wait") + sleepS(delayS + 20) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + // Long enough for the last rise and the cut after it to land. + sleepS(max(300, 4*delayS+120)) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s8", "fuse": fuse, "house_a": 15, "charger_delay_s": delayS, "soc_pct": 5}, err +} + +// ---- 9. Charge now switched from 3p to 1p at the same watts ---- + +func s9(c *runCtx, fuse, watts float64) (map[string]any, error) { + // A 32 A charger that can switch phases. 7 360 W is 10.7 A on three + // phases and 32 A on one. House 10 A per phase, empty battery: one phase + // has 14 A of room below FTW's 24 A limit. + spec := stackSpec{fuseA: fuse, loadsW: houseA(10), socPct: 5, chargers: []charger{{kw: 22.08, phaseSwitch: true}}, + lpExtra: map[string]any{"min_charge_w": 1380, "max_charge_w": 22080, "phase_mode": "auto"}} + err := with(c, spec, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 0, "3p", watts); err != nil { + return err + } + sleepS(90) + st.mark("switch") + if err := st.chargeNow("garage", 0, "1p", watts); err != nil { + return err + } + sleepS(150) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s9", "fuse": fuse, "house_a": 10, "watts": watts, "charger_a": 32}, err +} + +// ---- 10. Two chargers: one raised while the other has just been cut ---- + +func s10(c *runCtx, fuse, delayS float64) (map[string]any, error) { + // garage answers at once; carport is a cloud charger that applies a + // profile delayS late. House 3 A per phase, empty battery. Both charge + // at 10 A (23 A per phase). Then carport is cut to 6 A and garage raised + // to 16 A in the same second: the room carport frees is not there until + // its cut lands. + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 5, chargers: []charger{{}, {delayS: delayS}}}, func(st *stack) error { + if err := st.plug(1); err != nil { + return err + } + // The slow charger draws its maximum until FTW's first profile lands. + st.mark("plug_wait") + sleepS(delayS + 20) + if err := st.plug(0); err != nil { + return err + } + sleepS(20) + st.mark("charge_now") + if err := st.chargeNow("garage", 10, "3p", 0); err != nil { + return err + } + if err := st.chargeNow("carport", 10, "3p", 0); err != nil { + return err + } + sleepS(delayS + 60) + st.mark("step") + if err := st.chargeNow("carport", 6, "3p", 0); err != nil { + return err + } + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(2*delayS + 120) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s10", "fuse": fuse, "house_a": 3, "charger_delay_s": []float64{0, delayS}}, err +} + +// ---- 11. Pause and restart under a long declared response time (#1793) ---- + +func s11(c *runCtx, fuse float64) (map[string]any, error) { + // OCPP declares no response time, so Core reserves 180 s for each order, + // as for a charger that declares 180 s. Charge now 16 A, pause (0 W) for + // 25 s, then start at 6 A: the cut must end the old order's reservation. + // The charger applies a profile 5 s late (ev-104's plant). + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50, chargers: []charger{{delayS: 5}}}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(60) + st.mark("pause") + if err := st.chargeNow("garage", 0, "3p", -1); err != nil { + return err + } + sleepS(25) + st.mark("restart") + if err := st.chargeNow("garage", 6, "3p", 0); err != nil { + return err + } + sleepS(180) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s11", "fuse": fuse, "house_a": 3, "pause_s": 25, "charger_delay_s": 5}, err +} + +// ---- planned charging ---- + +// planTo tells FTW the car is at socPct and must reach goalPct by the first +// quarter hour at least readyIn from now. It returns the ready time. +func (st *stack) planTo(lp string, socPct, goalPct float64, readyIn time.Duration) (time.Time, error) { + if err := st.post("/api/loadpoints/"+lp+"/soc", map[string]any{"soc_pct": socPct}, nil); err != nil { + return time.Time{}, err + } + ready := time.Now().Add(readyIn).Truncate(15 * time.Minute).Add(15 * time.Minute) + if err := st.post("/api/loadpoints/"+lp+"/target", map[string]any{"schedule": map[string]any{ + "soc": goalPct / 100, "time_of_day_min": ready.Hour()*60 + ready.Minute(), "recurring": false, "days": 0}}, nil); err != nil { + return time.Time{}, err + } + return ready, nil +} + +// ---- 9r. s9 reversed: a 1p order raised to 3p beside a loaded L2 ---- + +func s9r(c *runCtx, fuse, watts float64) (map[string]any, error) { + // House 2/20/2 A on 25 A: L2 has 4 A left under FTW's 24 A limit. A + // 32 A charger that switches phases gets Charge now 7 360 W on 1p + // (L1 has room), then the same watts on 3p, which would put 10.7 A on + // L2 too. + spec := stackSpec{fuseA: fuse, loadsW: [3]float64{2 * volt, 20 * volt, 2 * volt}, socPct: 5, + chargers: []charger{{kw: 22.08, phaseSwitch: true}}, + lpExtra: map[string]any{"min_charge_w": 1380, "max_charge_w": 22080, "phase_mode": "auto"}} + err := with(c, spec, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 0, "1p", watts); err != nil { + return err + } + sleepS(90) + st.mark("switch") + if err := st.chargeNow("garage", 0, "3p", watts); err != nil { + return err + } + sleepS(150) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s9", "fuse": fuse, "house_a": []float64{2, 20, 2}, "watts": watts, "charger_a": 32, "order": "1p then 3p"}, err +} + +// ---- 5t. s5s with a goal: surplus below the three-phase minimum ---- + +func s5t(c *runCtx, fuse float64) (map[string]any, error) { + // PV 4.5 kW, house 1 A per phase, full battery: about 3.8 kW of surplus, + // under the 4 140 W three-phase minimum. With a goal saved, a + // surplus-only loadpoint must still charge, on one phase: the charger + // switches phases, as an Easee does. + extra := map[string]any{"surplus_only": true, "phase_mode": "auto", "allowed_steps_w": easeeSteps} + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(1), pvW: 4500, socPct: 100, lpExtra: extra, + chargers: []charger{{phaseSwitch: true}}}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + if _, err := st.planTo("garage", 40, 80, 8*time.Hour); err != nil { + return err + } + st.mark("plugged") + sleepS(300) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s5t", "fuse": fuse, "house_a": 1, "pv_w": 4500, "soc": 100}, err +} + +func judgeS5t(d *runData) (metrics, []criterion) { + m, cs := judgeS5(d) + t0, t1 := d.marks["plugged"], d.marks["end"] + w := d.window(t0+60, t1) + n := 0 + for _, r := range w { + if carW(r, 0) >= 0.95*1380 { + n++ + } + } + frac := round(float64(n)/math.Max(1, float64(len(w))), 3) + ones := 0 + prs := connector1(profiles(d.frames, t0, t1)) + for _, p := range prs { + if p.Limit > 0 && p.phasesOr3() == 1 { + ones++ + } + } + m["charging_fraction_ge_1p_min"], m["one_phase_offers"] = frac, ones + out := []criterion{} + for _, c := range cs { + if c.Name == "no offer above the charger's rating" || c.Name == "fuse holds" { + out = append(out, c) + } + } + return m, append(out, crit("charges at least 6 A on one phase 90 % of the time", frac >= 0.9)) +} + +// ---- 12. One-phase night: planned charging on auto with Easee's steps ---- + +// fieldSteps is the field site's step table (22 Oct): one-phase levels +// 1 600-3 800 W and three-phase levels 4 800-11 100 W. +var fieldSteps = []float64{0, 1600, 1840, 2070, 2300, 2530, 2760, 2990, 3220, 3450, 3680, 3800, + 4800, 5520, 6210, 6900, 7590, 8280, 8970, 9660, 10350, 11100} + +func s12(c *runCtx, fuse float64) (map[string]any, error) { + // 3 × 20 A, house 1.6/1.7/2.3 A, an Easee-like charger applying in + // 40 s, a three-phase car, and a plan for 6.6 kWh by the first quarter + // hour 40 min out: 7-10 kW on three phases. One phase cannot deliver + // that, so this tests the cuts and the shortfall of the field night, + // not its one-phase choice (#1799 cause 1, which needs a lighter plan). + extra := map[string]any{"phase_mode": "auto", "allowed_steps_w": fieldSteps, "min_charge_w": 1600, "max_charge_w": 11100} + var ready time.Time + err := with(c, stackSpec{fuseA: fuse, loadsW: [3]float64{1.6 * volt, 1.7 * volt, 2.3 * volt}, socPct: 5, + chargers: []charger{{delayS: 40}}, lpExtra: extra}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(20) + var err error + if ready, err = st.planTo("garage", 40, 51, 40*time.Minute); err != nil { + return err + } + st.mark("plan") + sleepS(time.Until(ready).Seconds()) + st.mark("ready") + sleepS(30) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s12", "fuse": fuse, "plan_wh": 0.11 * 60000, "charger_delay_s": 40}, err +} + +func judgeS12(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + t0, tReady := d.marks["plan"], d.marks["ready"] + m := metrics{} + // Energy the car took between the plan and the ready time. + wh := 0.0 + w := d.window(t0, tReady) + for i := 1; i < len(w); i++ { + wh += carW(w[i], 0) * (w[i].T - w[i-1].T) / 3600 + } + plan := d.metaF("plan_wh") + m["delivered_wh"], m["plan_wh"] = math.Round(wh), plan + prs := connector1(profiles(d.frames, t0, tReady)) + ones, cuts := 0, 0 + for i, p := range prs { + if p.Limit > 0 && p.phasesOr3() == 1 { + ones++ + } + // A lower order while every phase has room is a cut on room alone, + // unless the plan itself asked for less (a pause between slots). + if i > 0 && p.offerW() < prs[i-1].offerW()-1 { + if r := d.at(t0 + p.T); r != nil && maxAbsI(r) < fuse-1-0.5 { + if after := d.at(t0 + p.T + 2); after == nil || firstLP(after).CommandedReason != "plan" { + cuts++ + } + } + } + } + // Seconds FTW gave fuse_limit while the worst phase had 6 A of room: + // recorded, not judged, since a slow charger's pending order may hold it. + limited := 0 + for _, r := range w { + if r.LPs != nil && firstLP(r).CommandedReason == "fuse_limit" && maxAbsI(r) < fuse-1-6 { + limited++ + } + } + m["one_phase_orders"], m["cuts_with_room"], m["profiles"], m["fuse_limit_with_room_s"] = ones, cuts, len(prs), limited + isBlown := anyBlown(d.window(t0, d.marks["end"])) + return m, []criterion{ + crit("fuse holds", !isBlown), + crit("planned charging never orders one phase", ones == 0), + crit("no order cut while every phase has room", cuts == 0), + crit("at least 90 % of the plan by the ready time", wh >= 0.9*plan), + } +} + +// at is the last sample at or before t. +func (d *runData) at(t float64) *sample { + var last *sample + for _, r := range d.samples { + if r.T > t { + break + } + last = r + } + return last +} + +// ---- 13. Charge now across a charger socket reconnect ---- + +func s13(c *runCtx, fuse float64) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + if err := st.chargeNow("garage", 16, "3p", 0); err != nil { + return err + } + sleepS(60) + st.mark("drop") + if err := st.event(houseEvent{Do: "reconnect", Device: "garage", ForS: 10}); err != nil { + return err + } + sleepS(180) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s13", "fuse": fuse, "house_a": 3, "drop_s": 10}, err +} + +func judgeS13(d *runData) (metrics, []criterion) { + tDrop, t1 := d.marks["drop"], d.marks["end"] + car := func(r *sample) float64 { return carW(r, 0) } + minAfter := minOf(d.window(tDrop, t1), car) + plugged := 0 + for _, r := range d.window(tDrop, t1) { + if lp := firstLP(r); r.LPs != nil && !lp.Plugged { + plugged++ + } + } + m := metrics{"car_min_after_drop": math.Round(minAfter), "car_w_last60": math.Round(meanOf(d.window(t1-60, t1), car)), + "unplugged_samples": plugged} + return m, []criterion{ + crit("Charge now goes on after the charger returns", minAfter >= 0.95*minW), + crit("no unplug seen", plugged == 0), + } +} + +// ---- 14. A one-phase car under a three-phase order ---- + +func s14(c *runCtx, fuse float64) (map[string]any, error) { + // A one-phase car on an 11 kW charger draws on L1 whatever FTW offers. + // A plan asks for more than it can take, so FTW should plan its 16 A + // on one phase (3.5-3.7 kW) and not call the charger not following. + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 5, chargers: []charger{{carPhases: 1}}}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(20) + // 24 kWh in about 45 min: more than even 11 kW gives, so FTW must + // charge from the start. + if _, err := st.planTo("garage", 40, 80, 30*time.Minute); err != nil { + return err + } + st.mark("plan") + sleepS(20 * 60) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s14", "fuse": fuse, "house_a": 3, "car_phases": 1}, err +} + +func judgeS14(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["plan"], d.marks["end"] + car := func(r *sample) float64 { return carW(r, 0) } + last := meanOf(d.window(t1-600, t1), car) + notFollowing := 0 + for _, r := range d.window(t0+180, t1) { + if firstLP(r).ControlFeedback.Status == "not_following" { + notFollowing++ + } + } + m := metrics{"car_w_last10min": math.Round(last), "not_following_samples": notFollowing} + return m, []criterion{ + crit("delivers 3.4-3.8 kW over the last 10 min", last >= 3400 && last <= 3800), + crit("never called not following", notFollowing == 0), + } +} + +// ---- 15. A charger wired on one phase: Charge now and planned charging ---- + +// onePhaseCharger is an Easee wired on L1: 6-16 A there, nothing on three +// phases. Its loadpoint names no phase mode, as most do. +var onePhaseCharger = charger{kw: 3.68, wiredPhases: 1} + +var onePhaseExtra = map[string]any{"min_charge_w": 1380, "max_charge_w": 3680, + "allowed_steps_w": easeeSteps[:12]} // 0, 1 380 ... 3 680 W + +// chargeNowAsTheApp sends Charge now at 16 A on the phases /api/loadpoints +// reports, as the app does. +func (st *stack) chargeNowAsTheApp(lpID string) (int, error) { + var lps struct { + Loadpoints []ftwLoadpoint `json:"loadpoints"` + } + if err := st.get("/api/loadpoints", &lps); err != nil { + return 0, err + } + phases := 3 + for _, lp := range lps.Loadpoints { + if lp.ID == lpID && lp.Phases > 0 { + phases = lp.Phases + } + } + mode := "3p" + if phases == 1 { + mode = "1p" + } + return phases, st.chargeNow(lpID, 0, mode, 16*volt*float64(phases)) +} + +func s15(c *runCtx, fuse float64) (map[string]any, error) { return s15mode(c, fuse, "") } + +// s15mode is s15 with the loadpoint's phase mode set ("" leaves it out). +func s15mode(c *runCtx, fuse float64, mode string) (map[string]any, error) { + extra := map[string]any{} + for k, v := range onePhaseExtra { + extra[k] = v + } + if mode != "" { + extra["phase_mode"] = mode + } + phases := 0 + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50, chargers: []charger{onePhaseCharger}, + lpExtra: extra}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + st.mark("charge_now") + var err error + if phases, err = st.chargeNowAsTheApp("garage"); err != nil { + return err + } + sleepS(180) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s15", "fuse": fuse, "house_a": 3, "api_phases": phases, "phase_mode": mode}, err +} + +func judgeS15(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["charge_now"], d.marks["end"] + const target = 16 * volt // all a one-phase charger gives + m := d.carSummary(t0, t1, target) + m["target_w"], m["api_phases"] = target, d.meta["api_phases"] + hold, _ := m["hold_fraction"].(*float64) + return m, []criterion{ + crit("16 A on L1 within 60 s", le(m["time_to_target_s"].(*float64), 60)), + crit("holds 95 % of the time", hold != nil && *hold >= 0.95), + crit("fuse holds", !m["fuse_blown"].(bool)), + crit("no fuse_limit with room to spare", m["fuse_limit_samples"].(int) == 0), + } +} + +func s15p(c *runCtx, fuse float64) (map[string]any, error) { + // 600 Wh by the first quarter hour 15 min out: about 10 min at 3.68 kW. + var ready time.Time + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 5, chargers: []charger{onePhaseCharger}, + lpExtra: onePhaseExtra}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(20) + var err error + if ready, err = st.planTo("garage", 40, 41, 15*time.Minute); err != nil { + return err + } + st.mark("plan") + sleepS(time.Until(ready).Seconds()) + st.mark("ready") + sleepS(30) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s15p", "fuse": fuse, "house_a": 3, "plan_wh": 0.01 * 60000}, err +} + +func judgeS15p(d *runData) (metrics, []criterion) { + t0, tReady := d.marks["plan"], d.marks["ready"] + wh := 0.0 + w := d.window(t0, tReady) + for i := 1; i < len(w); i++ { + wh += carW(w[i], 0) * (w[i].T - w[i-1].T) / 3600 + } + plan := d.metaF("plan_wh") + notFollowing := 0 + for _, r := range w { + if r.LPs != nil && firstLP(r).ControlFeedback.Status == "not_following" { + notFollowing++ + } + } + m := metrics{"delivered_wh": math.Round(wh), "plan_wh": plan, "not_following_samples": notFollowing, + "car_w_max": math.Round(maxOf(w, func(r *sample) float64 { return carW(r, 0) }))} + return m, []criterion{ + crit("fuse holds", !anyBlown(d.window(t0, d.marks["end"]))), + crit("at least 90 % of the plan by the ready time", wh >= 0.9*plan), + crit("never called not following", notFollowing == 0), + } +} + +// s15bat is Charge now on a one-phase charger while Core is raising its own +// battery order: PV climbs every 30 s (+2.5 kW for the second step, 1.5 kW +// for the others) and self-consumption charges the battery with it. The +// battery follows 8 s after each write, and Charge now goes 6 s after the +// second step, while Core's raise is on its way. On +// 16 A with 3 A of house load a raise of more than +// (15 - 3) x 230 - (1 380 + 230) = 1 150 W on its way once kept the car from +// starting at all (#1805). With the battery's 2 s the car started 25 s late +// when Charge now caught a raise in flight, and at once when it did not. +func s15bat(c *runCtx, fuse float64) (map[string]any, error) { + err := with(c, stackSpec{fuseA: fuse, loadsW: houseA(3), socPct: 50, chargers: []charger{onePhaseCharger}, + lpExtra: onePhaseExtra, batteryDelayS: 8}, func(st *stack) error { + if err := st.plug(0); err != nil { + return err + } + sleepS(30) + for i, pv := range []float64{2100, 4600, 6100, 7600, 9100} { + if err := st.event(houseEvent{Do: "pv", Device: "inverter", PVW: f64(pv)}); err != nil { + return err + } + if i == 1 { + sleepS(6) + st.mark("charge_now") + if _, err := st.chargeNowAsTheApp("garage"); err != nil { + return err + } + sleepS(24) + continue + } + sleepS(30) + } + sleepS(90) + st.mark("end") + return nil + }) + return map[string]any{"scenario": "s15bat", "fuse": fuse, "house_a": 3, "battery_delay_s": 8}, err +} + +func judgeS15bat(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["charge_now"], d.marks["end"] + fuse := d.metaF("fuse") + // Core's battery order: the EMS command and setpoint it wrote. + order := func(r *sample) float64 { + switch r.Bat.EMSCmd { + case 170: + return r.Bat.EMSSetW + case 187: + return -r.Bat.EMSSetW + } + return 0 + } + rise, low := 0.0, math.Inf(1) + for _, r := range d.window(t0-20, t0) { + low = math.Min(low, order(r)) + rise = math.Max(rise, order(r)-low) + } + start := d.firstT(t0, func(r *sample) bool { return carW(r, 0) >= 0.95*1380 }) + over, longest := 0.0, 0.0 + for _, r := range d.window(t0, t1) { + if maxAbsI(r) > fuse-1 { + over++ + longest = math.Max(longest, over) + } else { + over = 0 + } + } + m := metrics{"battery_rise_w_20s_before": math.Round(rise), "car_start_s": roundp(start, 1), + "car_w_last60": math.Round(meanOf(d.window(t1-60, t1), func(r *sample) float64 { return carW(r, 0) })), + "over_fuse_minus1_longest_s": longest, "max_abs_i": round(maxOf(d.window(t0, t1), maxAbsI), 1)} + return m, []criterion{ + crit("the battery order rose over 1 150 W in the 20 s before Charge now", rise > 1150), // the setup + crit("car charging within 10 s of Charge now", le(start, 10)), + crit("fuse holds", !anyBlown(d.window(t0, t1))), + crit("over fuse - 1 A at most 3 s", longest <= 3), + } +} diff --git a/go/test/site/stack_test.go b/go/test/site/stack_test.go new file mode 100644 index 000000000..aa4b156bc --- /dev/null +++ b/go/test/site/stack_test.go @@ -0,0 +1,537 @@ +//go:build unix + +package site + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "sync" + "syscall" + "time" +) + +// ocppPass is the shared OCPP credential the rig gives FTW and the chargers. +const ocppPass = "sim-ocpp-rig-20261021" + +// lane is the first of the 100 ports one rig uses on 127.0.0.1. The house +// (device-simulator's headless run) takes base+0 to base+59; FTW takes +// base+90 (API) and base+91 (OCPP). +type lane int + +func (l lane) simHTTP() int { return int(l) } +func (l lane) ftwAPI() int { return int(l) + 90 } +func (l lane) ftwOCPP() int { return int(l) + 91 } + +// ---- processes ---- + +type proc struct { + cmd *exec.Cmd + done chan struct{} +} + +// procs runs each child in its own process group, so a stop reaches +// everything it started. +type procs struct { + mu sync.Mutex + m map[string]*proc +} + +func (ps *procs) start(name string, argv []string, dir string, env []string, logPath string) error { + f, err := os.OpenFile(logPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + cmd := exec.Command(argv[0], argv[1:]...) + cmd.Dir = dir + cmd.Env = append(append(os.Environ(), "GOMAXPROCS=2"), env...) + cmd.Stdout, cmd.Stderr = f, f + cmd.SysProcAttr = childProcAttr() + if err := cmd.Start(); err != nil { + f.Close() + return fmt.Errorf("start %s: %w", name, err) + } + p := &proc{cmd: cmd, done: make(chan struct{})} + go func() { cmd.Wait(); f.Close(); close(p.done) }() + ps.mu.Lock() + if ps.m == nil { + ps.m = map[string]*proc{} + } + ps.m[name] = p + ps.mu.Unlock() + return nil +} + +func (ps *procs) stop(name string, sig syscall.Signal, wait time.Duration) { + ps.mu.Lock() + p := ps.m[name] + delete(ps.m, name) + ps.mu.Unlock() + if p == nil { + return + } + syscall.Kill(-p.cmd.Process.Pid, sig) + select { + case <-p.done: + case <-time.After(wait): + syscall.Kill(-p.cmd.Process.Pid, syscall.SIGKILL) + <-p.done + } +} + +func (ps *procs) stopAll() { + ps.mu.Lock() + names := make([]string, 0, len(ps.m)) + for n := range ps.m { + names = append(names, n) + } + ps.mu.Unlock() + for _, n := range names { + ps.stop(n, syscall.SIGTERM, 10*time.Second) + } +} + +// ---- HTTP ---- + +var client = &http.Client{Timeout: 5 * time.Second} + +// call sends body as JSON and decodes the answer into out when out is not +// nil. It returns the raw answer. +func call(method, url string, body, out any) ([]byte, error) { + var rd io.Reader + if body != nil { + b, err := json.Marshal(body) + if err != nil { + return nil, err + } + rd = bytes.NewReader(b) + } + req, err := http.NewRequest(method, url, rd) + if err != nil { + return nil, err + } + if body != nil { + req.Header.Set("Content-Type", "application/json") + } + resp, err := client.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + raw, err := io.ReadAll(resp.Body) + if err != nil { + return nil, err + } + if resp.StatusCode >= 300 { + return raw, fmt.Errorf("%s %s: %s: %s", method, url, resp.Status, bytes.TrimSpace(raw)) + } + if out != nil && len(raw) > 0 { + if err := json.Unmarshal(raw, out); err != nil { + return raw, fmt.Errorf("%s %s: %w", method, url, err) + } + } + return raw, nil +} + +func waitHTTP(url string, timeout time.Duration) error { + deadline := time.Now().Add(timeout) + var last error + for time.Now().Before(deadline) { + if _, last = call("GET", url, nil, nil); last == nil { + return nil + } + time.Sleep(500 * time.Millisecond) + } + return fmt.Errorf("%s did not answer: %v", url, last) +} + +// waitUntil polls pred every step until it holds or the timeout passes. +func waitUntil(timeout, step time.Duration, pred func() bool) bool { + deadline := time.Now().Add(timeout) + for time.Now().Before(deadline) { + if pred() { + return true + } + time.Sleep(step) + } + return false +} + +func now() float64 { return float64(time.Now().UnixMicro()) / 1e6 } + +// ---- the house: device-simulator's headless run ---- + +// houseSpec is device-simulator's run spec (docs/HEADLESS_RUN.md there). +type houseSpec struct { + Name string `json:"name"` + Site houseSite `json:"site"` + Devices []houseDevice `json:"devices"` + Timeline []houseEvent `json:"timeline,omitempty"` + Random *houseRandom `json:"random,omitempty"` +} + +// houseRandom asks the house to draw its own timeline from a seed. +type houseRandom struct { + Seed int64 `json:"seed"` + DurationS float64 `json:"duration_s"` + GapS float64 `json:"gap_s,omitempty"` +} + +type houseSite struct { + FuseA int `json:"fuse_a"` + FuseProfile string `json:"fuse_profile,omitempty"` + HouseW [3]float64 `json:"house_w"` + SignedPhaseCurrents *bool `json:"signed_phase_currents,omitempty"` +} + +type houseDevice struct { + ID string `json:"id"` + Driver string `json:"driver"` + CommandResponseS *float64 `json:"command_response_s,omitempty"` + StartupDelayS *float64 `json:"startup_delay_s,omitempty"` + LoadBalancerA *float64 `json:"load_balancer_a,omitempty"` + BatteryKWh float64 `json:"battery_kwh,omitempty"` + SoCPct *float64 `json:"soc_pct,omitempty"` + MinSoCPct *float64 `json:"min_soc_pct,omitempty"` + PVW float64 `json:"pv_w,omitempty"` + OCPPURL string `json:"ocpp_url,omitempty"` + MaxKW float64 `json:"max_kw,omitempty"` + Vendor string `json:"vendor,omitempty"` + Model string `json:"model,omitempty"` + PhaseSwitch bool `json:"phase_switch,omitempty"` + OCPPConfig map[string]string `json:"ocpp_config,omitempty"` + CarMaxA float64 `json:"car_max_a,omitempty"` + CarPhases int `json:"car_phases,omitempty"` + WiredPhases int `json:"wired_phases,omitempty"` +} + +// houseEvent is one change to the house (device-simulator's runEvent). +type houseEvent struct { + AtS float64 `json:"at_s,omitempty"` + Do string `json:"do"` + Device string `json:"device,omitempty"` + Label string `json:"label,omitempty"` + W *[3]float64 `json:"w,omitempty"` + A *[3]float64 `json:"a,omitempty"` + PVW *float64 `json:"pv_w,omitempty"` + Pct *float64 `json:"pct,omitempty"` + On *bool `json:"on,omitempty"` + MaxA *float64 `json:"max_a,omitempty"` + Phases *int `json:"phases,omitempty"` + ForS float64 `json:"for_s,omitempty"` +} + +type housePorts struct { + HTTP int `json:"http"` + Devices map[string]struct { + Driver string `json:"driver"` + Modbus int `json:"modbus_port"` + UnitID int `json:"unit_id"` + Serial string `json:"serial"` + } `json:"devices"` +} + +// ---- one run ---- + +// charger is one OCPP charger and the car on it. FTW's loadpoint has the +// charger's id. +type charger struct { + kw float64 // three-phase rating; default 11.04 + phaseSwitch bool + delayS float64 // applies a profile this late, like a cloud charger + carPhases int // 0: as many as offered + wiredPhases int // 1: wired on L1 only, kw its rating there + carMaxA float64 +} + +type stackSpec struct { + // house replays a spec the house wrote (timeline.json); random asks it + // to draw a timeline. Without either the house has no timeline. + house *houseSpec + random *houseRandom + + fuseA float64 + loadsW [3]float64 + pvW float64 + socPct float64 + chargers []charger // nil: one 11 kW charger + lpExtra map[string]any + heartbeatS int // FTW's OCPP heartbeat; 0 leaves Core's default + // batteryDelayS is how long the battery takes to follow an EMS write; + // 0 leaves the model's (2 s for sungrow). + batteryDelayS float64 +} + +type stack struct { + env *rigEnv + lane lane + rundir string + spec stackSpec + ps procs + ports housePorts + ftw string // FTW's API base URL + sim string // the house's API base URL + rec *recorder + cfg string + // markErr is the first mark the house did not take; the run cannot be + // judged without it. + markErr error +} + +func newStack(env *rigEnv, l lane, rundir string, spec stackSpec) *stack { + if spec.chargers == nil { + spec.chargers = []charger{{}} + } + return &stack{env: env, lane: l, rundir: rundir, spec: spec, + ftw: fmt.Sprintf("http://127.0.0.1:%d", l.ftwAPI()), + sim: fmt.Sprintf("http://127.0.0.1:%d", l.simHTTP())} +} + +func f64(v float64) *float64 { return &v } + +func (st *stack) houseSpec() houseSpec { + if st.spec.house != nil { + // A replay: the same house on this lane's ports. + h := *st.spec.house + h.Name = filepath.Base(st.rundir) + h.Random = nil + h.Devices = append([]houseDevice(nil), h.Devices...) + for i := range h.Devices { + if h.Devices[i].Driver == "ocpp" { + h.Devices[i].OCPPURL = fmt.Sprintf("ws://ftw:%s@127.0.0.1:%d/", ocppPass, st.lane.ftwOCPP()) + } + } + return h + } + h := houseSpec{Name: filepath.Base(st.rundir), Random: st.spec.random, + Site: houseSite{FuseA: int(st.spec.fuseA), FuseProfile: "iec_gg", HouseW: st.spec.loadsW}, + Devices: []houseDevice{{ID: "inverter", Driver: "sungrow", BatteryKWh: 10, + SoCPct: f64(st.spec.socPct), MinSoCPct: f64(5), PVW: st.spec.pvW}}} + if st.spec.batteryDelayS > 0 { + h.Devices[0].CommandResponseS = f64(st.spec.batteryDelayS) + } + for i, c := range st.spec.chargers { + var delay *float64 + if st.spec.random == nil { + delay = f64(c.delayS) // a random house draws its own + } + kw := c.kw + if kw == 0 { + kw = 11.04 + } + h.Devices = append(h.Devices, houseDevice{ID: lpIDs[i], Driver: "ocpp", + OCPPURL: fmt.Sprintf("ws://ftw:%s@127.0.0.1:%d/", ocppPass, st.lane.ftwOCPP()), + MaxKW: kw, Vendor: "RigCo", Model: fmt.Sprintf("Rig%.0f", kw), PhaseSwitch: c.phaseSwitch, CommandResponseS: delay, + CarPhases: c.carPhases, CarMaxA: c.carMaxA, WiredPhases: c.wiredPhases, + OCPPConfig: map[string]string{"HeartbeatInterval": "60", "MeterValueSampleInterval": "10"}}) + } + return h +} + +func (st *stack) up() error { + if err := os.RemoveAll(st.rundir); err != nil { + return err + } + if err := os.MkdirAll(st.rundir, 0o755); err != nil { + return err + } + spec, _ := json.MarshalIndent(st.houseSpec(), "", " ") + specPath := filepath.Join(st.rundir, "house.json") + if err := os.WriteFile(specPath, spec, 0o644); err != nil { + return err + } + houseDir := filepath.Join(st.rundir, "house") + if err := st.ps.start("house", []string{st.env.devsimBin, "run", "-spec", specPath, + "-port-base", fmt.Sprint(int(st.lane)), "-out", houseDir}, st.rundir, nil, + filepath.Join(st.rundir, "house.log")); err != nil { + return err + } + if !waitUntil(30*time.Second, 200*time.Millisecond, func() bool { + b, err := os.ReadFile(filepath.Join(houseDir, "ports.json")) + return err == nil && json.Unmarshal(b, &st.ports) == nil && len(st.ports.Devices) > 0 + }) { + return fmt.Errorf("the house never wrote ports.json (see house.log)") + } + if err := waitHTTP(st.sim+"/api/run", 10*time.Second); err != nil { + return err + } + var err error + if st.cfg, err = st.ftwConfig(); err != nil { + return err + } + if err := st.startFTW(); err != nil { + return err + } + if _, err := call("POST", st.ftw+"/api/mode", map[string]any{"mode": "planner_self"}, nil); err != nil { + return err + } + st.rec = &recorder{ftw: st.ftw, path: filepath.Join(st.rundir, "ftw.jsonl")} + st.rec.start() + return nil +} + +func (st *stack) startFTW() error { + if err := st.ps.start("ftw", []string{st.env.ftwBin, "-config", st.cfg, + "-web", filepath.Join(st.env.root, "web"), "-drivers", filepath.Join(st.env.root, "drivers")}, + st.rundir, []string{"GOMAXPROCS=4"}, filepath.Join(st.rundir, "ftw.log")); err != nil { + return err + } + return waitHTTP(st.ftw+"/api/status", 90*time.Second) +} + +func (st *stack) stopFTW() { st.ps.stop("ftw", syscall.SIGTERM, 30*time.Second) } + +func (st *stack) down() { + if st.rec != nil { + st.rec.close() + } + st.ps.stop("ftw", syscall.SIGTERM, 30*time.Second) + st.ps.stop("house", syscall.SIGTERM, 15*time.Second) + st.ps.stopAll() +} + +// event changes the house now. +func (st *stack) event(ev houseEvent) error { + _, err := call("POST", st.sim+"/api/run/event", ev, nil) + return err +} + +// mark names this moment in the house's record. +func (st *stack) mark(label string) { + if _, err := call("POST", st.sim+"/api/run/mark", map[string]string{"label": label}, nil); err != nil && st.markErr == nil { + st.markErr = fmt.Errorf("mark %s: %w", label, err) + } +} + +func (st *stack) get(path string, out any) error { + _, err := call("GET", st.ftw+path, nil, out) + return err +} + +func (st *stack) post(path string, body, out any) error { + _, err := call("POST", st.ftw+path, body, out) + return err +} + +func (st *stack) setLoadsA(a float64) error { + return st.event(houseEvent{Do: "loads", A: &[3]float64{a, a, a}}) +} + +func (st *stack) serial(i int) string { return st.ports.Devices[lpIDs[i]].Serial } + +// ---- FTW's API, once a second ---- + +type recorder struct { + ftw string + path string + mu sync.Mutex + stop chan struct{} + done chan struct{} +} + +func (r *recorder) start() { + r.stop, r.done = make(chan struct{}), make(chan struct{}) + go r.run() +} + +func (r *recorder) close() { + if r.stop == nil { + return + } + close(r.stop) + <-r.done +} + +func (r *recorder) append(rec map[string]any) { + b, _ := json.Marshal(rec) + r.mu.Lock() + defer r.mu.Unlock() + f, err := os.OpenFile(r.path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return + } + f.Write(append(b, '\n')) + f.Close() +} + +func (r *recorder) run() { + defer close(r.done) + for { + t0 := time.Now() + rec := map[string]any{"t": now()} + for key, path := range map[string]string{"status": "/api/status", "lps": "/api/loadpoints"} { + if b, err := call("GET", r.ftw+path, nil, nil); err != nil { + rec[key+"_err"] = err.Error() + } else { + rec[key] = json.RawMessage(b) + } + } + r.append(rec) + select { + case <-r.stop: + return + case <-time.After(time.Second - time.Since(t0)): + } + } +} + +// ---- FTW's configuration ---- + +var lpIDs = []string{"garage", "carport"} + +func (st *stack) ftwConfig() (string, error) { + var lps []map[string]any + for i := range st.spec.chargers { + lp := map[string]any{ + "id": lpIDs[i], "driver_name": st.serial(i), + "min_charge_w": 4140, "max_charge_w": 11040, "vehicle_capacity_wh": 60000, + } + for k, v := range st.spec.lpExtra { + lp[k] = v + } + lps = append(lps, lp) + } + ocpp := map[string]any{"enabled": true, "bind": "127.0.0.1", "port": st.lane.ftwOCPP(), "path": "/", + "username": "ftw", "password": ocppPass} + // 60 s is what every box configured from the old example has; 0 leaves + // it unset so Core's default applies. + if st.spec.heartbeatS > 0 { + ocpp["heartbeat_interval_s"] = st.spec.heartbeatS + } + modbus := func(id string) map[string]any { + d := st.ports.Devices[id] + return map[string]any{"modbus": map[string]any{"host": "127.0.0.1", "port": d.Modbus, "unit_id": max(d.UnitID, 1)}} + } + cfg := map[string]any{ + "site": map[string]any{"name": "rig", "control_interval_s": 2, "grid_target_w": 0, + "grid_tolerance_w": 42, "watchdog_timeout_s": 60}, + "fuse": map[string]any{"max_amps": st.spec.fuseA, "phases": 3, "voltage": 230}, + "drivers": []map[string]any{ + {"name": "meter", "lua": "drivers/sdm630.lua", "is_site_meter": true, "capabilities": modbus("meter")}, + {"name": "sungrow", "lua": "drivers/sungrow.lua", "battery_capacity_wh": 10000, "capabilities": modbus("inverter")}, + }, + "api": map[string]any{"port": st.lane.ftwAPI()}, + "ocpp": ocpp, + "loadpoints": lps, + "state": map[string]any{"path": filepath.Join(st.rundir, "state.db")}, + "app_link": map[string]any{"enabled": false}, + "fleet_ping": map[string]any{"enabled": false}, + "device_repository": map[string]any{"enabled": false}, + // A real box plans. A box without a planner runs Self, where the + // battery chases the raw meter and covers the car (dispatch.go), + // which hides the stress on the fuse. + "planner": map[string]any{"enabled": true, "engine": "core", "mode": "self_consumption"}, + "price": map[string]any{"provider": "elprisetjustnu", "zone": "SE3"}, + } + path := filepath.Join(st.rundir, "config.yaml") + b, err := json.MarshalIndent(cfg, "", " ") // JSON is valid YAML + if err != nil { + return "", err + } + return path, os.WriteFile(path, b, 0o644) +} diff --git a/go/test/site/testdata/found/.gitkeep b/go/test/site/testdata/found/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/go/test/site/verdicts_test.go b/go/test/site/verdicts_test.go new file mode 100644 index 000000000..9cb180db0 --- /dev/null +++ b/go/test/site/verdicts_test.go @@ -0,0 +1,553 @@ +//go:build unix + +package site + +import ( + "math" + "sort" +) + +// Each judge turns a finished run into numbers and pass criteria. The +// numbers and limits are the ones reports/simulatorprov-20261021 used. + +func judgeS0(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["connected"], d.marks["end"] + w := d.window(t0+20, t1) + lq, lr, today := counter{}, counter{}, counter{} + nAllRead := 0 + for _, r := range w { + st := status(r) + lq[str(st.LoadQuality)]++ + lr[str(st.LoadReason)]++ + today[str(st.Energy.Today.HistoryQuality)]++ + if allRead(r) { + nAllRead++ + } + } + var last *float64 + if len(w) > 0 { + last = status(w[len(w)-1]).Energy.Today.LoadWh + } + disc := d.disconnects(t0, t1-2) // the rig stops FTW at the end + m := metrics{"load_quality": lq, "load_reason": lr, "today_history_quality": today, + "all_read_samples": nAllRead, "samples_window": len(w), "today_load_wh_last": last, + "disconnects_s": disc} + return m, []criterion{ + crit("Home use measured in 95 % of samples", float64(lq["measured"]) >= 0.95*float64(len(w))), + crit("no OCPP disconnects", len(disc) == 0), + } +} + +func judgeS1(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["charge_now"], d.marks["end"] + fuse := d.metaF("fuse") + target := maxW + if fuse < 25 { + // Room on a 16 A fuse: fuse - 1 A margin - house, in 1 A steps. + target = math.Min(maxW, math.Floor(fuse-1-d.metaF("house_a"))*volt*3) + } + m := d.carSummary(t0, t1, target) + m["target_w"] = target + hold, _ := m["hold_fraction"].(*float64) + cs := []criterion{ + crit("target within 60 s", le(m["time_to_target_s"].(*float64), 60)), + crit("holds 95 % of the time", hold != nil && *hold >= 0.95), + crit("fuse holds", !m["fuse_blown"].(bool)), + } + if fuse >= 25 { + cs = append(cs, crit("no fuse_limit with room to spare", m["fuse_limit_samples"].(int) == 0)) + } + return m, cs +} + +func judgeS2(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["charge_now"], d.marks["end"] + pre := d.window(t0-15, t0) + var exp [3]float64 + for k := 0; k < 3; k++ { + exp[k] = round(meanOf(pre, func(r *sample) float64 { return phaseI(r)[k] }), 2) + } + m := d.carSummary(t0, t1, maxW) + m["export_a_before"] = exp + return m, []criterion{ + crit("car starts within 60 s", le(m["time_to_start_s"].(*float64), 60)), + crit("fuse holds", !m["fuse_blown"].(bool)), + } +} + +func judgeS3(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + tCN, tStep, tBack, t1 := d.marks["charge_now"], d.marks["step"], d.marks["step_back"], d.marks["end"] + pre := d.window(tStep-30, tStep) + m := metrics{ + "pre_step_car_w": math.Round(meanOf(pre, func(r *sample) float64 { return carW(r, 0) })), + "pre_step_bat_w": math.Round(meanOf(pre, batSigned)), + "pre_step_max_i": round(maxOf(pre, maxAbsI), 2), + } + m["post_step_max_i"] = round(maxOf(d.window(tStep, tBack), maxAbsI), 2) + m["over_fuse_longest_s"], m["over_fuse_total_s"] = d.runsOver(tStep, tBack, fuse) + overLongest, overTotal := d.runsOver(tStep, tBack, fuse-1) + m["over_fuse_minus1_longest_s"], m["over_fuse_minus1_total_s"] = overLongest, overTotal + m["i2_over_fuse_minus1_s"], m["i2_peak_a"] = d.i2Over(tCN, t1, fuse-1) + m["back_under_fuse_minus1_s"] = roundp(d.firstT(tStep, func(r *sample) bool { return maxAbsI(r) <= fuse-1 && r.T > tStep+1 }), 1) + last := d.window(tBack-60, tBack) + m["post_step_car_w_last60"] = math.Round(meanOf(last, func(r *sample) float64 { return carW(r, 0) })) + m["post_step_bat_w_last60"] = math.Round(meanOf(last, batSigned)) + var after []*sample + for _, r := range d.samples { + if r.T >= tCN { + after = append(after, r) + } + } + m["fuse_damage_max"] = round(maxOf(after, damage), 4) + isBlown := anyBlown(d.samples) + m["fuse_blown"] = isBlown + // Device response + meter and tick (2 + 2 s) + 3 s. + allowed := d.metaF("charger_delay_s") + 2 + 2 + 3 + m["allowed_over_s"] = allowed + m["run_over_fuse_minus1_longest_s"], _ = d.runsOver(tCN, t1, fuse-1) + m["run_max_i"] = round(maxOf(after, maxAbsI), 2) + // Before the step, layers with different margins may hold a phase + // between fuse - 1 A and fuse - 0.5 A; reported, not judged. + m["pre_step_over_fuse_minus1_longest_s"], _ = d.runsOver(tCN, tStep, fuse-1) + m["pre_step_over_fuse_minus_half_longest_s"], _ = d.runsOver(tCN, tStep, fuse-0.5) + return m, []criterion{ + crit("fuse holds", !isBlown), + crit("over fuse - 1 A no longer than the response time + 3 s after the step", overLongest <= allowed), + } +} + +func judgeS4(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["charge_now"], d.marks["end"] + m := d.carSummary(t0, t1, maxW) + tx := connector1(profiles(d.frames, t0, t1)) + m["profiles_sent"] = len(tx) + offers := map[float64]bool{} + for _, p := range tx { + offers[math.Round(p.offerW())] = true + } + m["distinct_offers_w"] = sortedKeys(offers) + zeros := 0 + for i := 1; i < len(tx); i++ { + if tx[i-1].offerW() > 0 && tx[i].offerW() == 0 { + zeros++ + } + } + m["offer_drops_to_zero"] = zeros + return m, []criterion{ + crit("no stops", m["stops_after_start"].(int) == 0), + crit("no offer drops to 0 W", zeros == 0), + crit("car starts", m["time_to_start_s"].(*float64) != nil), + crit("full power over the last minute", m["car_w_last60_mean"].(float64) >= 0.95*maxW), + } +} + +func sortedKeys(set map[float64]bool) []float64 { + out := make([]float64, 0, len(set)) + for k := range set { + out = append(out, k) + } + sort.Float64s(out) + return out +} + +func judgeS5(d *runData) (metrics, []criterion) { + t0, t1 := d.marks["plugged"], d.marks["end"] + prs := connector1(profiles(d.frames, t0, t1)) + ratingA := maxW / (3 * volt) + var bad []profile + over, maxOffer := 0, 0.0 + offers := map[float64]bool{} + type ap struct { + A float64 + Phases float64 + } + offersAP := map[ap]bool{} + for _, p := range prs { + offers[math.Round(p.offerW())] = true + offersAP[ap{round(p.Limit, 2), p.phasesOr3()}] = true + maxOffer = math.Max(maxOffer, p.perPhaseA()) + if p.Limit <= 0 { + continue + } + a := p.perPhaseA() + // A valid step is 6-16 A per phase: the charger is rated 16 A. + if a < 6-1e-6 || a > ratingA+1e-6 { + bad = append(bad, p) + } + if a > ratingA+1e-6 { + over++ + } + } + w := d.window(t0+60, t1) + pre := d.window(t0-10, t0+5) + surplus := meanOf(pre, func(r *sample) float64 { + i := phaseI(r) + return -(i[0] + i[1] + i[2]) * volt + }) + carMean := math.Round(meanOf(w, func(r *sample) float64 { return carW(r, 0) })) + charging := 0 + for _, r := range w { + if carW(r, 0) >= 0.95*minW { + charging++ + } + } + frac := round(float64(charging)/math.Max(1, float64(len(w))), 3) + rs := counter{} + for _, r := range w { + for _, x := range reasons(r) { + rs[x]++ + } + } + all := d.window(t0, t1) + isBlown := anyBlown(all) + if len(bad) > 5 { + bad = bad[:5] + } + m := metrics{ + "surplus_w_at_plug": math.Round(surplus), + "profiles_sent": len(prs), + "invalid_examples": bad, + "offers_w": firstN(sortedKeys(offers), 12), + "car_w_mean_after_60s": carMean, + "charging_fraction_ge_min": frac, + "time_to_start_s": roundp(d.firstT(t0, func(r *sample) bool { return carW(r, 0) >= 0.95*minW }), 1), + "reasons": rs.top(6), + "offers_over_rating": over, + "max_offer_a": round(maxOffer, 2), + "fuse_blown": isBlown, + "fuse_damage_max": round(maxOf(all, damage), 4), + "surplus_used_fraction": round(carMean/math.Max(1, math.Round(surplus)), 3), + } + nInvalid := 0 + for _, p := range prs { + if a := p.perPhaseA(); p.Limit > 0 && (a < 6-1e-6 || a > ratingA+1e-6) { + nInvalid++ + } + } + m["invalid_steps"] = nInvalid + return m, []criterion{ + // The beta criteria (Fredrik, 21 Oct): no offer above the charger's + // rating per phase, and the fuse never trips. + crit("no offer above the charger's rating", over == 0), + crit("fuse holds", !isBlown), + crit("only valid steps (6-16 A)", nInvalid == 0), + crit("charges at least the minimum 90 % of the time", frac >= 0.9), + } +} + +func firstN(xs []float64, n int) []float64 { + if len(xs) > n { + return xs[:n] + } + return xs +} + +type qualitySummary struct { + Samples int `json:"samples"` + LoadWShown int `json:"load_w_shown"` + LoadQuality map[string]int `json:"load_quality"` + TodayFirst *float64 `json:"today_load_wh_first"` + TodayLast *float64 `json:"today_load_wh_last"` +} + +func summarize(ws []*sample) qualitySummary { + q := qualitySummary{Samples: len(ws), LoadQuality: counter{}} + for i, r := range ws { + st := status(r) + q.LoadQuality[str(st.LoadQuality)]++ + if st.LoadW != nil { + q.LoadWShown++ + } + if i == 0 { + q.TodayFirst = st.Energy.Today.LoadWh + } + q.TodayLast = st.Energy.Today.LoadWh + } + return q +} + +func judgeS6(d *runData) (metrics, []criterion) { + tOff, tOn, t1 := d.marks["offline"], d.marks["online"], d.marks["end"] + before := summarize(d.window(tOff-40, tOff)) + during := summarize(d.window(tOff+15, tOn)) + after := summarize(d.window(tOn+30, t1)) + // From 20 s after plug-in: FTW's start and the charger's first + // MeterValues after plug-in are reported apart (plug_in_approx_s). + plugT, ok := d.marks["plug"] + if !ok && len(d.samples) > 0 { + plugT = d.samples[0].T + } + var viol []*sample + for _, r := range d.samples { + if r.T >= plugT+20 && allRead(r) && str(status(r).LoadQuality) != "measured" && !(tOff <= r.T && r.T < tOn+30) { + viol = append(viol, r) + } + } + plugIn := 0 + for _, r := range d.window(plugT, plugT+20) { + if allRead(r) && str(status(r).LoadQuality) != "measured" { + plugIn++ + } + } + var firstViol *float64 + if len(viol) > 0 { + v := round(viol[0].T-tOff, 1) + firstViol = &v + } + marked := 0 + for k, v := range during.LoadQuality { + if k != "null" && k != "" && k != "measured" { + marked += v + } + } + m := metrics{"before": before, "during": during, "after": after, + "approx_while_all_read": len(viol), "approx_while_all_read_first": firstViol, + "load_w_missing_during": during.Samples - during.LoadWShown, "plug_in_approx_s": plugIn} + return m, []criterion{ + crit("Home use shown throughout the outage", during.LoadWShown == during.Samples), + crit("Home use marked estimated throughout the outage", marked == during.Samples), + crit("measured before the outage", before.LoadQuality["measured"] == before.Samples), + crit("measured after the outage", after.LoadQuality["measured"] == after.Samples), + crit("never estimated while every driver reads", len(viol) == 0), + } +} + +func judgeS7(d *runData) (metrics, []criterion) { + tCN, tStop, tUp, t1 := d.marks["charge_now"], d.marks["ftw_stop"], d.marks["ftw_up"], d.marks["end"] + car := func(r *sample) float64 { return carW(r, 0) } + pre := d.window(tStop-30, tStop) + preCar := meanOf(pre, car) + post := d.window(tUp, t1) + m := metrics{"pre_car_w": math.Round(preCar), "downtime_s": round(tUp-tStop, 1)} + var before *float64 + if len(pre) > 0 { + m["pre_load_w"] = status(pre[len(pre)-1]).LoadW + before = status(pre[len(pre)-1]).Energy.Today.LoadWh + } + m["today_load_wh_before"] = before + var afterFirst, afterLast *float64 + nullAfter := 0 + hq, lq := counter{}, counter{} + for _, r := range post { + // A sample before the rig's first status read after the restart + // has no reading; only a status without the day total counts. + if r.Status == nil { + continue + } + st := status(r) + v := st.Energy.Today.LoadWh + if v == nil { + nullAfter++ + } else if afterFirst == nil { + afterFirst = v + } + afterLast = v + hq[str(st.Energy.Today.HistoryQuality)]++ + lq[str(st.LoadQuality)]++ + } + m["today_load_wh_after_first"], m["today_load_wh_null_after"], m["today_load_wh_after_last"] = afterFirst, nullAfter, afterLast + m["history_quality_after"], m["load_quality_after"] = hq, lq + loadBack := d.firstT(tUp, func(r *sample) bool { return status(r).LoadW != nil }) + m["load_w_back_s"] = roundp(loadBack, 1) + m["car_back_s"] = roundp(d.firstT(tUp, func(r *sample) bool { return preCar > 0 && car(r) >= 0.95*preCar }), 1) + m["car_resumed_s"] = roundp(d.firstT(tUp, func(r *sample) bool { return car(r) >= 0.95*minW }), 1) + m["car_w_150s_after_up"] = math.Round(meanOf(d.window(tUp+90, tUp+150), car)) + rs := counter{} + for _, r := range d.window(tUp+10, t1) { + for _, x := range reasons(r) { + rs[x]++ + } + } + m["reasons_after_up"] = rs.top(4) + m["car_min_during_restart"] = math.Round(minOf(d.window(tStop, tUp+60), car)) + m["car_w_last60"] = math.Round(meanOf(d.window(t1-60, t1), car)) + if tk, ok := d.marks["today_known"]; ok { + m["today_known_after_s"] = math.Round(tk - tCN) + } + minAfterStop := minOf(d.window(tStop, t1), car) + m["car_min_after_stop"] = math.Round(minAfterStop) + var fullFrom *float64 + dip, dipMin := 0, preCar + for _, r := range post { + if car(r) >= 0.95*preCar { + if fullFrom == nil { + v := r.T - tUp + fullFrom = &v + } + } else { + fullFrom = nil + dip++ + dipMin = math.Min(dipMin, car(r)) + } + } + m["full_power_from_s"], m["dip_s"], m["dip_min_w"] = roundp(fullFrom, 1), dip, math.Round(dipMin) + cs := []criterion{ + crit("Home use back within 30 s", le(loadBack, 30)), + // Charge now continues without a stop and is back at full power + // within 420 s of Core being up, for at least the last 30 s. The + // dip is recorded. + crit("Charge now never stops", minAfterStop >= 0.95*minW), + crit("full power within 420 s", fullFrom != nil && *fullFrom <= 420 && t1-tUp-*fullFrom >= 30), + } + if d.meta["short"] != true { + cs = append([]criterion{crit("the day's Home use is kept", before != nil && nullAfter == 0 && + afterLast != nil && *afterLast >= *before*0.99)}, cs...) + } + return m, cs +} + +func minOf(ws []*sample, f func(*sample) float64) float64 { + if len(ws) == 0 { + return 0 + } + m := math.Inf(1) + for _, r := range ws { + m = math.Min(m, f(r)) + } + return m +} + +func judgeS8(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + t0, t1 := d.marks["charge_now"], d.marks["end"] + m := d.carSummary(t0, t1, maxW) + m["over_fuse_longest_s"], m["over_fuse_total_s"] = d.runsOver(t0, t1, fuse) + overLongest, overTotal := d.runsOver(t0, t1, fuse-1) + m["over_fuse_minus1_longest_s"], m["over_fuse_minus1_total_s"] = overLongest, overTotal + m["i2_over_fuse_minus1_s"], m["i2_peak_a"] = d.i2Over(t0, t1, fuse-1) + var prs [][2]float64 + for i, p := range connector1(profiles(d.frames, t0, t1)) { + if i >= 20 { + break + } + prs = append(prs, [2]float64{p.T, math.Round(p.offerW())}) + } + m["profiles"] = prs + allowed := d.metaF("charger_delay_s") + 2 + 2 + 3 + m["allowed_over_s"] = allowed + if plug, ok := d.marks["plug"]; ok { + // Before FTW's first profile reaches the charger it draws its maximum. + m["plugin_max_i"] = round(maxOf(d.window(plug, t0), maxAbsI), 2) + m["plugin_over_fuse_longest_s"], _ = d.runsOver(plug, t0, fuse) + } + return m, []criterion{ + crit("fuse holds", !m["fuse_blown"].(bool)), + crit("over fuse - 1 A no longer than the charger's delay + 7 s", overLongest <= allowed), + } +} + +func judgeS9(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + tCN, tSw, t1 := d.marks["charge_now"], d.marks["switch"], d.marks["end"] + before, after := d.overage(tCN, tSw, fuse), d.overage(tSw, t1, fuse) + car := func(r *sample) float64 { return carW(r, 0) } + m := metrics{"before": before, "after": after, + "before_phase_a": d.phaseMean(tSw-30, tSw), "after_phase_a_last60": d.phaseMean(t1-60, t1), + "car_w_before": math.Round(meanOf(d.window(tSw-30, tSw), car)), + "car_w_last60": math.Round(meanOf(d.window(t1-60, t1), car))} + var seq [][3]any + for _, p := range connector1(profiles(d.frames, tSw-5, t1)) { + a := round(p.Limit, 2) + if len(seq) == 0 || seq[len(seq)-1][1] != a || !samePhases(seq[len(seq)-1][2], p.Phases) { + seq = append(seq, [3]any{p.T, a, p.Phases}) + } + } + if len(seq) > 12 { + seq = seq[:12] + } + m["profiles_after_switch"] = seq + return m, []criterion{ + crit("fuse holds", !before["fuse_blown"].(bool) && !after["fuse_blown"].(bool)), + crit("over fuse - 1 A at most 3 s on 3p", before["over_limit_longest_s"].(float64) <= 3), + crit("over fuse - 1 A at most 3 s after the switch to 1p", after["over_limit_longest_s"].(float64) <= 3), + } +} + +func samePhases(a any, b *float64) bool { + pa, _ := a.(*float64) + if pa == nil || b == nil { + return pa == nil && b == nil + } + return *pa == *b +} + +func judgeS10(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + tCN, tSt, t1 := d.marks["charge_now"], d.marks["step"], d.marks["end"] + before, after := d.overage(tCN, tSt, fuse), d.overage(tSt, t1, fuse) + m := metrics{"before": before, "after": after} + pre, last := d.window(tSt-30, tSt), d.window(t1-60, t1) + var preW, lastW [2]float64 + for i := 0; i < 2; i++ { + i := i + preW[i] = math.Round(meanOf(pre, func(r *sample) float64 { return carW(r, i) })) + lastW[i] = math.Round(meanOf(last, func(r *sample) float64 { return carW(r, i) })) + } + m["pre_step_w"], m["last60_w"] = preW, lastW + for i, name := range lpIDs { + var seq [][2]float64 + for _, p := range connector1(profiles(d.loadFrames(i), tSt-5, t1)) { + a := round(p.Limit, 1) + if len(seq) == 0 || seq[len(seq)-1][1] != a { + seq = append(seq, [2]float64{p.T, a}) + } + } + if len(seq) > 10 { + seq = seq[:10] + } + m[name+"_profiles_after_step"] = seq + } + return m, []criterion{ + crit("fuse holds", !before["fuse_blown"].(bool) && !after["fuse_blown"].(bool)), + crit("over fuse - 1 A at most 3 s after the step", after["over_limit_longest_s"].(float64) <= 3), + } +} + +func judgeS11(d *runData) (metrics, []criterion) { + fuse := d.metaF("fuse") + tCN, tP, tR, t1 := d.marks["charge_now"], d.marks["pause"], d.marks["restart"], d.marks["end"] + car := func(r *sample) float64 { return carW(r, 0) } + prs := connector1(profiles(d.frames, tCN-1, t1)) + var firstPos, paused *profile + for i := range prs { + p := &prs[i] + if firstPos == nil && p.T >= tR-tCN && p.Limit > 0 { + firstPos = p + } + if paused == nil && p.T >= tP-tCN && p.Limit == 0 { + paused = p + } + } + m := metrics{ + "car_w_before_pause": math.Round(meanOf(d.window(tP-20, tP), car)), + "car_started_s": roundp(d.firstT(tR, func(r *sample) bool { return car(r) >= 0.95*minW }), 1), + "car_w_last60": math.Round(meanOf(d.window(t1-60, t1), car)), + } + var firstPosS *float64 + if paused != nil { + m["pause_profile_at_s"] = round(paused.T-(tP-tCN), 1) + } + if firstPos != nil { + v := round(firstPos.T-(tR-tCN), 1) + firstPosS = &v + m["first_positive_offer_a"] = firstPos.Limit + } + m["first_positive_offer_s"] = firstPosS + rs := counter{} + for _, r := range d.window(tR+2, t1) { + for _, x := range reasons(r) { + rs[x]++ + } + } + m["reasons_after_restart"] = rs.top(4) + run := d.overage(tCN, t1, fuse) + for k, v := range run { + m["run_"+k] = v + } + return m, []criterion{ + crit("an offer within 60 s of the restart", le(firstPosS, 60)), + crit("over fuse - 1 A at most 3 s", run["over_limit_longest_s"].(float64) <= 3), + crit("fuse holds", !run["fuse_blown"].(bool)), + } +} diff --git a/package-lock.json b/package-lock.json index 749734c15..ffe734cdd 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "ftw", - "version": "0.154.2", + "version": "0.155.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "ftw", - "version": "0.154.2", + "version": "0.155.1", "license": "SEE LICENSE IN LICENSE", "devDependencies": { "@changesets/cli": "^3.0.3" diff --git a/package.json b/package.json index aea080754..875718486 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "ftw", - "version": "0.154.2", + "version": "0.155.1", "private": true, "type": "module", "description": "FTW — local-first home energy coordination. Version metadata only; the runtime is Go and the mathematical planner is the compiled Energyplan worker.", diff --git a/web/app-link-tab.test.mjs b/web/app-link-tab.test.mjs index 5dca9e744..4b798b00b 100644 --- a/web/app-link-tab.test.mjs +++ b/web/app-link-tab.test.mjs @@ -91,16 +91,16 @@ describe("the app tab", () => { assert.match(render({}), /id="app-link-pair"[^>]*disabled/); }); - it("keeps fleet statistics with the app, but independent of its switch", () => { + it("keeps statistics with the app, but independent of its switch", () => { const appWithFleet = loadTab(true); const config = { app_link: { enabled: false } }; const html = appWithFleet.render({ config }); - assert.match(html, /Anonymous daily report<\/legend>/); - assert.match(html, /data-checkbox-path="fleet_ping\.enabled"/); - assert.equal(config.fleet_ping.enabled, true, "fleet sharing should still default on"); + assert.match(html, /Anonymous statistics<\/legend>/); + assert.match(html, /id="statistics-share"/); + assert.doesNotMatch(html, /data-checkbox-path="fleet_ping\.enabled"/); + assert.equal(config.fleet_ping, undefined, "statistics are not a config field"); assert.doesNotMatch(html, /id="app-link-enabled"[^>]*checked/); - assert.match(html, /id="fleet-ping-enabled"[^>]*checked/); }); it("refreshes the fleet payload after saving from the app page", () => { @@ -119,7 +119,7 @@ describe("the app tab", () => { vm.runInContext(fleetSource, sandbox); win.FTWSettings.tabs.app.afterSave(); - assert.deepEqual(asked, ["/api/fleet-ping", "/api/app-link/status"]); + assert.deepEqual([...asked].sort(), ["/api/app-link/status", "/api/cloud", "/api/cloud/stats-policy", "/api/fleet-ping"]); }); it("does not leave a standalone Fleet ping destination behind", () => { diff --git a/web/app.js b/web/app.js index e0be5262e..9a795e1ee 100644 --- a/web/app.js +++ b/web/app.js @@ -722,7 +722,8 @@ loadW.textContent = formatW(data.load_w); var loadDir = document.getElementById("load-dir"); if (loadDir) { - loadDir.textContent = data.load_w == null || !isFinite(data.load_w) ? "no data" : "using now"; + loadDir.textContent = data.load_w == null || !isFinite(data.load_w) ? "no data" + : data.load_quality === "estimated" ? "estimated" : "using now"; } // EV tile (tile-mode parity with the energy-flow's EV planet). @@ -2967,7 +2968,9 @@ ? "Charging on plan until " + evFmtClock(lp.plan_next_end_ms) + "." + kwPlanned : "Charging."; if (lp.commanded_reason === "fuse_limit") { - text += " Rate is limited by site power limits right now."; + text += " Rate is limited by the main fuse right now."; + } else if (lp.commanded_reason === "import_limit") { + text += " Rate is limited by the hourly import limit right now."; } } else if (lp.goal_complete === true) { text = "The car confirmed this charging goal is complete."; @@ -2992,7 +2995,10 @@ // The specific pauses beat the generic branches — this is the // line that stops an operator debugging cable and charger while // the box is protecting the main fuse (#1009). - text = "Paused by site power limits — the house is using the headroom; charging resumes on its own." + kwPlanned; + text = "Paused by the main fuse — the house is using the headroom; charging resumes on its own." + kwPlanned; + tone = "var(--text)"; + } else if (lp.commanded_known && !lp.commanded_w && lp.commanded_reason === "import_limit") { + text = "Paused by the hourly import limit — this hour’s import budget is used; charging resumes on its own." + kwPlanned; tone = "var(--text)"; } else if (lp.commanded_known && !lp.commanded_w && lp.commanded_reason === "site_meter_stale") { text = "Paused for safety: site-meter data is stale — charging resumes when telemetry recovers."; @@ -3337,6 +3343,7 @@ battery_hold: "a battery hold took priority", core_mode: "the site mode does not allow it", fuse_safety_block: "fuse protection stopped it", + import_limit_block: "the hourly import limit stopped it", restart_lease_invalid: "the saved lease was no longer safe after a restart", }; diff --git a/web/components/energy-flow-readings.js b/web/components/energy-flow-readings.js index 0ccd48a8c..a4741baba 100644 --- a/web/components/energy-flow-readings.js +++ b/web/components/energy-flow-readings.js @@ -207,13 +207,15 @@ export function flowReadingsFromStatus(status, opts) { if (!live[role]) planets.push(...offline[role]); } - const hasEstimate = status?.load_estimate != null; + // Core says "estimated" when it left an unread device out of Home use. + const hasEstimate = status?.load_quality === "estimated"; // Core qualifies this balance, including missing flows and stale meters. // A past energy interval is separate from Power now: it must neither hide // nor overwrite current household power. Never derive another balance from // independently smoothed display readings here. const rawLoad = num(status?.load_w); - const powerBalanceW = rawLoad !== null && rawLoad >= 0 ? rawLoad : null; + // Shown as Core sends it. Negative means production Core cannot see. + const powerBalanceW = rawLoad; const loadW = powerBalanceW; // Today's share is kWh already in the box, not the live meter. A // quiet meter blanks % SELF-POWERED NOW (unknown instantaneous load) diff --git a/web/control-feedback.js b/web/control-feedback.js index dcdbbfed0..eaa25a169 100644 --- a/web/control-feedback.js +++ b/web/control-feedback.js @@ -63,9 +63,12 @@ const REASON_TITLES = { no_plan_budget: 'Waiting for the plan', pv_surplus_pause: 'Waiting for spare solar', wake_kick: 'Starting the charger', vehicle_complete: 'Car is full', vehicle_limit_completion: 'Car reached its limit', vehicle_not_requesting: 'Car not taking power', battery_full: 'Battery full', battery_nearly_full: 'Battery nearly full', battery_nearly_empty: 'Battery nearly empty', - core_limit: 'Limited by FTW', fuse_limit: 'Limited by site power limits', fuse_cooldown: 'Waiting after a site power limit', + core_limit: 'Limited by FTW', fuse_limit: 'Limited by the main fuse', fuse_cooldown: 'Waiting after a site power limit', + import_limit: 'Limited by the hourly import limit', charger_limit: 'Limited by FTW’s charger setting', device_limit: 'Charger limit', offered_current_lower: 'Charger offers less current', load_balancer_limit: 'Limited by the charger’s load balancer', + one_phase: 'Charging on one phase', + device_fault: 'Device fault', telemetry_stale: 'No fresh readings', readings_lost: 'Lost control', command_failed: 'Commands failing', command_unconfirmed: 'Command not confirmed', default_failed: 'Cannot hand back control', response_unknown: 'No power reading', observe_only: 'Monitoring only', disabled: 'Disabled', device_control: 'Own control', @@ -93,7 +96,8 @@ function sentence(row) { case 'battery_nearly_full': return `Taking ${controlPower(actual)} of ${controlPower(row.sent_w)} at ${percent(row.battery_soc)}. A battery charges slower when it is nearly full.`; case 'battery_nearly_empty': return `Giving ${controlPower(actual)} of ${controlPower(row.sent_w)} at ${percent(row.battery_soc)}. A battery limits discharge when it is nearly empty.`; case 'core_limit': return `FTW’s safety limits set the target to ${target(row)}.`; - case 'fuse_limit': return `FTW lowered the rate to ${target(row)} to stay within the site’s fuse and import limits.`; + case 'fuse_limit': return `FTW lowered the rate to ${target(row)} to protect the main fuse.`; + case 'import_limit': return `FTW lowered the rate to ${target(row)} to keep this hour’s grid import within its limit.`; case 'fuse_cooldown': return 'FTW waits a moment after a site power limit before it raises the rate.'; case 'charger_limit': return 'The charger limit in FTW’s settings caps the rate.'; case 'device_limit': return `The charger’s own limit is ${amps(row.device_limit_a)}, below the ${amps(row.requested_a)} FTW asked for.`; @@ -101,6 +105,8 @@ function sentence(row) { case 'load_balancer_limit': return num(actual) && actual >= 100 ? `Asked for ${target(row)}, delivering ${controlPower(actual)}. The charger’s load balancer is limiting the car.` : `Asked for ${target(row)}, but the charger’s load balancer gives the car no current.`; + case 'one_phase': return `The car charges on ${row.charging_phase || 'one phase'} only${num(row.charging_phase_max_w) ? `, at most ${controlPower(row.charging_phase_max_w)}` : ''}. FTW plans with what it takes.`; + case 'power_below_target': case 'power_above_target': return `Asked for ${target(row)}, delivering ${controlPower(actual)}.`; case 'power_wrong_direction': return `Asked for ${target(row)}, but it is ${doing(row, actual)} ${controlPower(actual)}.`; case 'no_power_response': return `Asked for ${target(row)}, but it delivers no power.`; @@ -153,6 +159,14 @@ function proofLine(row) { // can act, the next step. Tone follows Core's severity and status. export function controlStatus(row, live = true) { if (!live) return {title: 'Status not current', text: 'FTW has not reported for a while.', proof: '', next: 'Waiting for a fresh report.', tone: 'stale'}; + const headroom = row.kind === 'ev' && row.reason === 'power_below_target' && row.readings_fresh && row.site_headroom?.evidence === 'calculated' ? row.site_headroom : null; + if (headroom && [headroom.fuse_a, headroom.margin_a, headroom.projected_max_a, headroom.battery_charge_w].every(num)) { + const battery = headroom.battery_charge_w > 100 ? ` The home battery is charging at ${controlPower(headroom.battery_charge_w)}.` : ''; + const current = value => `${Number(value.toFixed(1))} A`; + return {title: 'Insufficient site headroom', text: sentence(row) + battery, + next: `With other loads unchanged, the requested EV power would raise the busiest phase to about ${current(headroom.projected_max_a)}, above the ${current(headroom.fuse_a - headroom.margin_a)} working limit (${current(headroom.fuse_a)} fuse, ${current(headroom.margin_a)} safety margin).`, + proof: 'Calculated from site measurements. This is a projection, not a charger-reported cause.', tone: 'warning'}; + } const title = REASON_TITLES[row.reason] || STATUS_TITLES[row.status] || 'Checking'; const tone = row.severity === 'alarm' ? 'alarm' : row.severity === 'warning' ? 'warning' : row.status === 'following' ? 'ok' : 'neutral'; return {title, text: sentence(row), proof: proofLine(row), next: nextStep(row), tone}; diff --git a/web/control-feedback.test.mjs b/web/control-feedback.test.mjs index 317df0962..27b430d76 100644 --- a/web/control-feedback.test.mjs +++ b/web/control-feedback.test.mjs @@ -67,6 +67,13 @@ test('expected limits stay calm and explain themselves', () => { assert.equal(balanced.tone, 'neutral'); const held = controlStatus(row({kind: 'ev', status: 'limited', reason: 'load_balancer_limit', sent_w: 11000, actual_w: 0})); assert.equal(held.text, 'Asked for 11.0 kW charge, but the charger’s load balancer gives the car no current.'); + const onePhase = controlStatus(row({kind: 'ev', mode: 'plan', status: 'limited', reason: 'one_phase', severity: 'info', + sent_w: 3680, actual_w: 3650, charging_phase: 'L1', charging_phase_max_w: 3680})); + assert.equal(onePhase.title, 'Charging on one phase'); + assert.equal(onePhase.text, 'The car charges on L1 only, at most 3.7 kW. FTW plans with what it takes.'); + assert.equal(onePhase.next, ''); + assert.equal(onePhase.tone, 'neutral'); + }); test('the receipt walks from sent to confirmed without inventing proof', () => { @@ -149,12 +156,51 @@ test('the summary answers the question for the whole site', () => { assert.equal(controlSummary([row()], false).tone, 'stale'); }); +test('calculated site headroom explains an EV shortfall without claiming a vendor cause', () => { + const ev = row({kind: 'ev', status: 'not_following', reason: 'power_below_target', severity: 'warning', sent_w: 10350, actual_w: 4155, + site_headroom: {evidence: 'calculated', fuse_a: 25, margin_a: 0.5, projected_max_a: 29.5, battery_charge_w: 9383}}); + const words = controlStatus(ev); + assert.equal(words.title, 'Insufficient site headroom'); + assert.match(words.text, /home battery is charging at 9.4 kW/); + assert.match(words.next, /other loads unchanged/); + assert.match(words.next, /25 A/); + assert.match(words.proof, /Calculated.*projection/); + assert.equal(controlStatus(ev, false).title, 'Status not current'); + assert.equal(controlStatus({...ev, reason: 'command_failed'}).title, 'Commands failing'); + assert.equal(controlStatus({...ev, readings_fresh: false}).title, 'Not following'); + assert.doesNotMatch(controlStatus({...ev, site_headroom: {...ev.site_headroom, battery_charge_w: 0}}).text, /battery/); + assert.equal(controlStatus({...ev, site_headroom: {...ev.site_headroom, projected_max_a: NaN}}).title, 'Not following'); +}); + test('limited positive EV offer does not explain prolonged zero delivery', () => { const missing=controlStatus(row({kind:'ev',status:'not_following',reason:'no_power_response',severity:'warning',requested_w:10350,sent_w:6900,actual_w:0,requested_a:15,offered_a:10,device_reason:'limited by dynamic charger current'})); assert.equal(missing.title,'Not following'); assert.match(missing.text,/6.9 kW charge.*no power/); const capped=controlStatus(row({kind:'ev',status:'limited',reason:'fuse_limit',sent_w:6900,actual_w:6800})); - assert.equal(capped.title,'Limited by site power limits'); - assert.match(capped.text,/fuse and import limits/); + assert.equal(capped.title,'Limited by the main fuse'); + assert.match(capped.text,/protect the main fuse/); + const hour=controlStatus(row({kind:'ev',status:'limited',reason:'import_limit',sent_w:4140,actual_w:4100})); + assert.equal(hour.title,'Limited by the hourly import limit'); + assert.match(hour.text,/this hour’s grid import/); + assert.doesNotMatch(hour.text,/fuse/); +}); + +test('a projected shortfall names the working limit rather than claiming the fuse rating is exceeded', () => { + const words = controlStatus(row({kind: 'ev', status: 'not_following', reason: 'power_below_target', severity: 'warning', + sent_w: 10350, actual_w: 8300, + site_headroom: {evidence: 'calculated', fuse_a: 25, margin_a: 1, projected_max_a: 24.8, battery_charge_w: 0}})); + assert.match(words.next, /24\.8 A/); + assert.match(words.next, /24 A working limit/); + assert.match(words.next, /25 A fuse/); + assert.doesNotMatch(words.next, /above the 25 A/); +}); + +test('a calculated projection does not deny raw charger reason evidence', () => { + const words = controlStatus(row({kind: 'ev', status: 'not_following', reason: 'power_below_target', severity: 'warning', + sent_w: 10350, actual_w: 4155, device_reason: 'limited by charger max current', + site_headroom: {evidence: 'calculated', fuse_a: 25, margin_a: .5, projected_max_a: 29.5, battery_charge_w: 9383}})); + assert.match(words.proof, /projection/); + assert.match(words.proof, /not a charger-reported cause/); + assert.doesNotMatch(words.proof, /has not reported|has not said/); }); diff --git a/web/energy-flow-readings.test.mjs b/web/energy-flow-readings.test.mjs index 4af56e4c3..2977a3f10 100644 --- a/web/energy-flow-readings.test.mjs +++ b/web/energy-flow-readings.test.mjs @@ -5,10 +5,11 @@ import { flowReadingsFromStatus, fmtKwhShort } from "./components/energy-flow-re it("shows current household power without a note about the past energy mean", () => { const estimate = {watts: 886, from_ms: Date.UTC(2026,9,18,10), to_ms: Date.UTC(2026,9,18,10,1), age_ms: 90000, end_age_ms:60000, energy_basis:'mixed', timing_basis: 'mixed'}; - const status = {load_w: 1800, load_estimate: estimate, energy:{today:{load_wh:2000,import_wh:500,history_quality:'estimated'}}}; + const status = {load_w: 1800, load_quality: "measured", load_estimate: estimate, energy:{today:{load_wh:2000,import_wh:500,history_quality:'estimated'}}}; const result = flowReadingsFromStatus(status); assert.equal(result.load, 1.8); - assert.equal(result.loadEstimated, true); + // An older Core also sent an energy mean; it never marks live Home use. + assert.equal(result.loadEstimated, false); assert.equal(status.load_w, 1800); assert.equal(result.selfPoweredPctToday, null); assert.equal("homeEstimateNote" in result, false); @@ -17,6 +18,14 @@ it("shows current household power without a note about the past energy mean", () assert.equal(flowReadingsFromStatus({...status,load_w:null}).load, null); }); +it("marks Home use estimated only when Core says so", () => { + for (const [quality, estimated] of [["measured", false], [undefined, false], ["estimated", true], ["unknown", false]]) { + const result = flowReadingsFromStatus({load_w: 640, load_quality: quality, history_quality: "complete"}); + assert.equal(result.load, 0.64); + assert.equal(result.loadEstimated, estimated, String(quality)); + } +}); + it("never lets a past zero mean overwrite a positive current Load", () => { const status={load_w:1234,history_quality:"complete",load_estimate:{watts:0,from_ms:1000,to_ms:11000,end_age_ms:60000}}; const result=flowReadingsFromStatus(status); @@ -27,7 +36,7 @@ it("never lets a past zero mean overwrite a positive current Load", () => { it("keeps a qualified power balance visible while the energy interval is unavailable", () => { for (const reason of ["incomplete_source_window", "charger_source_time_unavailable", "stale_source_window"]) { - const status = {load_w: 1234, history_quality: "complete", load_estimate:{watts:null, reason}}; + const status = {load_w: 1234, history_quality: "complete", load_quality: "estimated", load_estimate:{watts:null, reason}}; const result = flowReadingsFromStatus(status); assert.equal(result.load, 1.234); assert.equal(result.loadEstimated, true); @@ -38,7 +47,9 @@ it("keeps a qualified power balance visible while the energy interval is unavail it("distinguishes a real zero balance from missing, invalid or stale inputs", () => { const base = {history_quality:"complete",load_estimate:{watts:null,reason:"incomplete_source_window"}}; assert.equal(flowReadingsFromStatus({...base,load_w:0}).load,0); - for (const value of [undefined, null, "0", NaN, Infinity, -1]) { + // A negative balance is shown: it points at production Core cannot see. + assert.equal(flowReadingsFromStatus({...base,load_w:-300}).load,-0.3); + for (const value of [undefined, null, "0", NaN, Infinity]) { const result = flowReadingsFromStatus({...base,load_w:value}); assert.equal(result.load,null); } @@ -226,3 +237,11 @@ it('marks only devices that need attention and keeps an offline sibling visible assert.equal(failed.clickable,true); assert.equal(flowReadingsFromStatus(status,{live:false}).planets.find(p=>p.name==='b').controlMark,undefined); }); + +it("draws Core's estimated Home use as an estimate, not a blank", () => { + const status = {load_w: 980, load_quality: "estimated", load_reason: "offline:growatt:pv", history_quality: "incomplete"}; + const result = flowReadingsFromStatus(status); + assert.equal(result.load, 0.98); + assert.equal(result.loadEstimated, true); + assert.equal(flowReadingsFromStatus({load_w: 980, load_quality: "measured"}).loadEstimated, false); +}); diff --git a/web/ev-commanded-reason.test.mjs b/web/ev-commanded-reason.test.mjs index 745ec3bc5..dd2a094e0 100644 --- a/web/ev-commanded-reason.test.mjs +++ b/web/ev-commanded-reason.test.mjs @@ -9,7 +9,9 @@ const source = readFileSync(new URL('./app.js', import.meta.url), 'utf8'); test('every pause reason has its own sentence', () => { assert.match(source, /commanded_reason === "fuse_cooldown" \|\| lp\.commanded_reason === "fuse_limit"/); - assert.match(source, /Paused by site power limits/); + assert.match(source, /Paused by the main fuse/); + assert.match(source, /commanded_reason === "import_limit"/); + assert.match(source, /Paused by the hourly import limit/); assert.match(source, /commanded_reason === "site_meter_stale"/); assert.match(source, /Paused for safety: site-meter data is stale/); assert.match(source, /commanded_reason === "pv_surplus_pause"/); @@ -19,6 +21,7 @@ test('every pause reason has its own sentence', () => { assert.match(source, /commanded_reason === "no_plan_budget"/); assert.match(source, /The ready time has passed, so FTW is not charging/); assert.match(source, /Battery level is assumed, not read from the car/); - // A fuse-limited ongoing charge says so too. - assert.match(source, /Rate is limited by site power limits right now/); + // A limited ongoing charge names its limit too. + assert.match(source, /Rate is limited by the main fuse right now/); + assert.match(source, /Rate is limited by the hourly import limit right now/); }); diff --git a/web/fleet-ping-tab.test.mjs b/web/fleet-ping-tab.test.mjs index 1d3749784..84b458077 100644 --- a/web/fleet-ping-tab.test.mjs +++ b/web/fleet-ping-tab.test.mjs @@ -45,69 +45,71 @@ function render(config) { } const payload = { - schema: "ftw.fleet/1", + schema: "ftw.fleet/2", ftw_version: "v1.4.0-beta.2", channel: "beta", drivers: ["easee_cloud", "sungrow"], battery_kwh: "5-15", price_zone: "SE3", install_age: "6-12m", + plan_solve: "1-5s", + plan_fallbacks: "0", + driver_errors: "0-1%", + update: "failed", }; describe("the fleet statistics panel", () => { - it("offers a checkbox bound to fleet_ping.enabled", () => { - const html = render({}); - assert.match(html, /data-checkbox-path="fleet_ping\.enabled"/); - assert.match(html, /type="checkbox"/); - }); - - it("reflects the saved setting", () => { - assert.doesNotMatch(render({ fleet_ping: { enabled: false } }), /checkbox"[^>]*checked/); - assert.match(render({ fleet_ping: { enabled: true } }), /checked/); - }); - - it("mirrors the box's default when the section is missing", () => { - // A box that has never had fleet_ping in its YAML is sending, because - // applyDefaults turns it on. An unchecked box here would say the opposite - // of what is true, which is the one thing this screen must never do. + it("offers one statistics switch, not a config setting", () => { + // Statistics are one answer for both reports, kept by the box, not a + // field in the config document that Save writes. const config = {}; const html = render(config); - // Compared field by field: the object is built inside the vm sandbox, so - // its prototype comes from another realm and deepStrictEqual refuses it. - assert.ok(config.fleet_ping, "fleet_ping was not created"); - assert.equal(config.fleet_ping.enabled, true); - assert.match(html, /checked/); + assert.match(html, /id="statistics-share"/); + assert.doesNotMatch(html, /data-checkbox-path/); + assert.equal(config.fleet_ping, undefined, "render() wrote a config section"); }); - it("says the switch needs no restart", () => { - // The one thing about this setting that differs from the FTW app tab - // next to it, where saving is not enough. - assert.match(render({}), /[Nn]o restart/); + it("says what goes beside each message through the relay", () => { + // The sealed request carries a random report id and the send day. A + // screen that says "no id, no timestamp" and leaves those out would be + // the claim this tab exists not to make. + const html = render({}); + assert.match(html, /random number, drawn new for each report so FTW Cloud counts it once/); + assert.match(html, /UTC day it was sent/); + assert.match(html, /with what goes beside it/); + }); + + it("names both reports and says they never travel together", () => { + const html = render({}); + assert.match(html, /two anonymous reports a day/); + assert.match(html, /never travel together/); + assert.match(html, /part of Cloud/); }); it("names every field the message carries", () => { // This screen is the whole argument for a feature that is on by default, // and the sentence people actually read is the paragraph, not the table - // under it. A paragraph that names four of six fields is the screen + // under it. A paragraph that names some of the fields is the screen // making its case with the awkward ones left out — the price zone above // all, since it is the only field that says roughly where the house is. const html = render({}); - for (const field of [/version/i, /channel/i, /device|driver/i, /battery/i, /price zone/i, /how old/i]) { + for (const field of [/version/i, /channel/i, /device|driver/i, /battery/i, /price zone/i, /how old/i, + /slowest plan/i, /fell back/i, /failed to read/i, /update/i]) { assert.match(html, field, `the paragraph does not name ${field}`); } }); it("counts the fields it names, and the count is the fixture's", () => { - // "six things" is a claim like any other on this screen, and the only one + // "ten things" is a claim like any other on this screen, and the only one // that can go stale in silence. // // What this holds is the sentence against the fixture above, which is a // copy of the payload written here by hand. It is not a guard on the box: - // a seventh field would leave this passing. That guard is + // an eleventh field would leave this passing. That guard is // TestPayloadCarriesOnlyTheAgreedFields in go/internal/fleetping, which - // counts the marshalled fields and fails on the seventh — and whoever + // counts the marshalled fields and fails on the eleventh — and whoever // answers it has to come back through here. - const words = ["no", "one", "two", "three", "four", "five", "six", "seven", "eight"]; + const words = ["no", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "eleven"]; const fields = Object.keys(payload).length - 1; // schema names the format, not the box assert.ok(words[fields], `no number word for ${fields} fields`); assert.match(render({}), new RegExp(words[fields] + " things")); @@ -200,6 +202,42 @@ describe("the line under the payload", () => { }); }); +describe("the statistics switch", () => { + const { shareLine } = tab._pure; + + it("is part of FTW Cloud while Cloud is on", () => { + const line = shareLine({ statistics: true, statisticsRequired: true }); + assert.match(line, /Part of FTW Cloud/); + assert.match(line, /no separate off switch/); + }); + + it("says nothing goes under earlier Cloud terms", () => { + const line = shareLine({ statistics: false, statisticsRequired: false, statisticsNeedTerms: true }); + assert.match(line, /earlier terms/); + assert.match(line, /Nothing is sent until you accept the current terms/); + }); + + it("says what on and off mean without Cloud", () => { + assert.match(shareLine({ statistics: true, statisticsRequired: false }), /takes effect at once, with no restart/); + assert.match(shareLine({ statistics: false, statisticsRequired: false }), /^Off\. Nothing is sent/); + assert.match(shareLine(null), /Could not read/); + }); + + it("does not claim a report goes while it waits for the relay", () => { + const line = whereLine({ enabled: false, waits_for_relay: true, endpoint: "https://cloud.ftw.energy/v1/fleet", ftw_relay: true, payload }); + assert.match(line, /^Not sent yet/); + assert.match(line, /only through an independent relay/); + assert.doesNotMatch(line, /^Sent/); + }); + + it("says who sees what when the report goes through the relay", () => { + const line = whereLine({ enabled: true, endpoint: "https://relay.example.net/ftw", ftw_relay: true, through_relay: true, payload }); + assert.match(line, /^Sent through https:\/\/relay\.example\.net\/ftw, an independent relay, to FTW Cloud/); + assert.match(line, /sees this box's address but not the report/); + assert.match(line, /reads the report but not the address/); + }); +}); + describe("keeping the line true after a save", () => { it("registers a post-save hook, because render() runs only on a tab switch", () => { // Without this the line keeps whatever the box said when the tab opened: @@ -222,7 +260,8 @@ describe("keeping the line true after a save", () => { vm.createContext(sandbox); vm.runInContext(source, sandbox); sandbox.window.FTWSettings.fleetPing.afterSave(); - assert.deepEqual(asked, ["/api/fleet-ping"]); + // And the statistics answer, which the switch shows. + assert.deepEqual([...asked].sort(), ["/api/cloud", "/api/cloud/stats-policy", "/api/fleet-ping"]); }); }); @@ -237,8 +276,13 @@ describe("payloadRows", () => { assert.equal(byKey.battery_kwh.value, "5 to 15 kWh"); assert.equal(byKey.price_zone.value, "SE3"); assert.equal(byKey.install_age.value, "six to twelve months"); + assert.equal(byKey.plan_solve.value, "one to five seconds"); + assert.equal(byKey.plan_fallbacks.value, "none"); + assert.equal(byKey.driver_errors.value, "under 1 in 100"); + assert.equal(byKey.update.value, "failed"); assert.equal(byKey.battery_kwh.label, "Battery size"); + assert.equal(byKey.plan_solve.label, "Slowest plan, last day"); }); it("drops only the schema name", () => { @@ -271,3 +315,50 @@ describe("payloadRows", () => { assert.equal(payloadRows("not an object").length, 0); }); }); + +describe("the daily energy report", () => { + const { energyRows, energyLine } = tab._pure; + const energy = { + schema: "ftw.energy/1", date: "2026-10-23", pv_generation_kwh: 12, grid_import_kwh: 30, grid_export_kwh: 4, + battery_charge_kwh: 9, battery_discharge_kwh: 8, vehicle_charge_kwh: 0, + }; + + it("shows every field the report carries, in kWh", () => { + const rows = energyRows(energy); + const byKey = Object.fromEntries(rows.map((r) => [r.key, r])); + assert.equal(rows.length, Object.keys(energy).length - 1); + assert.equal(byKey.pv_generation_kwh.value, "12 kWh"); + assert.equal(byKey.vehicle_charge_kwh.label, "Vehicle charge"); + assert.equal(byKey.date.value, "2026-10-23"); + const extra = energyRows({ ...energy, home: "x" }).find((r) => r.key === "home"); + assert.ok(extra, "an unexpected field was silently dropped"); + }); + + it("says nothing is sent from a release without a relay", () => { + const line = energyLine({ relay: "", sending: false, day: "2026-10-23", payload: energy }); + assert.match(line, /no relay yet/i); + assert.doesNotMatch(line, /^Sent/); + }); + + it("does not point at a report it is not showing", () => { + assert.doesNotMatch(energyLine({ relay: "", day: "2026-10-23", payload: null }), /this is the report/i); + assert.doesNotMatch(energyLine({ relay: "https://relay.example.net/ftw", day: "2026-10-23", payload: null }), /this is the report/i); + }); + + it("says who sees what while it is sent, and nothing is sent while it is off", () => { + const relay = "https://relay.example.net/ftw"; + const on = energyLine({ relay, sending: true, payload: energy }); + assert.match(on, /^Sent through https:\/\/relay\.example\.net/); + assert.match(on, /sees this box's address but not the report/); + assert.match(on, /reads the report but not the address/); + assert.match(energyLine({ relay, sending: false, payload: energy }), /^Nothing is sent/); + }); + + it("names the fields, the missing id and the relay in its paragraph", () => { + const html = render({}); + assert.match(html, /id="fleet-energy-payload"/); + for (const want of [/solar generation/i, /grid import and export/i, /battery charge/i, /vehicle charge/i, /whole kWh/i, /no box id/i, /independent relay/i, /skipped/i]) { + assert.match(html, want, `the paragraph does not say ${want}`); + } + }); +}); diff --git a/web/home-assistant.js b/web/home-assistant.js new file mode 100644 index 000000000..02ad5b749 --- /dev/null +++ b/web/home-assistant.js @@ -0,0 +1,42 @@ +// Loaded only by Core's trusted Home Assistant ingress response. +(function () { + 'use strict'; + window.FTWHomeAssistant = true; + var script = document.querySelector('script[src$="/home-assistant.js"]'); + if (!script) return; + var base = new URL('.', script.src).pathname; + var password = document.getElementById('assistant-password'); + if (!password) return; + ['assistant-password', 'assistant-password-confirm'].forEach(function (id) { + document.getElementById(id).parentElement.hidden = true; + }); + var intro = password.parentElement.previousElementSibling; + if (intro) intro.textContent = 'Home Assistant manages access to FTW. Start this new box, then approve your assistant here. You choose whether to enable FTW Cloud.'; + window.startAssistantSetup = async function () { + var button = document.getElementById('assistant-start'); + var note = document.getElementById('assistant-status'); + button.disabled = true; + note.textContent = 'Starting FTW…'; + try { + var response = await fetch(base + 'api/setup/assistant', { + method: 'POST', headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({confirmed: true}) + }); + if (!response.ok) { + var problem = await response.json(); + note.textContent = problem.error || 'Could not start setup.'; + button.disabled = false; + return; + } + } catch (_) { /* The process may restart before its reply arrives. */ } + for (var attempt = 0; attempt < 30; attempt++) { + await new Promise(function (resolve) { setTimeout(resolve, 1000); }); + try { + var status = await fetch(base + 'api/auth/status', {cache: 'no-store'}); + if (status.ok) { window.location.href = base + '?setup=assistant'; return; } + } catch (_) { /* Wait for Core's restart. */ } + } + note.textContent = 'FTW has not answered yet. Reopen it from Home Assistant to check setup before trying again.'; + document.getElementById('assistant-resume').hidden = false; + }; +}()); diff --git a/web/home-assistant.test.mjs b/web/home-assistant.test.mjs new file mode 100644 index 000000000..c8d89ef3e --- /dev/null +++ b/web/home-assistant.test.mjs @@ -0,0 +1,23 @@ +import {readFileSync} from 'node:fs'; +import vm from 'node:vm'; +import {test} from 'node:test'; +import assert from 'node:assert/strict'; + +test('HA setup hides the extra password and starts without creating one',async () => { + const prefix='/api/hassio_ingress/example/'; + const fields = Object.fromEntries(['assistant-password','assistant-password-confirm','assistant-start','assistant-status','assistant-resume'].map(id=>[id,{parentElement:{hidden:false,previousElementSibling:{textContent:''}}}])); + const calls=[]; + const window={location:{}}; + const context=vm.createContext({window,URL,Promise,document:{ + querySelector:()=>({src:'https://ha.example'+prefix+'home-assistant.js'}), + getElementById:id=>fields[id] + },fetch:async (url,options)=>{calls.push([url,options]);return {ok:true};},setTimeout:fn=>fn()}); + vm.runInContext(readFileSync(new URL('./home-assistant.js',import.meta.url),'utf8'),context); + await window.startAssistantSetup(); + assert.equal(window.FTWHomeAssistant,true); + assert.equal(fields['assistant-password'].parentElement.hidden,true); + assert.equal(fields['assistant-password-confirm'].parentElement.hidden,true); + assert.deepEqual(JSON.parse(calls[0][1].body),{confirmed:true}); + assert.equal(calls[0][0],prefix+'api/setup/assistant'); + assert.equal(window.location.href,prefix+'?setup=assistant'); +}); diff --git a/web/loadpoint-battery-boost.test.mjs b/web/loadpoint-battery-boost.test.mjs index 67d4e3b36..822b29d5a 100644 --- a/web/loadpoint-battery-boost.test.mjs +++ b/web/loadpoint-battery-boost.test.mjs @@ -36,7 +36,7 @@ test('every controller stop reason has words', () => { 'cancelled', 'expired', 'vehicle_unplugged', 'ev_target_reached', 'departure_reached', 'surplus_only', 'site_safety_block', 'loadpoint_driver_unavailable', 'battery_unavailable', 'battery_reserve_reached', 'battery_hold', 'core_mode', - 'fuse_safety_block', 'restart_lease_invalid', + 'fuse_safety_block', 'import_limit_block', 'restart_lease_invalid', ]) { assert.match(app, new RegExp(reason + ': "')); } diff --git a/web/loadpoints.js b/web/loadpoints.js index dcd40c84a..031159e0d 100644 --- a/web/loadpoints.js +++ b/web/loadpoints.js @@ -109,6 +109,7 @@ battery_hold: 'Battery hold took priority', core_mode: 'Core mode does not allow boost', fuse_safety_block: 'Fuse protection stopped boost', + import_limit_block: 'Hourly import limit stopped boost', restart_lease_invalid: 'Saved lease was no longer safe', }; diff --git a/web/project-statistics.js b/web/project-statistics.js index dd9200359..f7abbbf9f 100644 --- a/web/project-statistics.js +++ b/web/project-statistics.js @@ -1,4 +1,6 @@ -// One owner choice: statistics come with Cloud; local use asks first. +// FTW Cloud's terms. Statistics are part of Cloud; a box without Cloud shares +// anonymous statistics until its owner turns them off under Anonymous +// statistics (settings/tabs/fleet.js), so nobody is asked here any more. (function (window, document) { "use strict"; var read = window.fetch.bind(window); @@ -35,11 +37,23 @@ function write(method,path,body) { return window.fetch(path,{method:method,headers:{"Content-Type":"application/json"},body:body?JSON.stringify(body):undefined}).then(function(r){return r.json().then(function(v){if(!r.ok)throw Error(v.error||"The box refused the choice.");justAnswered=true;return v;});}); } + // What FTW Cloud's statistics add, for a panel that offers or holds Cloud. + function cloudScope(parent) { + say(parent,"Shares recorded hourly energy and configured system capacities. Reports stay for 48 hours; public totals require at least 20 homes."); + var disclosure=node("details",null,parent); + node("summary","What is shared and how it is used",disclosure); + say(disclosure,details); + say(disclosure,privacy); + var link=node("a","Privacy and Cloud terms",parent);link.href="https://ftw.sourceful.energy/privacy#statistics";link.target="_blank";link.rel="noopener"; + if(!policy.available) say(parent,"You can save your choice now. Hourly reports wait for privacy review and activation."); + } function paint(host,settings) { host.textContent=""; - if (!status || !policy || (!settings && !status.statisticsDecisionRequired && !justAnswered)) {host.hidden=true;return;} + // Only a Cloud box under older terms has something to answer. + var needsTerms=!!status && status.link==="on" && !status.statisticsOnly && !status.cloudTermsAccepted; + if (!status || !policy || (!settings && !needsTerms && !justAnswered)) {host.hidden=true;return;} host.hidden=false; - if(!settings && !status.statisticsDecisionRequired && justAnswered) { + if(!settings && !needsTerms && justAnswered) { var saved=status.link==="ending"?"Stopping Cloud and deleting its stored copy. The box continues locally.":status.cloudTermsAccepted?"Cloud terms accepted and saved on this box.":status.statisticsActive?"Statistics choice saved. Change it in Settings → FTW app.":"Statistics sharing is off. The box continues locally."; var receipt=say(host,saved);receipt.className="project-statistics-saved";receipt.setAttribute("role","status"); return; @@ -48,24 +62,21 @@ panel.className="project-statistics-panel"; var assistant=host.id === "access-cloud-settings"; var cloud=status.link==="on" && !status.statisticsOnly; - node("legend",assistant?"FTW Cloud for assistants":cloud?(status.cloudTermsAccepted?"FTW Cloud":"Review FTW Cloud terms"):"Contribute to FTW statistics?",panel); + node("legend",assistant?"FTW Cloud for assistants":cloud?(status.cloudTermsAccepted?"FTW Cloud":"Review FTW Cloud terms"):"FTW Cloud",panel); if(assistant && !cloud) say(panel,"FTW Cloud lets your assistant reach this box. Sourceful can read its Cloud copy of your energy data. Cloud includes project statistics. Review the terms below before accepting."); else if(cloud) say(panel,status.cloudTermsAccepted?"The current Cloud terms are accepted. FTW Cloud includes project statistics.":"FTW Cloud includes project statistics. Sourceful can read its copy of your energy data. Accept the updated terms for the app and remote access, or turn Cloud off and keep using the box locally."); - else say(panel,"For local FTW use, sharing is your choice. Yes is the suggested choice. Nothing starts until you answer, and saying no leaves local operation working. Statistics-only sharing gives no app, phone or assistant remote access."); - say(panel,"Shares recorded hourly energy and configured system capacities. Reports stay for 48 hours; public totals require at least 20 homes."); - var disclosure=node("details",null,panel); - node("summary","What is shared and how it is used",disclosure); - say(disclosure,details); - say(disclosure,privacy); - var link=node("a","Privacy and Cloud terms",panel);link.href="https://ftw.sourceful.energy/privacy#statistics";link.target="_blank";link.rel="noopener"; - if(!policy.available) say(panel,"You can save your choice now. Hourly reports wait for privacy review and activation."); - if(!status.cloudTermsAccepted) { - say(panel,authStatus && authStatus.lan_auth + else say(panel,"FTW Cloud is off, and the box works locally. The anonymous statistics it shares, and the switch for them, are under Anonymous statistics below."); + if(cloud || assistant) cloudScope(panel); + var signIn=function(parent){ + say(parent,authStatus && authStatus.home_assistant + ?"Home Assistant manages sign-in for this choice." + :authStatus && authStatus.lan_auth ?"Your house password protects this choice. The box will ask you to log in if needed." :authStatus && authStatus.lan_auth===false ?"No house password is required. With password protection off, anyone who can reach this box on your private network can save this choice." :"This choice uses the same sign-in rules as Settings. A house password is optional."); - } + }; + if(!status.cloudTermsAccepted && (cloud || assistant)) signIn(panel); var actions=node("div",null,panel);actions.className="app-link-actions"; var feedback=node("p",null,panel);feedback.className="project-statistics-error";feedback.setAttribute("role","alert");feedback.hidden=true; if(status.link==="ending") {say(panel,"Cloud is stopping and deleting its stored copy. Refresh after the box receives the deletion receipt.");return;} @@ -78,22 +89,12 @@ },false,feedback); } else if(assistant) { button(actions,"Accept Cloud terms and turn it on",function(){return write("POST","/api/cloud/grant",{scopes:policy.cloud_scopes,policy:policy.cloud_policy});},true,feedback); - var local=node("details",null,panel);node("summary","Statistics without assistant remote access",local); - say(local,"Statistics-only sharing gives no app, phone or assistant remote access."); - if(status.statisticsOnly) button(local,"Stop sharing statistics",function(){return write("DELETE","/api/cloud/stats");},false,feedback); - else {button(local,"Yes, contribute",function(){return write("POST","/api/cloud/stats",{policy:policy.policy});},false,feedback);} - } else if(status.statisticsOnly) { - say(panel,"Local statistics sharing is on. Remote Cloud functions are off."); - button(actions,"Stop sharing statistics",function(){return write("DELETE","/api/cloud/stats");},false,feedback); } else { - button(actions,"Yes, contribute",function(){return write("POST","/api/cloud/stats",{policy:policy.policy});},true,feedback); - if(status.statisticsDecisionRequired) button(actions,"No, keep it local",function(){return write("DELETE","/api/cloud/stats");},false,feedback); - else say(panel,"Local statistics sharing is off."); - } - if(settings && !cloud && !assistant) { var more=node("details",null,panel); node("summary","Use FTW Cloud and the app",more); - say(more,"FTW Cloud adds app and assistant remote access and a readable state copy. Statistics are included for every Cloud user. Read the information above before accepting."); + say(more,"FTW Cloud adds app and assistant remote access and a readable state copy. Statistics are part of Cloud, with no separate off switch."); + cloudScope(more); + signIn(more); button(more,"Accept Cloud terms and turn it on",function(){return write("POST","/api/cloud/grant",{scopes:policy.cloud_scopes,policy:policy.cloud_policy});},true,feedback); } } diff --git a/web/project-statistics.test.mjs b/web/project-statistics.test.mjs index 6fa771ea0..15ba6223d 100644 --- a/web/project-statistics.test.mjs +++ b/web/project-statistics.test.mjs @@ -24,7 +24,7 @@ function page(cloud=false,authorised=true,options={}) { const window={addEventListener(){},confirm:()=>true,fetch:async(path,req)=>{ if(path==='/api/auth/status') { if(options.authError)throw Error('status unavailable'); - return {ok:true,json:async()=>({lan_auth:options.lanAuth===true})}; + return {ok:true,json:async()=>({lan_auth:options.lanAuth===true,home_assistant:options.homeAssistant===true})}; } if(req){ writes.push({path,...req}); @@ -39,21 +39,20 @@ function page(cloud=false,authorised=true,options={}) { return {window,writes,main,all,buttons:()=>all().filter(e=>e.tag==='button')}; } -test('local question sends nothing before a real Yes or No',async()=>{ +test('a box without Cloud is not asked, and Settings offers Cloud only',async()=>{ const p=page();await p.window.FTWStatistics.refresh(); assert.equal(p.writes.length,0); - assert.deepEqual(p.buttons().map(e=>e.textContent),['Yes, contribute','No, keep it local']); - assert.equal(p.buttons()[0].className,'btn-send'); - p.buttons()[1].events.click();await setImmediate(); - assert.equal(p.writes.length,1);assert.equal(p.writes[0].method,'DELETE'); - assert.equal(p.writes[0].path,'/api/cloud/stats'); - assert.match(p.main.textContent,/sharing is off/); -}); - -test('local Yes grants only statistics, not Cloud access',async()=>{ - const p=page();await p.window.FTWStatistics.refresh();p.buttons()[0].events.click();await setImmediate(); - assert.equal(p.writes.length,1);assert.equal(p.writes[0].path,'/api/cloud/stats'); - assert.deepEqual(JSON.parse(p.writes[0].body),{policy:'stats-current'}); + assert.equal(p.buttons().length,0); + const prompt=p.all().find(e=>e.id==='project-statistics-prompt'); + assert.equal(prompt.hidden,true); + const settings={tag:'div',children:[],events:{},style:{},attributes:{},ownText:'',appendChild(e){this.children.push(e);return e;},set textContent(v){this.children=[];},get textContent(){return this.children.map(e=>e.textContent).join(' ');}}; + p.window.FTWStatistics.mount(settings); + const labels=[];const visit=e=>{if(e.tag==='button')labels.push(e.textContent);(e.children||[]).forEach(visit);};visit(settings); + assert.deepEqual(labels,['Accept Cloud terms and turn it on']); + assert.match(settings.textContent,/FTW Cloud is off/); + assert.match(settings.textContent,/Anonymous statistics below/); + assert.doesNotMatch(settings.textContent,/Yes, contribute|keep it local/); + assert.equal(p.writes.length,0); }); test('Cloud has bundled terms and a whole-service off action',async()=>{ @@ -105,3 +104,10 @@ test('an unauthenticated page neither asks nor writes on behalf of an owner',asy const p=page(false,false);await p.window.FTWStatistics.refresh(); assert.equal(p.buttons().length,0);assert.equal(p.writes.length,0); }); + + +test('HA ingress explains the existing HA login instead of an open LAN',async()=>{ + const p=page(true,true,{homeAssistant:true});await p.window.FTWStatistics.refresh(); + assert.match(p.main.textContent,/Home Assistant manages sign-in/); + assert.doesNotMatch(p.main.textContent,/anyone who can reach/); +}); diff --git a/web/settings/tabs/access.js b/web/settings/tabs/access.js index fc5394c25..5f4b25e4f 100644 --- a/web/settings/tabs/access.js +++ b/web/settings/tabs/access.js @@ -254,7 +254,7 @@ if (state.cloudChecked && (!state.cloud || state.cloud.link !== "on" || state.cloud.statisticsOnly || !state.cloud.cloudTermsAccepted)) return {kind:"cloud"}; if (unavailable) return { kind: "unavailable" }; if (!list) return { kind: "loading" }; - if (!auth || !auth.lan_auth || !auth.configured) return { kind: "password" }; + if (!auth || (!auth.home_assistant && (!auth.lan_auth || !auth.configured))) return { kind: "password" }; var box = list.box || {}; // The box's answer for this very request: on the box itself, or through // an SSH tunnel to it, the house password does not count. @@ -1133,7 +1133,9 @@ '

Let an AI assistant, such as Claude or ChatGPT, read this home and, ' + "if you allow it, control it within limits. It connects through FTW Cloud. This box " + "decides what it may do and checks every call. No phone app is needed.

" + - '

For a new box: sign in with the house password you chose, review Cloud terms, then ask your assistant to begin FTW setup. Approve its setup connection below. If its connection page offers a choice, choose “Set up and configure”. The assistant can resume the draft stored on this box.

' + + '

' + (window.FTWHomeAssistant ? + 'Home Assistant has signed you in. Review Cloud terms, then ask your assistant to begin FTW setup and approve its connection below.' : + 'For a new box: sign in with the house password you chose, review Cloud terms, then ask your assistant to begin FTW setup. Approve its setup connection below. If its connection page offers a choice, choose “Set up and configure”. The assistant can resume the draft stored on this box.') + '

' + '
' + '
' + '
' + diff --git a/web/settings/tabs/access.test.mjs b/web/settings/tabs/access.test.mjs index 66474bd90..28a81ae7b 100644 --- a/web/settings/tabs/access.test.mjs +++ b/web/settings/tabs/access.test.mjs @@ -245,6 +245,7 @@ describe("what stops a grant before a code is spent", () => { it("is nothing once the house password is on and the box can take an offer", () => { assert.equal(P.readiness(ok, "", on), null); + assert.equal(P.readiness(ok, "", { home_assistant: true, lan_auth: false, configured: false }), null); }); it("names each block, in the order to fix them", () => { diff --git a/web/settings/tabs/app.js b/web/settings/tabs/app.js index d077f03ea..805f72b2f 100644 --- a/web/settings/tabs/app.js +++ b/web/settings/tabs/app.js @@ -506,8 +506,8 @@ S.tabs.app = { afterSave: function () { - // These two switches share a page, not a state. The app link has its - // own restart boundary; fleet sharing takes effect as soon as it saves. + // These share a page, not a state. The app link has its own restart + // boundary; the statistics switch writes at once. var panel = S.fleetPing; if (panel && panel.afterSave) panel.afterSave(); refreshStatus(pairingCtx); diff --git a/web/settings/tabs/fleet.js b/web/settings/tabs/fleet.js index 4ed970d92..de0b974e9 100644 --- a/web/settings/tabs/fleet.js +++ b/web/settings/tabs/fleet.js @@ -1,4 +1,5 @@ -// Settings → Fleet ping: the message this box adds to FTW's fleet totals. +// Settings → FTW app → Anonymous statistics: the one switch for both daily +// reports, and the messages this box adds to FTW's fleet and energy totals. // // This tab is not a privacy policy. A policy is a promise; this screen shows // the actual payload, built by the same call the sender uses, so somebody @@ -31,6 +32,18 @@ battery_kwh: "Battery size", price_zone: "Price zone", install_age: "Time since install", + plan_solve: "Slowest plan, last day", + plan_fallbacks: "Fallback plans, last day", + driver_errors: "Failed driver reads, last day", + update: "Newest update, last week", + }; + + // The health fields, each a few words wide. + var HEALTH = { + plan_solve: { none: "no plan", "0-1s": "under a second", "1-5s": "one to five seconds", "5s+": "five seconds or more" }, + plan_fallbacks: { none: "no plan", "0": "none", "1-3": "one to three", "4+": "four or more" }, + driver_errors: { none: "too few reads to say", "0": "none", "0-1%": "under 1 in 100", "1%+": "1 in 100 or more" }, + update: { none: "none", ok: "succeeded", failed: "failed" }, }; var BATTERY = { @@ -53,6 +66,7 @@ if (Array.isArray(value)) return value.length ? value.join(", ") : "none"; if (key === "battery_kwh" && BATTERY[value]) return BATTERY[value]; if (key === "install_age" && AGE[value]) return AGE[value]; + if (HEALTH[key] && HEALTH[key][value]) return HEALTH[key][value]; if (value === "unknown") return "not known"; return String(value); } @@ -79,16 +93,81 @@ // clock, and until then one box's arrival times are still loosely related. function whereLine(body) { var endpoint = (body && body.endpoint) || "nowhere"; - var handling = body && body.ftw_relay - ? " FTW Cloud adds the report to that day's totals, then drops it and does not save the source address. These are reports, not unique boxes." - : " This box uses a custom collector. FTW cannot say what that service stores or whether it counts unique boxes."; + var relayed = !!(body && body.through_relay); + var where = relayed ? "through " + endpoint + ", an independent relay, to FTW Cloud" : "to " + endpoint; + var handling = relayed + ? " The relay sees this box's address but not the report. FTW Cloud reads the report but not the address, adds it to that day's totals and keeps nothing of it. These are reports, not unique boxes." + : body && body.ftw_relay + ? " FTW Cloud adds the report to that day's totals, then drops it and does not save the source address. These are reports, not unique boxes." + : " This box uses a custom collector. FTW cannot say what that service stores or whether it counts unique boxes."; + if (body && body.waits_for_relay) { + return "Not sent yet. This box shares statistics without having said yes, so this report goes to FTW Cloud only through an independent relay, which hides this box's address, and this release names none yet."; + } if (body && body.enabled) { - return "Sent to " + endpoint + + return "Sent " + where + ", once a day, at a time that moves from day to day — it takes a few days to work its way round the clock." + handling; } - return "Nothing is sent. This is the message that would go to " + endpoint + - ", once a day, if you switched it on." + handling; + return "Nothing is sent. This is the message that would go " + where + + ", once a day, if statistics were on." + handling; + } + + // shareLine says what the one statistics switch means right now. With FTW + // Cloud it is part of Cloud; without, it is the owner's to turn off. + function shareLine(status) { + if (!status) return "Could not read this box's statistics answer."; + if (status.statisticsRequired) { + return "Part of FTW Cloud, with no separate off switch. To stop, turn FTW Cloud off, then turn statistics off here."; + } + if (status.statisticsNeedTerms) { + return "FTW Cloud is on under earlier terms, which did not include statistics. Nothing is sent until you accept the current terms."; + } + return status.statistics + ? "On. Turning it off takes effect at once, with no restart, and the box keeps the answer." + : "Off. Nothing is sent. Turning it on takes effect at once."; + } + + // Labels for the daily energy report. As above, a field with no label is + // shown under its own name. + var ENERGY_LABELS = { + date: "Day (UTC)", + pv_generation_kwh: "Solar generation", + grid_import_kwh: "Grid import", + grid_export_kwh: "Grid export", + battery_charge_kwh: "Battery charge", + battery_discharge_kwh: "Battery discharge", + vehicle_charge_kwh: "Vehicle charge", + }; + + // energyRows is payloadRows for the energy report: driven by its keys, so + // nothing it carries can be missing from this screen. + function energyRows(payload) { + if (!payload || typeof payload !== "object") return []; + return Object.keys(payload) + .filter(function (key) { return key !== "schema"; }) + .map(function (key) { + var value = payload[key]; + var text = /_kwh$/.test(key) && typeof value === "number" ? value + " kWh" : String(value); + return { key: key, label: ENERGY_LABELS[key] || key, value: text }; + }); + } + + // energyLine is whereLine for the energy report. It follows the box's own + // answer, and says so when this release has no relay to send through. + // + // Without a report to show, it does not point at one. + function energyLine(energy) { + var shown = !!(energy && energy.payload); + if (!energy || !energy.relay) { + return "This release has no relay yet, so no energy report is sent." + + (shown ? " This is the report it would send." : ""); + } + var how = " The relay sees this box's address but not the report. FTW Cloud reads the report but not the address, adds it to that day's totals and keeps nothing of it."; + if (energy.sending) { + return "Sent through " + energy.relay + ", an independent relay, once a day, 1 to 12 hours after the UTC day ends." + how; + } + return "Nothing is sent. " + (shown ? "This is the report that would go" : "A report would go") + " through " + + energy.relay + ", an independent relay, once a day, if sharing were on." + how; } function row(label, value) { @@ -132,6 +211,80 @@ exact.className = "fleet-ping-json"; exact.textContent = JSON.stringify(body.payload, null, 2); slot.appendChild(exact); + if (body.through_relay) sealedRows(slot); + } + + // sealedRows shows what goes beside a message sent through the relay, + // inside the sealed request: drawn when it is sent, so not shown as values. + function sealedRows(slot) { + slot.appendChild(row("Report id", "a random number, new for each report")); + slot.appendChild(row("Send day", "the UTC day it is sent")); + } + + function paintEnergy(energy) { + var slot = document.getElementById("fleet-energy-payload"); + if (!slot) return; + slot.textContent = ""; + var rows = energyRows(energy && energy.payload); + var note = document.createElement("p"); + note.className = "hint fleet-energy-note"; + if (!energy) { + note.textContent = "This box has no energy report."; + slot.appendChild(note); + return; + } + if (rows.length === 0) { + note.textContent = "The energy record for " + energy.day + " has a gap, so this box has no report for that day. A day with a gap is skipped, never sent in part."; + slot.appendChild(note); + } + rows.forEach(function (r) { slot.appendChild(row(r.label, r.value)); }); + + var where = document.createElement("p"); + where.className = "hint fleet-energy-note"; + where.textContent = energyLine(energy); + slot.appendChild(where); + + if (rows.length === 0) return; + var exact = document.createElement("pre"); + exact.className = "fleet-ping-json"; + exact.textContent = JSON.stringify(energy.payload, null, 2); + slot.appendChild(exact); + sealedRows(slot); + } + + var sharePolicy = null; + + // refreshShare reads the box's one statistics answer into the switch. + function refreshShare() { + var box = document.getElementById("statistics-share"); + var note = document.getElementById("statistics-share-note"); + if (!box || !note) return; + Promise.all([fetch("/api/cloud"), fetch("/api/cloud/stats-policy")]) + .then(function (rs) { return Promise.all(rs.map(function (r) { return r.ok ? r.json() : null; })); }) + .then(function (v) { + sharePolicy = v[1]; + box.checked = !!(v[0] && v[0].statistics); + box.disabled = !v[0] || !v[1] || !!v[0].statisticsRequired || !!v[0].statisticsNeedTerms; + note.textContent = shareLine(v[0]); + }) + .catch(function () { note.textContent = shareLine(null); }); + } + + // The switch writes at once, as the Cloud terms do, and not on Save: it is + // the box's answer, not part of the config document. + function changeShare(box) { + var note = document.getElementById("statistics-share-note"); + box.disabled = true; + var write = box.checked + ? fetch("/api/cloud/stats", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ policy: sharePolicy && sharePolicy.policy }) }) + : fetch("/api/cloud/stats", { method: "DELETE" }); + write + .then(function (r) { return r.json().then(function (v) { if (!r.ok) throw Error(v.error || "The box refused the change."); }); }) + .then(function () { refreshShare(); refresh(); }) + .catch(function (e) { + refreshShare(); + if (note) note.textContent = e.message || "The change could not be saved. Try again."; + }); } function refresh() { @@ -149,6 +302,7 @@ return; } paint(body); + paintEnergy(body.energy); }) .catch(function () { slot.textContent = ""; @@ -159,46 +313,65 @@ }); } - // The FTW app and fleet statistics share a settings page, but not a - // setting: a box can share its anonymous daily report without using the - // app relay. Keep this as a reusable panel instead of a tab so the payload - // and privacy wording stay in one place. + // The FTW app and statistics share a settings page, but not a setting: a + // box shares its anonymous reports whether or not it uses the app. Keep + // this as a reusable panel instead of a tab so the payloads and privacy + // wording stay in one place. S.fleetPing = { - _pure: { payloadRows: payloadRows, describe: describe, whereLine: whereLine }, - - // The line under the payload is the box's answer, not the checkbox's, so - // it can only be right again once the box has the new config. Saving is - // the moment that happens. - afterSave: refresh, + _pure: { payloadRows: payloadRows, describe: describe, whereLine: whereLine, shareLine: shareLine, energyRows: energyRows, energyLine: energyLine }, - render: function (ctx) { - // Mirrors the box's own default (see applyDefaults in - // go/internal/config): a box that has never had the section is sending. - // Rendering it unchecked here would say the opposite of what is true. - if (!ctx.config.fleet_ping) ctx.config.fleet_ping = { enabled: true }; - var enabled = !!ctx.config.fleet_ping.enabled; + // Saving the config does not change the answer, but the payload under it + // is built from the config, so read both again. + afterSave: function () { refresh(); refreshShare(); }, - setTimeout(refresh, 0); + render: function () { + setTimeout(function () { + var box = document.getElementById("statistics-share"); + if (box) box.addEventListener("change", function () { changeShare(box); }); + refreshShare(); + refresh(); + }, 0); return ( - "
Anonymous daily report" + - '

The project statistics choice above controls whether this box contributes. This additional daily report describes versions and integrations. It does not count unique boxes.

' + - '

When sharing is on, this box sends six things once a day: its ' + + "

Anonymous statistics" + + '

This box sends two anonymous reports a day, each on its own and at its own time: ' + + "a fleet report and an energy report. With FTW Cloud they are part of Cloud. Without Cloud they " + + "are on until you turn them off here.

" + + '' + + '

Checking…

' + + '

Daily fleet report

' + + '

The fleet report carries ten things once a day: its ' + "FTW version, its release channel, the kinds of device it talks to, " + - "roughly how much battery, its price zone, and roughly how old the " + - "install is. Nothing else — no id, no key, no serial, no site name, no " + + "roughly how much battery, its price zone, roughly how old the " + + "install is, and four rough marks of how it ran: how slow its slowest " + + "plan of the last day was, how many plans fell back, how often its " + + "drivers failed to read, and whether its newest update of the last week " + + "failed. Nothing else — no id, no key, no serial, no site name, no " + "counter, no timestamp — so nothing in the message says which box sent " + - "it. The message itself cannot prove how many unique boxes sent reports.

" + - '

Two limits, plainly. Those six things still describe ' + + "it. The message itself cannot prove how many unique boxes sent reports. " + + "Sent through the relay, two things go with it, sealed: a random number, " + + "drawn new for each report so FTW Cloud counts it once, and the UTC day it " + + "was sent. Neither says which box sent it.

" + + '

Two limits, plainly. Those ten things still describe ' + "you: a beta box in a small price zone with a big battery may be the " + - "only one of its kind. And the receiving service sees the address during " + + "only one of its kind. And without a relay the receiving service sees the address during " + "the request, the way any website does. What it keeps depends on the " + - "endpoint shown below. Here is the whole message, as this box would send it right now.

" + - '" + - '

Takes effect as soon as you save. No restart.

' + + "endpoint shown below. Here is the whole message, as this box would send it right now, " + + "with what goes beside it.

" + '
' + + '

Daily energy report

' + + '

The other report: the date, and the ' + + "day's solar generation, grid import and export, battery charge and " + + "discharge, and vehicle charge, each in whole kWh. It carries no box id and " + + "nothing from the fleet report, and leaves at its own time, so the two " + + "cannot be matched. They never travel together: device kinds and a price " + + "zone can point at one home, and the energy beside them would then be " + + "that home's. It goes through an independent relay, so FTW Cloud reads " + + "the numbers without this box's address, and the relay sees the address " + + "without the numbers. Sealed with it go a random number, drawn new for " + + "each report so FTW Cloud counts it once, and the UTC day it was sent. " + + "A day with a gap in the energy record is skipped.

" + + '
' + "
" ); }, diff --git a/web/settings/tabs/system.js b/web/settings/tabs/system.js index aab19d48d..98cdc9ef2 100644 --- a/web/settings/tabs/system.js +++ b/web/settings/tabs/system.js @@ -104,6 +104,8 @@ '' + '

The card compares with no solar and no battery. Tick this to compare with a battery that only stores surplus solar and covers the house. It changes at once.

' + '
' + + (window.FTWHomeAssistant ? + '
Home Assistant access

Home Assistant manages access to FTW. No separate FTW password is needed.

' : '
' + 'LAN password' + '

…

' + @@ -117,7 +119,7 @@ '' + '

' + '

Asks for this password on the LAN before settings and writes. Live status stays visible. curl still uses Bearer.

' + - '
' + + '') + '
' + 'Host' + '
' + diff --git a/web/style.css b/web/style.css index 4c76dec73..a8505891f 100644 --- a/web/style.css +++ b/web/style.css @@ -1836,6 +1836,15 @@ footer { color: var(--text-dim); } +.fleet-energy-title { + margin: 18px 0 4px; + font-size: 0.9rem; +} +/* .hint carries no margin; without this the report's notes read as one slab. */ +.fleet-energy-note { + margin-top: 8px; +} + /* ---- Access settings tab ---- */ /* Who outside the house may reach it. The page reads top to bottom in the order an owner acts: what is missing, the code, the offer, what waits for