Skip to content

fix(easee_cloud): reject out-of-order power observations - #183

Open
segran2 wants to merge 2 commits into
srcfl:mainfrom
segran2:fix/easee-observation-order
Open

segran2 wants to merge 2 commits into
srcfl:mainfrom
segran2:fix/easee-observation-order

Conversation

@segran2

@segran2 segran2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem

An older Easee Cloud TotalPower observation can replace a newer one. The regression harness reproduces a 10.242 kW reading followed by an older 0 W reading. This demonstrates the driver behavior. The cause of the individual incidents in srcfl/ftw#1674 still needs field evidence.

Fix

  • Port the observation-order fix to Easee Cloud 1.4.1 on main, where SignalR supplies live updates and REST supplies fallback snapshots and periodic confirmation. The ordering fix is independent of the load-balancer reporting in fix(easee_cloud): report a load-balancer limit while the car charges #172.
  • Use one RFC3339 parser for observation times. Reject older power, conflicting values at the same instant, and connected power readings without a valid timestamp. Confirm unchanged readings and accept a newer measured zero.
  • Reset power ordering on mode 1. Recover from a future timestamp or backwards clock jump when readings on the lower timeline advance after 30 seconds. Repetition of a cached sample cannot trigger recovery.
  • Bump Easee Cloud to 1.4.2 and regenerate the catalog metadata. The charging commands and 7 A start boost keep their existing policy.

Validation

  • make test-driver ID=easee_cloud passed with 43 tests and 15 skips. Coverage includes REST and SignalR ordering, equivalent timestamps, measured zero, clock recovery, mode-1 reset, and the existing start-boost regressions.
  • GitHub Actions for this candidate

Hardware validation

Tested exact driver commit c9b4a8812a412b0cd50ab06456dbcc602eaaf197 on 12 November 2026 (CET), using FTW Core v0.163.0-beta.1 with local diagnostics, an Easee charger and an Audi e-tron.

  • SignalR: Recorded positive charging power, a real stop at 0 W, mode 1 on unplug, and a new session after reconnecting.
  • 6 A start with boost allowed: A three-phase 6 A request started with a 7 A boost. The driver stepped down to 6 A after about 3.7 seconds when charging was observed. Measured power reached 4.13 kW.
  • REST fallback: With SignalR unavailable to the FTW process, the same candidate recorded 4.12 kW at a 6 A setting after the boost, then 0 W after stopping.
  • 6 A start with boost disallowed: With FTW's three-phase loadpoint capped at 4.14 kW, the driver logged a 6 A offer and no 7 A boost. In the repeat with Easee load balancing also capped at 6 A per phase, measured power peaked at 4.13 kW and the dynamic setting settled at 6 A.

Follow-up

In the first capped test, Easee reported a 32 A dynamic setting and measured power reached 6.14 kW despite the 6 A offer and reassertion. In the repeat, Easee again briefly reported 32 A before reading back 6 A, with measured power within the 4.14 kW cap. The 32 A value is a reported setting. The cause of the discrepancy is unresolved and belongs to the phase/current trace investigation in #225.

The boost timeout and synthetic clock jumps were exercised in regression tests. The hardware runs exercised the boost's step-down on measured charging. The original driver and FTW configuration were restored after testing.

After merge, advance FTW's bundled-driver pin so the fix reaches offline gateways.

Vendor references: Get observations, observation IDs, SignalR stream.

@miravoss26 miravoss26 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Easee cloud driver 1.3.9: rejects out-of-order or conflicting power observations (compares RFC3339 source instants normalised to UTC), and adds a one-shot 7 A "start boost" for 6 A orders to work around the ~5 min Easee start delay.

  • Correctness, fine: power_time_key handles offsets, fractions and leap days, and the zero-padded key compares correctly as a string. Disconnect resets the ordering. Boost is bounded (90 s, once per start, gated on cap_amps_per_phase, max_a and the device limit all allowing 7 A) and start_boost: false turns it off. Tests cover both paths and CI is green.
  • Low, worth a thought: a charger that reboots while a car stays connected and restarts its clock behind the last accepted timestamp will have every poll rejected as "out-of-order" until it passes the old value. The disconnect reset covers the normal case, but there's no timeout or escape hatch for this one. Maybe a reset after N consecutive rejections.
  • Low: this PR carries two unrelated behaviours (observation ordering and start boost) under one version bump. Fine to ship together, just noting that a revert would take both.
  • Note: the boost deliberately offers 1 A above what Core ordered. The cap_amps_per_phase guard is what keeps that safe, and a Core that doesn't send the cap gets no boost. Good default.
  • Security screen: no secrets, no new hosts or dependencies, no authz changes.

Safe to merge from my read. The field-data claim (4 runs, 22 Oct) is the author's, so I couldn't verify it. The PR itself says the ordering fix isn't hardware-verified yet.

@frahlg

frahlg commented Nov 12, 2026

Copy link
Copy Markdown
Member

Review (Claude CLI, 2026-11-12): do not merge this PR as it stands.

Merging would squash into easee-load-balancer-reason (#172), which is still a draft and conflicts with main. The fix would not land on main and would not trigger ftw-drivers-release.

Blockers

  • Current main already has Easee 1.4.0 (SignalR) and 1.4.1. Observation validation now lives in note_observation / state_problem, and main already drops a REST sample older than a stream sample but otherwise applies REST so one bad stamp cannot hold a value. This PR's latching rejection is a different rule and must be re-implemented on that model (REST and stream), with a single time parser. easee_cloud.lua and the catalog files conflict with main.
  • The candidate version is still 1.3.9. A rebase needs ≥ 1.4.2 and regenerated hashes.

The ordering logic on this branch is otherwise sound (RFC3339 comparison, same-instant conflicts, measured-zero rules, disconnect reset). During the rebase, please also add an escape hatch for a bogus future timestamp or a charger clock that jumps backwards — today that can reject every later poll until disconnect — plus tests for that path and for a mode-1 reset.

CI being green on a7b6a3d only covers the stale #172 base, not current main.

@segran2
segran2 force-pushed the fix/easee-observation-order branch from a7b6a3d to a83417e Compare November 12, 2026 12:32
@segran2
segran2 changed the base branch from easee-load-balancer-reason to main November 12, 2026 12:32
@segran2
segran2 marked this pull request as draft November 12, 2026 12:32
@segran2
segran2 force-pushed the fix/easee-observation-order branch from a83417e to c9b4a88 Compare November 12, 2026 12:33
@segran2
segran2 marked this pull request as ready for review November 12, 2026 15:44

@miravoss26 miravoss26 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: easee_cloud 1.4.2. Orders TotalPower by parsed RFC3339 source instant across REST and SignalR, so an older or conflicting reading can't replace a newer one. Mode 1 resets the watermark, and a backwards clock recovers after 30 s of advancing readings. Version, index, manifest and sha bumps are consistent with the driver change.

Findings:

  • nit, drivers/lua/easee_cloud.lua:~108-140 (power_problem): recovery needs key > recovery_key after 30 s. A source whose clock stays on the lower timeline but only repeats a coarse stamp (same second, changing values) stays rejected indefinitely and the driver reports blocked. Probably acceptable (it logs a warn and retries every 10 s), but worth knowing for chargers with 1 s stamp resolution and fast-changing power.
  • nit, easee_cloud.lua:~1058 (driver_poll): apply(OBS_TOTAL_POWER) rejection now skips the remaining observations for that poll (session id, energy, etc.). Intended as far as I can tell (the whole read is treated as a problem), just noting that one bad power stamp holds the rest of the snapshot too.
  • info, test_easee_cloud_stream.lua: existing tests were edited to add stamps and to flip two assertions ("a pushed sample applies even when older" now means the opposite). That matches the new intent and is called out in the diff, so fine.
  • Security screen: no secrets, no new hosts, no new dependencies, no authz changes. Pure Lua + metadata.

The hardware validation in the description (REST fallback, SignalR, 6 A boost) covers the realistic paths well. The 32 A reading discrepancy is correctly parked for #225.

Safe to merge from my read. Reminder from the description: bump FTW's bundled-driver pin after merge.

@segran2

segran2 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up to frahlg’s November 12 review. The PR now targets main directly and includes the current base as of b52a558. GitHub reports no merge conflicts.

Easee Cloud is now at 1.4.2 with regenerated manifest metadata. REST and SignalR share the RFC3339 parser and validate power in note_observation. A stale REST power observation can be ignored while a healthy stream supplies newer power.

The driver can recover from a bogus future timestamp or a backwards clock jump after 30 seconds if the lower source timestamps advance. Mode 1 clears the power watermark. The tests cover future-dated REST power, a backwards jump on SignalR, and the mode-1 reset. Repeated identical timestamps do not qualify for clock recovery.

make check and all five GitHub checks passed on the updated branch.

@segran2

segran2 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@frahlg, the follow-up to your November 12 review is ready for another look. It covers the REST and SignalR timestamp handling, recovery after clock jumps, and the passing checks on b52a558.

@miravoss26 miravoss26 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Easee Cloud 1.4.2: one RFC3339 parser that compares instants (offset + fraction), a power-ordering watermark shared by SignalR and REST, rejection of older power / conflicting values at one instant / connected power without a valid timestamp, and a 30 s strictly-advancing recovery path for a future timestamp or clock jump. Resets on mode 1.

Read the whole diff, no blocking findings.

  • Security: nothing. No new hosts or deps, and the error text still goes through redact_http_err.
  • Checked: source_time_key produces a zero-padded day/second key plus a trailing-zero-stripped fraction, so plain string comparison orders correctly ("12" < "5" for .12 vs .5). The gsub two-value return is safe in that assignment. Missing TotalPower on REST in mode ≥2 was already a state_problem, so applying power unconditionally doesn't add a new failure.
  • Needs a human eye (operational): after a rejected power observation, stream_power_problem blocks all reporting until a valid newer reading lands, and a lower-timeline recovery needs ≥30 s plus two advancing readings. A single bad future-dated packet therefore blinds the driver for at least ~30 s. That seems an intended trade for "never report a stale 0 W", but it's the behaviour I'd want a field run to confirm on a real charger, since the PR itself says the cause of the ftw#1674 incidents is still unproven.
  • Nit: the recovery state machine (recovery_key / recovery_since) is subtle. A short comment with the timeline (accept at t≥30 s when key > recovery_key) would help the next reader.

Safe to merge from my read; I'd want the on-charger confirmation above before relying on it for #1674.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants