Skip to content

Ingestors install the stored role on role sync - #78

Merged
VibeSWE merged 4 commits into
mainfrom
fix/role-sync-ordering
Oct 25, 2026
Merged

VibeSWE merged 4 commits into
mainfrom
fix/role-sync-ordering

Conversation

@VibeSWE

@VibeSWE VibeSWE commented Oct 25, 2026 •

Copy link
Copy Markdown

Summary

Follow-up from the #67 review (pre-existing, Medium): role syncs from the Query node to ingestors could be applied out of order.

  • querier_role::put commits the role to object storage, releases UPDATE_LOCK, then fans the sync out in parallel.
  • The ingestor receiver (ingestor_role::put) laid the request body over the stored metadata and installed the body, with no serialisation of its own.
  • So two quick PUTs, A then B, could leave an ingestor running A.

Fix (receiver only, no wire-format change): the ingestor now:

  1. takes its own UPDATE_LOCK;
  2. re-reads the tenant's metadata from storage;
  3. installs the stored role, or removes it from the live map if storage no longer has it;
  4. refreshes affected sessions, then writes staging as before.

The querier commits to storage before sending any sync, and each receiver reads and installs under the lock. So whichever sync an ingestor processes last reflects the latest write, and a late PUT sync can no longer bring back a deleted role.

Mixed versions: SyncRole and the route are unchanged. The body's tenant_id is still used; its privileges are now ignored.

A failed staging write after a sync is now logged as a warning instead of being dropped with let _.

Tests

The install step is factored into a helper. The handler path needs storage, which the test binary can't initialise. Two tests, committed red first:

  • a stale body doesn't override the stored role;
  • a sync for a role storage no longer has removes the live role.

Verification

  • cargo fmt --check ✅, cargo clippy -- -D warnings ✅
  • cargo test --lib: 1023 passed, 0 failed, 2 ignored

Not in this PR

  • Role DELETE never reaches ingestors: the receiver route DELETE /role/{name}/sync is commented out in ingest_server.rs (since upstream 91debc3a), so deleted roles stay live on ingestors until restart. Restoring it changes the ingestor's exposed routes, so it's left for a separate decision.
  • Failed syncs are only logged: an ingestor that misses one stays stale until the next sync of that role, or a restart.
  • Querier role DELETE locking: fixed in Take UPDATE_LOCK in querier role DELETE and before password resets #77.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

VibeSWE and others added 4 commits October 24, 2026 22:35
Extract the ingestor role-sync install into apply_synced_role (behaviour
unchanged) and add failing tests: a sync whose body is older than the
role in storage, or whose role was deleted since, must converge on the
stored state.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The querier releases UPDATE_LOCK before sending role syncs, so two quick
PUTs can have their syncs arrive out of order and leave ingestors with
the older body. The receiver already read the tenant's metadata from
object storage; it now installs the role found there (removing it if
deleted) instead of the request body, holding its own UPDATE_LOCK across
the read and install so a later sync never installs older state. The
wire format is unchanged. Staging-write failures are now logged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Internal roles such as super-admin exist only in memory, so a sync naming
one would remove it now that ingestors install what storage holds.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

@VibeSWE VibeSWE left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review of PR #78 (at a196fc34, fixes in fff61ed5)

One Sol 6.1 and one Opus 5.5 reviewer. Neither found anything blocking.

Both confirmed:

  • Commit before send: the querier commits to remote storage before sending the sync.
  • Reads are fresh: get_metadata reads object storage directly, with no staging fallback or cache.
  • Locks: no lock inversion, no parking_lot guard held across .await, and nothing re-enters UPDATE_LOCK.
  • HTTP contract and mixed versions: both unchanged. No ingestor error can fail the user's PUT, before or after this PR.
  • Nothing else depends on the body: no bootstrap, migration or test path relies on the request body being installed.
Severity Finding Outcome
Low stale_sync_body_does_not_override_stored_role no longer tested an overwrite after the helper stopped taking a body Renamed to installs_stored_role_over_live_role; it now seeds live A first
Low Session removal was untested New test: direct holders and group members lose their sessions; unrelated users and other tenants keep theirs
Low A sync naming an internal role (e.g. super-admin, which is never in storage) would now remove it Fixed: internal live roles are left alone. Red test first
Low Convergence assumes read-after-write-consistent storage and delivery of the latest sync Documented. Failed syncs are still only logged (see the PR body)
Nit The ingestor holds UPDATE_LOCK across one storage GET Kept: ingestor_rbac.rs already does the same

Separate High (pre-existing) found during review: role removals from users never apply on ingestors. The receiver validates against remote metadata the querier has already updated. A fix is in progress on its own branch.

Local at fff61ed5: fmt and clippy -- -D warnings clean; cargo test --lib 1025 passed, 0 failed.

@VibeSWE
VibeSWE merged commit f02972f into main Oct 25, 2026
9 checks passed
@VibeSWE
VibeSWE deleted the fix/role-sync-ordering branch October 25, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant