Repository navigation
Ingestors install the stored role on role sync - #78
Merged
Merged
Conversation
Extract the ingestor role-sync install into apply_synced_role (behaviour unchanged) and add failing tests: a sync whose body is older than the role in storage, or whose role was deleted since, must converge on the stored state. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The querier releases UPDATE_LOCK before sending role syncs, so two quick PUTs can have their syncs arrive out of order and leave ingestors with the older body. The receiver already read the tenant's metadata from object storage; it now installs the role found there (removing it if deleted) instead of the request body, holding its own UPDATE_LOCK across the read and install so a later sync never installs older state. The wire format is unchanged. Staging-write failures are now logged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Internal roles such as super-admin exist only in memory, so a sync naming one would remove it now that ingestors install what storage holds. Co-Authored-By: Claude Opus 5.5 <[email protected]>
VibeSWE
commented
Oct 25, 2026
VibeSWE
left a comment
Author
There was a problem hiding this comment.
Adversarial review of PR #78 (at a196fc34, fixes in fff61ed5)
One Sol 6.1 and one Opus 5.5 reviewer. Neither found anything blocking.
Both confirmed:
- Commit before send: the querier commits to remote storage before sending the sync.
- Reads are fresh:
get_metadatareads object storage directly, with no staging fallback or cache. - Locks: no lock inversion, no parking_lot guard held across
.await, and nothing re-entersUPDATE_LOCK. - HTTP contract and mixed versions: both unchanged. No ingestor error can fail the user's PUT, before or after this PR.
- Nothing else depends on the body: no bootstrap, migration or test path relies on the request body being installed.
| Severity | Finding | Outcome |
|---|---|---|
| Low | stale_sync_body_does_not_override_stored_role no longer tested an overwrite after the helper stopped taking a body |
Renamed to installs_stored_role_over_live_role; it now seeds live A first |
| Low | Session removal was untested | New test: direct holders and group members lose their sessions; unrelated users and other tenants keep theirs |
| Low | A sync naming an internal role (e.g. super-admin, which is never in storage) would now remove it |
Fixed: internal live roles are left alone. Red test first |
| Low | Convergence assumes read-after-write-consistent storage and delivery of the latest sync | Documented. Failed syncs are still only logged (see the PR body) |
| Nit | The ingestor holds UPDATE_LOCK across one storage GET |
Kept: ingestor_rbac.rs already does the same |
Separate High (pre-existing) found during review: role removals from users never apply on ingestors. The receiver validates against remote metadata the querier has already updated. A fix is in progress on its own branch.
Local at fff61ed5: fmt and clippy -- -D warnings clean; cargo test --lib 1025 passed, 0 failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up from the #67 review (pre-existing, Medium): role syncs from the Query node to ingestors could be applied out of order.
querier_role::putcommits the role to object storage, releasesUPDATE_LOCK, then fans the sync out in parallel.ingestor_role::put) laid the request body over the stored metadata and installed the body, with no serialisation of its own.Fix (receiver only, no wire-format change): the ingestor now:
UPDATE_LOCK;The querier commits to storage before sending any sync, and each receiver reads and installs under the lock. So whichever sync an ingestor processes last reflects the latest write, and a late PUT sync can no longer bring back a deleted role.
Mixed versions:
SyncRoleand the route are unchanged. The body'stenant_idis still used; itsprivilegesare now ignored.A failed staging write after a sync is now logged as a warning instead of being dropped with
let _.Tests
The install step is factored into a helper. The handler path needs storage, which the test binary can't initialise. Two tests, committed red first:
Verification
cargo fmt --check✅,cargo clippy -- -D warnings✅cargo test --lib: 1023 passed, 0 failed, 2 ignoredNot in this PR
DELETE /role/{name}/syncis commented out iningest_server.rs(since upstream91debc3a), so deleted roles stay live on ingestors until restart. Restoring it changes the ingestor's exposed routes, so it's left for a separate decision.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.