Repository navigation
fix(substrate): scope Substrate reads and page tokens to the watched namespaces - #3261
Conversation
EItanya
left a comment
There was a problem hiding this comment.
🤖 AI-generated review. I looked through them to make sure they weren't crazy and they all seem worth looking at.
Thanks for the fix! I found one correctness issue and two follow-ups:
-
Can we deduplicate
observedNamespacesonce when constructing the service?watch_namespacesdoes not require unique values, andresolveSubstrateScopepreserves duplicates.walkActorsandsubstrateActorTemplatesthen query each entry, so["tenant-a", "tenant-a"]reports every actor and template twice. I reproduced this with one actor and one template: the summary returnedActorCount == 2and two template entries. A regression test should include non-adjacent duplicates as well. -
The public protobuf comments for
GetSubstrateSummaryRequest.atespaceandListSubstrateActorsRequest.atespacestill say that an empty value lists all atespaces. With this change, it means the watched namespaces unless the controller is cluster-wide, and an explicitly unwatched atespace returnsPermissionDenied. Can we update the source comments and regenerate so clients don't interpret a scoped result as cluster-wide inventory? -
TestSubstrateTenantIsolationuses an empty Kubernetes client, so it doesn't exercise worker-pool isolation even though the PR applies the scope to every Substrate read. Can we seed in-scope and foreign pools and assert the exact namespace/name pairs for scoped, multi-namespace, and cluster-wide summaries?
|
Thanks @EItanya. All three were real. Fixed in 3 commits:
|
|
🤖 AI-generated comment. Can we have Preserving the upstream token means the multi-namespace case still reads globally and filters afterward. That requires global Substrate permissions and leaves users paging through empty results for other tenants. It also leaves a hole in the isolation fix: Substrate's cursor contains the last actor's atespace and name, encoded as base64 JSON. Filtering the actor out of the response still exposes those identifiers through the token. For |
This comment was marked as outdated.
This comment was marked as outdated.
|
LGTM, just needs rebase |
…entory Signed-off-by: Danielle Mustillo <[email protected]>
A kagent that watches only its own namespace listed every tenant's actors and ActorTemplates from a shared Substrate, and served a foreign atespace or namespace when a caller named one. Resolve one scope from the watched namespaces for every Substrate read. An empty request means the watched namespaces, and a namespace or atespace outside them is refused with PermissionDenied. A cluster-wide install still sees everything. Signed-off-by: Danielle Mustillo <[email protected]>
…refusal wording A controller watching tenant-a and tenant-c must never page tenant-b actors to the caller, must ask ate-api only for its own atespaces when it builds the summary, and must hide a worker that reports no namespace. The refusal message must say the namespace is not watched, since the gRPC layer sends only the message. Signed-off-by: Danielle Mustillo <[email protected]>
…workers The summary for a controller watching several namespaces asked ate-api for every atespace and filtered the rows itself, so ate-api served every tenant's actors and templates to it. It now walks actors and drains templates once per atespace in scope. ListSubstrateActors still reads one upstream page across all atespaces and filters it, because one upstream token cannot span several requests. A worker that reports no namespace now belongs only to a cluster-wide scope, and the refusal message names the namespace that is not watched, since the gRPC layer sends only the message. Signed-off-by: Danielle Mustillo <[email protected]>
The isolation test built its fake cluster with no WorkerPools, so it never checked that the summary lists only pools in the watched namespaces. It also never listed a namespace twice in watchNamespaces, which counted tenant-a's actors, templates and pools twice. Signed-off-by: Danielle Mustillo <[email protected]>
watchNamespaces allows duplicates, and every Substrate read asks ate-api and Kubernetes once per watched namespace, so a duplicate entry doubled the summary's actors, templates and worker pools. Drop duplicates once when the service is built, keeping the configured order. Signed-off-by: Danielle Mustillo <[email protected]>
GetSubstrateSummaryRequest.atespace and ListSubstrateActorsRequest.atespace still said an empty value lists all atespaces. It now means the watched namespaces unless the controller is cluster-wide, and an unwatched atespace returns PermissionDenied. Signed-off-by: Danielle Mustillo <[email protected]>
Sorting before Compact removes duplicates that are not adjacent. ListNamespaces already sorts its output, so the order change is not visible to callers. Verified with go test ./core/internal/service/system, including the case that watches [tenant-a, tenant-c, tenant-a]. Signed-off-by: Danielle Mustillo <[email protected]>
A controller watching tenant-a and tenant-c must page through its own actors in watched-atespace order, scope every ate-api read to one watched atespace, never hand out an empty page with a token, and never return a token that names a tenant-b actor. A token it did not issue is InvalidArgument, without echoing the token and without an ate-api call. The cluster-wide pass-through test now runs cluster-wide, and the assertion that a filtered-out first page still leads on is dropped: that page is the leak. Signed-off-by: Danielle Mustillo <[email protected]>
With two or more watched namespaces, ListSubstrateActors asked ate-api for every atespace and passed its page token through. ate-api tokens are readable base64 JSON holding the last row's atespace and name, so a token could name another tenant's actor, and a page filtered to nothing still carried a token. A controller that watches some namespaces now reads each watched atespace with its own scoped list, in sorted order, and returns its own token: the atespace to read and ate-api's token inside it. A token it did not issue, or one longer than 1 KiB, is InvalidArgument and never reaches ate-api. NewService sorts and dedupes the watched list, since tokens resume in its order. A cluster-wide controller still passes ate-api's page and token through. Signed-off-by: Danielle Mustillo <[email protected]>
… workers A controller watching tenant-a and tenant-c must page its own workers once each, in worker name order whatever order ate-api sends, never hand out an empty page with a token, and never send its own token to ate-api. An ate-api worker list error must reach the caller as its gRPC code only, since the text can name another tenant's worker. The tenant isolation fixtures now name their workers, as ate-api does. Signed-off-by: Danielle Mustillo <[email protected]>
A controller that watches some namespaces passed ate-api's worker page token through. The token holds the last worker's name, so it could name another tenant's worker, and a page filtered to nothing still carried a token. Such a controller now drains ate-api's workers, keeps the ones in scope, and pages them in worker name order. Its token is the last worker's name: a cursor only, never sent to ate-api. ate-api worker list errors reach the caller as the gRPC code only, for the list and the summary, since the text can name another tenant's worker; the full error is logged. A cluster-wide controller still passes ate-api's page, token and error through. Signed-off-by: Danielle Mustillo <[email protected]>
…spaces The proto comments, the generated Go and TypeScript, the UI page type and the Playwright deferral notes said an empty atespace lists all atespaces and that every page passes Substrate's token through. A controller that watches some namespaces lists only its watched atespaces, issues its own tokens, and never returns an empty page with a token. The atespace filter placeholder now reads "All watched atespaces". Signed-off-by: Danielle Mustillo <[email protected]>
Drop comments that restate the code. The proto now says that an actor page can be empty when another page exists, and gives the worker order for each mode. Signed-off-by: Danielle Mustillo <[email protected]>
… tokens The fake ate-api now hands out base64 JSON tokens naming the page's last row, as ate-api does, so assertOwnToken can see a token that would name another tenant's row. New tests: scoped worker page tokens must be refused unless kagent issued them, a scoped summary hides ate-api worker error text, forged tokens never reach or return an unwatched namespace, and the longest names fit the page token. Signed-off-by: Danielle Mustillo <[email protected]>
A controller that watches some namespaces used any worker page_token raw as the name to resume after, so an ate-api token or junk silently returned a wrong or short page. Worker tokens now share the actor tokens' shape (base64url of a watched namespace, a slash, then the cursor) and the same 1 KiB cap, and anything else is InvalidArgument. The length check now runs before decoding. Cluster-wide installs still pass ate-api's tokens through. Signed-off-by: Danielle Mustillo <[email protected]>
1c6a6a0 to
a5d74ef
Compare
Why
A kagent install that watches only its own namespace can share a Substrate with other installs. The Substrate API showed it every install's actors and actor templates, and it served another tenant's atespace or namespace when a caller named it. The paging rewrite in #2771 dropped the watched-namespace filter for actors and templates. Page tokens leaked too. ate-api tokens are readable base64 JSON that hold the last row's key, so a token passed through to the caller can name another tenant's actor or worker.
What changed
PermissionDenied.base64url("atespace/inner").base64url("namespace/name"). A bad or foreign token returnsInvalidArgument, and ate-api worker errors return only the gRPC code.Scope
ListSubstrateActors,ListSubstrateWorkers, andGetSubstrateSummary. Only proto comments change. A cluster-wide install passes ate-api's pages and tokens through, as before.Tradeoffs
Blast Radius
Only installs that watch some namespaces change. They see less, and requests outside their scope fail. A page token from an older controller returns
InvalidArgument, so reload the UI page.Verification
ListSubstrateActors {}returned both tenants before and only tenant-a after. A request that named tenant-b returned its data before andPermissionDeniedafter. A worker page token decoded to a tenant-b worker before and to a tenant-a worker name after. A cluster-wide install still sees both tenants.tenant-a,tenant-b, andtenant-c, and a controller that watchestenant-aandtenant-c. Actor pages no longer reachtenant-b, and a forgedtenant-b/token returnsInvalidArgument.go test ./core/internal/service/...andmake lintpass.