Skip to content

fix(substrate): scope Substrate reads and page tokens to the watched namespaces - #3261

Merged
EItanya merged 16 commits into
kagent-dev:mainfrom
danielle-mustillo:danielle-mustillo/substrate-atespace-scope-indep
Oct 21, 2026
Merged

EItanya merged 16 commits into
kagent-dev:mainfrom
danielle-mustillo:danielle-mustillo/substrate-atespace-scope-indep

Conversation

@danielle-mustillo

@danielle-mustillo danielle-mustillo commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

A kagent install that watches only its own namespace can share a Substrate with other installs. The Substrate API showed it every install's actors and actor templates, and it served another tenant's atespace or namespace when a caller named it. The paging rewrite in #2771 dropped the watched-namespace filter for actors and templates. Page tokens leaked too. ate-api tokens are readable base64 JSON that hold the last row's key, so a token passed through to the caller can name another tenant's actor or worker.

What changed

  • One scope, built from the watched namespaces, applies to every Substrate read: actors, templates, workers, and worker pools. An empty request means the watched namespaces. Any other namespace or atespace returns PermissionDenied.
  • The controller asks ate-api for actors and templates one watched atespace at a time. The actor page token is kagent's own, base64url("atespace/inner").
  • ate-api cannot filter workers by namespace. So the controller reads all workers, keeps the watched ones, and pages them by name with the token base64url("namespace/name"). A bad or foreign token returns InvalidArgument, and ate-api worker errors return only the gRPC code.

Scope

  • Covers ListSubstrateActors, ListSubstrateWorkers, and GetSubstrateSummary. Only proto comments change. A cluster-wide install passes ate-api's pages and tokens through, as before.
  • The UI atespace placeholder says "All watched atespaces".
  • Not included: duplicate row checks after scoped ate-api lists.

Tradeoffs

  • Each worker list request reads every tenant's workers. Passing ate-api's token through costs less but leaks names. Replace this with an ate-api namespace filter when one exists.

Blast Radius

Only installs that watch some namespaces change. They see less, and requests outside their scope fail. A page token from an older controller returns InvalidArgument, so reload the UI page.

Verification

  • Kind cluster with one shared Substrate and two installs that each watch one tenant. As tenant-a, ListSubstrateActors {} returned both tenants before and only tenant-a after. A request that named tenant-b returned its data before and PermissionDenied after. A worker page token decoded to a tenant-b worker before and to a tenant-a worker name after. A cluster-wide install still sees both tenants.
  • Kind cluster with actors in tenant-a, tenant-b, and tenant-c, and a controller that watches tenant-a and tenant-c. Actor pages no longer reach tenant-b, and a forged tenant-b/ token returns InvalidArgument.
  • Unit tests send over 600 forged tokens to both lists. No read, row, or token leaves the watched namespaces. Each test commit fails on its parent. go test ./core/internal/service/... and make lint pass.

@danielle-mustillo
danielle-mustillo requested a review from a team as a code owner October 19, 2026 00:39
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 19, 2026
@danielle-mustillo
danielle-mustillo marked this pull request as draft October 19, 2026 02:01
@danielle-mustillo
danielle-mustillo marked this pull request as ready for review October 19, 2026 05:25

@EItanya EItanya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review. I looked through them to make sure they weren't crazy and they all seem worth looking at.

Thanks for the fix! I found one correctness issue and two follow-ups:

  1. Can we deduplicate observedNamespaces once when constructing the service? watch_namespaces does not require unique values, and resolveSubstrateScope preserves duplicates. walkActors and substrateActorTemplates then query each entry, so ["tenant-a", "tenant-a"] reports every actor and template twice. I reproduced this with one actor and one template: the summary returned ActorCount == 2 and two template entries. A regression test should include non-adjacent duplicates as well.

  2. The public protobuf comments for GetSubstrateSummaryRequest.atespace and ListSubstrateActorsRequest.atespace still say that an empty value lists all atespaces. With this change, it means the watched namespaces unless the controller is cluster-wide, and an explicitly unwatched atespace returns PermissionDenied. Can we update the source comments and regenerate so clients don't interpret a scoped result as cluster-wide inventory?

  3. TestSubstrateTenantIsolation uses an empty Kubernetes client, so it doesn't exercise worker-pool isolation even though the PR applies the scope to every Substrate read. Can we seed in-scope and foreign pools and assert the exact namespace/name pairs for scoped, multi-namespace, and cluster-wide summaries?

@danielle-mustillo

Copy link
Copy Markdown
Contributor Author

Thanks @EItanya. All three were real. Fixed in 3 commits:

  1. Duplicate watch namespaces. e70146e adds a [tenant-a, tenant-c, tenant-a] case. It failed on the old code with 3 actors, agent-a twice and tenant-a/pool-a twice. 9b19e56 drops duplicates once in NewService and keeps the configured order. I left the config itself accepting duplicates, so an existing install with a repeated entry still starts.
  2. Proto comments. 4774891 updates both atespace comments and regenerates the Go and TS output.
  3. Worker pools. e70146e seeds pools in tenant-a, tenant-b and tenant-c. It asserts the exact namespace/name pairs for the single-namespace, two-namespace, duplicate and cluster-wide summaries. To check that the test can fail, I removed the namespace filter in listWorkerPools. Three subtests failed.

@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 19, 2026
Comment thread go/core/internal/service/system/service.go Outdated
@danielle-mustillo danielle-mustillo self-assigned this Oct 19, 2026
@EItanya

EItanya commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

🤖 AI-generated comment.

Can we have ListSubstrateActors query each atespace in watchNamespaces, like the summary already does, and let kagent own the continuation token?

Preserving the upstream token means the multi-namespace case still reads globally and filters afterward. That requires global Substrate permissions and leaves users paging through empty results for other tenants. It also leaves a hole in the isolation fix: Substrate's cursor contains the last actor's atespace and name, encoded as base64 JSON. Filtering the actor out of the response still exposes those identifiers through the token.

For watchNamespaces: [tenant-a, tenant-c], we could finish paging through tenant-a, then continue with tenant-c. The kagent cursor would track the current atespace and its upstream token. An explicit atespace would still be checked against the watched namespaces, and a cluster-wide controller could keep the global query. I think revisiting the upstream-token contract is worthwhile here so the query and pagination both respect the scope.

@danielle-mustillo

This comment was marked as outdated.

@danielle-mustillo danielle-mustillo changed the title fix(substrate): scope actors and templates to the watched namespaces fix(substrate): scope Substrate reads and page tokens to the watched namespaces Oct 20, 2026
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 20, 2026
@EItanya

EItanya commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

LGTM, just needs rebase

A kagent that watches only its own namespace listed every tenant's actors
and ActorTemplates from a shared Substrate, and served a foreign atespace
or namespace when a caller named one.

Resolve one scope from the watched namespaces for every Substrate read.
An empty request means the watched namespaces, and a namespace or atespace
outside them is refused with PermissionDenied. A cluster-wide install
still sees everything.

Signed-off-by: Danielle Mustillo <[email protected]>
…refusal wording

A controller watching tenant-a and tenant-c must never page tenant-b actors
to the caller, must ask ate-api only for its own atespaces when it builds
the summary, and must hide a worker that reports no namespace. The refusal
message must say the namespace is not watched, since the gRPC layer sends
only the message.

Signed-off-by: Danielle Mustillo <[email protected]>
…workers

The summary for a controller watching several namespaces asked ate-api for
every atespace and filtered the rows itself, so ate-api served every
tenant's actors and templates to it. It now walks actors and drains
templates once per atespace in scope. ListSubstrateActors still reads one
upstream page across all atespaces and filters it, because one upstream
token cannot span several requests.

A worker that reports no namespace now belongs only to a cluster-wide
scope, and the refusal message names the namespace that is not watched,
since the gRPC layer sends only the message.

Signed-off-by: Danielle Mustillo <[email protected]>
The isolation test built its fake cluster with no WorkerPools, so it never
checked that the summary lists only pools in the watched namespaces. It
also never listed a namespace twice in watchNamespaces, which counted
tenant-a's actors, templates and pools twice.

Signed-off-by: Danielle Mustillo <[email protected]>
watchNamespaces allows duplicates, and every Substrate read asks ate-api
and Kubernetes once per watched namespace, so a duplicate entry doubled
the summary's actors, templates and worker pools. Drop duplicates once
when the service is built, keeping the configured order.

Signed-off-by: Danielle Mustillo <[email protected]>
GetSubstrateSummaryRequest.atespace and ListSubstrateActorsRequest.atespace
still said an empty value lists all atespaces. It now means the watched
namespaces unless the controller is cluster-wide, and an unwatched
atespace returns PermissionDenied.

Signed-off-by: Danielle Mustillo <[email protected]>
Sorting before Compact removes duplicates that are not adjacent.
ListNamespaces already sorts its output, so the order change is not
visible to callers.

Verified with go test ./core/internal/service/system, including the
case that watches [tenant-a, tenant-c, tenant-a].

Signed-off-by: Danielle Mustillo <[email protected]>
A controller watching tenant-a and tenant-c must page through its own
actors in watched-atespace order, scope every ate-api read to one watched
atespace, never hand out an empty page with a token, and never return a
token that names a tenant-b actor. A token it did not issue is
InvalidArgument, without echoing the token and without an ate-api call.

The cluster-wide pass-through test now runs cluster-wide, and the
assertion that a filtered-out first page still leads on is dropped: that
page is the leak.

Signed-off-by: Danielle Mustillo <[email protected]>
With two or more watched namespaces, ListSubstrateActors asked ate-api for
every atespace and passed its page token through. ate-api tokens are
readable base64 JSON holding the last row's atespace and name, so a token
could name another tenant's actor, and a page filtered to nothing still
carried a token.

A controller that watches some namespaces now reads each watched atespace
with its own scoped list, in sorted order, and returns its own token: the
atespace to read and ate-api's token inside it. A token it did not issue,
or one longer than 1 KiB, is InvalidArgument and never reaches ate-api.
NewService sorts and dedupes the watched list, since tokens resume in its
order. A cluster-wide controller still passes ate-api's page and token
through.

Signed-off-by: Danielle Mustillo <[email protected]>
… workers

A controller watching tenant-a and tenant-c must page its own workers once
each, in worker name order whatever order ate-api sends, never hand out an
empty page with a token, and never send its own token to ate-api. An
ate-api worker list error must reach the caller as its gRPC code only,
since the text can name another tenant's worker.

The tenant isolation fixtures now name their workers, as ate-api does.

Signed-off-by: Danielle Mustillo <[email protected]>
A controller that watches some namespaces passed ate-api's worker page
token through. The token holds the last worker's name, so it could name
another tenant's worker, and a page filtered to nothing still carried a
token.

Such a controller now drains ate-api's workers, keeps the ones in scope,
and pages them in worker name order. Its token is the last worker's name:
a cursor only, never sent to ate-api. ate-api worker list errors reach the
caller as the gRPC code only, for the list and the summary, since the text
can name another tenant's worker; the full error is logged. A
cluster-wide controller still passes ate-api's page, token and error
through.

Signed-off-by: Danielle Mustillo <[email protected]>
…spaces

The proto comments, the generated Go and TypeScript, the UI page type and
the Playwright deferral notes said an empty atespace lists all atespaces
and that every page passes Substrate's token through. A controller that
watches some namespaces lists only its watched atespaces, issues its own
tokens, and never returns an empty page with a token. The atespace filter
placeholder now reads "All watched atespaces".

Signed-off-by: Danielle Mustillo <[email protected]>
Drop comments that restate the code. The proto now says that an actor
page can be empty when another page exists, and gives the worker order
for each mode.

Signed-off-by: Danielle Mustillo <[email protected]>
… tokens

The fake ate-api now hands out base64 JSON tokens naming the page's last row, as
ate-api does, so assertOwnToken can see a token that would name another tenant's
row. New tests: scoped worker page tokens must be refused unless kagent issued
them, a scoped summary hides ate-api worker error text, forged tokens never reach
or return an unwatched namespace, and the longest names fit the page token.

Signed-off-by: Danielle Mustillo <[email protected]>
A controller that watches some namespaces used any worker page_token raw as the
name to resume after, so an ate-api token or junk silently returned a wrong or
short page. Worker tokens now share the actor tokens' shape (base64url of a
watched namespace, a slash, then the cursor) and the same 1 KiB cap, and
anything else is InvalidArgument. The length check now runs before decoding.
Cluster-wide installs still pass ate-api's tokens through.

Signed-off-by: Danielle Mustillo <[email protected]>
@danielle-mustillo
danielle-mustillo force-pushed the danielle-mustillo/substrate-atespace-scope-indep branch from 1c6a6a0 to a5d74ef Compare October 21, 2026 14:10
@EItanya
EItanya added this pull request to the merge queue Oct 21, 2026
Merged via the queue into kagent-dev:main with commit 45cd3b4 Oct 21, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants