Repository navigation
feat: promote allowlisted caller context onto every agent span - #2575
rtemperini wants to merge 23 commits into
Conversation
krisztianfekete
left a comment
There was a problem hiding this comment.
Thanks, added a few initial comments!
Agent spans record what an agent did but not who asked for it, so traces cannot be filtered or grouped by the calling user, Slack thread, or ticket that triggered them. Read an operator-defined allowlist of context keys from W3C baggage and A2A message metadata and merge them into the request-scoped attribute bag. The existing span processor then stamps them on every span of the request, which is what trace-level filtering in Langfuse and comparable backends requires; attaching them to the root span alone leaves most views unfilterable. Baggage is the primary source because it already survives the controller, agent, sub-agent, and tool hops under the composite propagator, so a value set once at the edge needs no further plumbing. A2A message metadata covers callers that cannot set headers and, being per-message, takes precedence. Caller data is untrusted, so only allowlisted keys are read, the allowlist is capped, values are truncated and stripped of control characters, and every attribute is namespaced under kagent.context. so it cannot shadow a semantic convention attribute such as service.name. The allowlist is empty by default, which disables promotion entirely. Signed-off-by: Ricardo Temperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Bring the Python runtimes to parity with the Go ADK. The Go runtime already reads A2A message metadata into span attributes (kagent-dev#1734, kagent-dev#1737); the Python runtimes ignored inbound metadata entirely, so agents on the Python runtime had no way to get caller identity onto their traces. Mirror the Go implementation exactly: the same KAGENT_TRACE_CONTEXT_KEYS allowlist, the same baggage-then-metadata precedence, the same limits, the same control character stripping, and the same kagent.context. namespace, so the two runtimes cannot drift. Values are merged into the request-scoped attribute bag that KagentAttributesSpanProcessor stamps onto every span. The ADK, LangGraph, and CrewAI executors all promote context through the shared helper. Reading a Message's protobuf Struct metadata moves into kagent.core.a2a.read_message_metadata rather than being repeated per package. Also assert that the OTel SDK's default propagator carries baggage: the Python runtime relies on that default rather than configuring a propagator, so an SDK change that dropped it would silently break the baggage path. Signed-off-by: Ricardo Temperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Expose the allowlist as the Helm value otel.tracing.contextKeys, rendered into the controller ConfigMap as KAGENT_TRACE_CONTEXT_KEYS and forwarded to the agents the controller creates. The variable needs explicit forwarding because collectOtelEnvFromProcess carries only OTEL_ prefixed names. Which caller-supplied data reaches a trace backend is cluster-wide operator policy, so the value is applied after the Harness environment, and any inherited entry of the same name is dropped first. Without that second step a Harness could enable promotion whenever the operator had configured nothing. The value defaults to an empty list, so the ConfigMap key is absent and the runtimes promote nothing unless an operator opts in. Signed-off-by: Ricardo Temperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Cover the configuration knob, why baggage is the primary propagation mechanism, why the attributes are stamped on every span rather than the root, the safety properties that bound untrusted caller input, and how to rename attributes in the OTel Collector for a backend that expects its own names. Signed-off-by: Ricardo Temperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Address review of the caller-context tracing allowlist: use an OIDC subject as the user.id example, leave user.*, enduser.*, and session.id unprefixed, and hash mapped values with HMAC-SHA256 instead of putting the original on the span. Signed-off-by: Ricardo Temperini <[email protected]> Co-authored-by: Cursor <[email protected]>
7c4da91 to
2056591
Compare
Thanks for the review @krisztianfekete , just updated with your comments, let me know if anything else required |
Caller-supplied attributes no longer override runtime keys. Registry passthrough is an explicit set, empty metadata cannot wipe baggage, and the allowlist is parsed once per process. Signed-off-by: Ricardo Temperini <[email protected]>
Reject non-string hash/to so Python cannot emit plaintext, format 1e-20 as Go does, and stamp caller context from the OpenAI executor. Signed-off-by: Ricardo Temperini <[email protected]>
Skip KAGENT_TRACE_CONTEXT_KEYS source keys in generic A2A metadata span attributes so hashed or omitted values cannot leak on a2a.message.metadata.<from>. Signed-off-by: Ricardo Temperini <[email protected]>
|
@krisztianfekete thanks for the reviews, I've added a few updates, let me know if should I take a look on anything else |
Signed-off-by: rtemperini <[email protected]> Co-authored-by: Cursor <[email protected]>
|
@krisztianfekete I merged main to clear the conflict with the new OTEL env forwarding. Could you take another look when you have a moment? The earlier review comments should still be the open questions. |
There was a problem hiding this comment.
Before we go deeper on the details though, I think we can get to the same behaviour with a lot less code by using upstream building blocks.
Both runtimes ship a baggage -> spanattribute processor, each taking a filter/predicate, which is basically what MergeCallerContextAttributes + the attributes span processor are doing by hand.
If we then promote allowlisted A2A message.metadata keys into baggage at the executor boundary rather than reading them as a second source with its own precedence rule, we'd have one code path instead of two, and metadata-sourced values would survive subagent hops as well. That would also let us drop most of the policy layer to where OTel already puts it: value truncation is OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, renaming can be done via a collector transformation, and hashing is really an collector or gw concern.
I'd rather not hardcode am unprefixed set, as to: is operator config, not caller input, so prefixing it doesn't really defend against anything, and right now it makes gen_ai.conversation.id unreachable even though our own ADK executor already sets it.
Resolve conflicts from the v2 translator move and the rewritten architecture docs so the allowlist still forwards after main. Signed-off-by: rtemperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Use the upstream baggage-to-span processors instead of a second request-scoped merge path, so allowlisted identity reaches every span including generate_content and survives sub-agent hops. Signed-off-by: rtemperini <[email protected]> Co-authored-by: Cursor <[email protected]>
Leftover {hash:...} allowlist entries must fail closed so a raw email cannot land on user.hash. Forward OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT into agent runtimes so the documented truncation control actually applies.
Signed-off-by: rtemperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
|
@krisztianfekete I reworked this around the upstream baggage-to-span processors and promote allowlisted metadata into baggage at the executor, so there is one path. The custom hash/prefix/merge layer is gone; leftover hash mappings fail closed. Could you take another look when you have a moment? |
Keep TelemetryConfig, OtelEnvFromProcess, and applyOperatorOnlyEnv together so compiled OTEL export and the operator-only allowlist both survive. Signed-off-by: rtemperini <[email protected]>
|
Hi @krisztianfekete I've rebased and fixed conflicts with the latests main rebase, can you take a look on this when you have a minute, will be good to address this to be able to have the proper context attributes propagation. Thanks in advance |
Leftover {hash:...} allowlist entries were dropped before their from
key entered the suppression set, so the executor still stamped the
raw value as a2a.message.metadata.<from>. Keep those sources covered
without re-implementing hashing.
Signed-off-by: rtemperini <[email protected]>
Signed-off-by: rtemperini <[email protected]>
|
Hi @krisztianfekete, I rebased onto the latest main and fixed the conflicts. The new telemetry stack is kept, and the caller context promotion sits on top of it. Can you take a look when you have a minute? Thanks |
Signed-off-by: rtemperini <[email protected]>
|
Hi @EItanya @supreme-gg-gg I rebased this again onto the latest main and fixed the conflicts. The checks that can run from the fork are green, can you take a look when you have a minute, or let me know if anything looks off? Thanks in advance |
Signed-off-by: rtemperini <[email protected]>
|
This pull request has been marked as stale because of no activity in the last 15 days. It will be closed in the next 5 days unless it is tagged "no stalebot" or other activity occurs. |
Signed-off-by: rtemperini <[email protected]>
Signed-off-by: rtemperini <[email protected]>
|
Thanks @moezdil, I removed the unrestricted metadata export and added regression coverage. The PR is also updated for current main. Could you and @krisztianfekete take another look? Let me know if there is anything else to take a look and thanks again for the review |
Signed-off-by: rtemperini <[email protected]>
Summary
Promote an operator-selected allowlist of caller context onto agent spans, so traces can be filtered by an opaque user identifier, conversation thread, or ticket. Follows #1734 / #1737.
Both runtimes use the upstream baggage-to-span processors. Allowlisted A2A
message.metadatais promoted into baggage before the invocation span starts, so the invocation and descendant spans share one path. Python uses the common A2A telemetry wrapper for ADK, LangGraph, CrewAI, and OpenAI Agents.This update removes the PR's unrestricted
a2a.message.metadata.*export path flagged in review and integrates the feature with currentmain's telemetry setup.Configuration
The Helm value sets
KAGENT_TRACE_CONTEXT_KEYSon the controller, which forwards that policy to managed kagent runtimes. Direct runtime configuration accepts comma-separated keys or JSON strings and{from, to}mappings. The previous draft'sotel.tracing.contextKeys/otel.traces.contextKeyssetting is replaced bycontroller.traceContextKeysto fit the current chart.sub, never names or email addresses. Hashing belongs at the gateway or Collector; leftoverhashmappings are rejected instead of emitting plaintext.OTEL_PROPAGATORS=tracecontext,baggageon each participating process. Current kagent's tracecontext-only default is preserved, and controllerOTEL_*overrides remain process-local.OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT.No CRD or API changes. See
docs/architecture/trace-context.mdfor configuration and propagation details.Validation
Regression coverage checks disabled promotion, allowlisted invocation/model attributes, rejected unrelated metadata, preserved trusted identity, real protobuf metadata decoding, and request-context cleanup. No live Kubernetes deployment was performed.