Skip to content

feat: promote allowlisted caller context onto every agent span - #2575

Open
rtemperini wants to merge 23 commits into
kagent-dev:mainfrom
rtemperini:feat/trace-caller-context-attributes
Open

rtemperini wants to merge 23 commits into
kagent-dev:mainfrom
rtemperini:feat/trace-caller-context-attributes

Conversation

@rtemperini

@rtemperini rtemperini commented Aug 26, 2026 •

Copy link
Copy Markdown

Summary

Promote an operator-selected allowlist of caller context onto agent spans, so traces can be filtered by an opaque user identifier, conversation thread, or ticket. Follows #1734 / #1737.

Both runtimes use the upstream baggage-to-span processors. Allowlisted A2A message.metadata is promoted into baggage before the invocation span starts, so the invocation and descendant spans share one path. Python uses the common A2A telemetry wrapper for ADK, LangGraph, CrewAI, and OpenAI Agents.

This update removes the PR's unrestricted a2a.message.metadata.* export path flagged in review and integrates the feature with current main's telemetry setup.

Configuration

controller:
  traceContextKeys:
    - {from: sub, to: user.id}
    - thread_id

The Helm value sets KAGENT_TRACE_CONTEXT_KEYS on the controller, which forwards that policy to managed kagent runtimes. Direct runtime configuration accepts comma-separated keys or JSON strings and {from, to} mappings. The previous draft's otel.tracing.contextKeys / otel.traces.contextKeys setting is replaced by controller.traceContextKeys to fit the current chart.

  • Empty allowlist disables promotion. Other metadata is never exported by this feature.
  • Existing non-empty destination baggage wins. Empty and non-scalar metadata is skipped.
  • Destination attribute names are used as configured. The policy is capped at 32 mappings and 64 characters per key, and parsing is cached for the process lifetime.
  • Runtime identity remains authoritative where the request processors set it. A Harness cannot enable or widen the operator's allowlist.
  • Use opaque identifiers such as OIDC sub, never names or email addresses. Hashing belongs at the gateway or Collector; leftover hash mappings are rejected instead of emitting plaintext.
  • Cross-process baggage remains separately opt-in through OTEL_PROPAGATORS=tracecontext,baggage on each participating process. Current kagent's tracecontext-only default is preserved, and controller OTEL_* overrides remain process-local.
  • Value truncation uses the SDK's OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT.

No CRD or API changes. See docs/architecture/trace-context.md for configuration and propagation details.

Validation

  • Go race checks passed for telemetry, A2A executors/server, and all translator packages.
  • 133 Python core and ADK executor tests passed, including common-handler coverage for all four Python runtime profiles.
  • All 327 Helm chart tests passed.
  • Python lint/format and whitespace checks passed.

Regression coverage checks disabled promotion, allowlisted invocation/model attributes, rejected unrelated metadata, preserved trusted identity, real protobuf metadata decoding, and request-context cleanup. No live Kubernetes deployment was performed.

@rtemperini
rtemperini requested review from a team and supreme-gg-gg as code owners August 26, 2026 13:44
@github-actions github-actions Bot added the enhancement New feature or request label Aug 26, 2026

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added a few initial comments!

Comment thread docs/architecture/trace-context.md Outdated
Comment thread helm/kagent/values.yaml Outdated
Comment thread docs/architecture/trace-context.md Outdated
Comment thread docs/architecture/trace-context.md Outdated
Comment thread go/core/pkg/env/otel.go Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
rtemperini and others added 5 commits August 28, 2026 12:09
Agent spans record what an agent did but not who asked for it, so traces
cannot be filtered or grouped by the calling user, Slack thread, or ticket
that triggered them.

Read an operator-defined allowlist of context keys from W3C baggage and A2A
message metadata and merge them into the request-scoped attribute bag. The
existing span processor then stamps them on every span of the request, which
is what trace-level filtering in Langfuse and comparable backends requires;
attaching them to the root span alone leaves most views unfilterable.

Baggage is the primary source because it already survives the controller,
agent, sub-agent, and tool hops under the composite propagator, so a value
set once at the edge needs no further plumbing. A2A message metadata covers
callers that cannot set headers and, being per-message, takes precedence.

Caller data is untrusted, so only allowlisted keys are read, the allowlist is
capped, values are truncated and stripped of control characters, and every
attribute is namespaced under kagent.context. so it cannot shadow a semantic
convention attribute such as service.name.

The allowlist is empty by default, which disables promotion entirely.

Signed-off-by: Ricardo Temperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Bring the Python runtimes to parity with the Go ADK. The Go runtime already
reads A2A message metadata into span attributes (kagent-dev#1734, kagent-dev#1737); the Python
runtimes ignored inbound metadata entirely, so agents on the Python runtime
had no way to get caller identity onto their traces.

Mirror the Go implementation exactly: the same KAGENT_TRACE_CONTEXT_KEYS
allowlist, the same baggage-then-metadata precedence, the same limits, the
same control character stripping, and the same kagent.context. namespace, so
the two runtimes cannot drift. Values are merged into the request-scoped
attribute bag that KagentAttributesSpanProcessor stamps onto every span.

The ADK, LangGraph, and CrewAI executors all promote context through the
shared helper. Reading a Message's protobuf Struct metadata moves into
kagent.core.a2a.read_message_metadata rather than being repeated per package.

Also assert that the OTel SDK's default propagator carries baggage: the
Python runtime relies on that default rather than configuring a propagator,
so an SDK change that dropped it would silently break the baggage path.

Signed-off-by: Ricardo Temperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Expose the allowlist as the Helm value otel.tracing.contextKeys, rendered
into the controller ConfigMap as KAGENT_TRACE_CONTEXT_KEYS and forwarded to
the agents the controller creates.

The variable needs explicit forwarding because collectOtelEnvFromProcess
carries only OTEL_ prefixed names.

Which caller-supplied data reaches a trace backend is cluster-wide operator
policy, so the value is applied after the Harness environment, and any
inherited entry of the same name is dropped first. Without that second step a
Harness could enable promotion whenever the operator had configured nothing.

The value defaults to an empty list, so the ConfigMap key is absent and the
runtimes promote nothing unless an operator opts in.

Signed-off-by: Ricardo Temperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Cover the configuration knob, why baggage is the primary propagation
mechanism, why the attributes are stamped on every span rather than the root,
the safety properties that bound untrusted caller input, and how to rename
attributes in the OTel Collector for a backend that expects its own names.

Signed-off-by: Ricardo Temperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Address review of the caller-context tracing allowlist: use an OIDC
subject as the user.id example, leave user.*, enduser.*, and session.id
unprefixed, and hash mapped values with HMAC-SHA256 instead of putting
the original on the span.

Signed-off-by: Ricardo Temperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
@rtemperini
rtemperini force-pushed the feat/trace-caller-context-attributes branch from 7c4da91 to 2056591 Compare August 28, 2026 10:21
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Aug 28, 2026
@rtemperini

Copy link
Copy Markdown
Author

Thanks, added a few initial comments!

Thanks for the review @krisztianfekete , just updated with your comments, let me know if anything else required

Comment thread go/adk/pkg/a2a/executor.go Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go
Comment thread go/adk/pkg/telemetry/context_attributes.go
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
Comment thread python/packages/kagent-adk/src/kagent/adk/_agent_executor.py Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
Comment thread go/adk/pkg/telemetry/context_attributes.go Outdated
Caller-supplied attributes no longer override runtime keys. Registry
passthrough is an explicit set, empty metadata cannot wipe baggage,
and the allowlist is parsed once per process.

Signed-off-by: Ricardo Temperini <[email protected]>
Reject non-string hash/to so Python cannot emit plaintext, format
1e-20 as Go does, and stamp caller context from the OpenAI executor.

Signed-off-by: Ricardo Temperini <[email protected]>
Skip KAGENT_TRACE_CONTEXT_KEYS source keys in generic A2A metadata
span attributes so hashed or omitted values cannot leak on
a2a.message.metadata.<from>.

Signed-off-by: Ricardo Temperini <[email protected]>
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Aug 31, 2026
@rtemperini

Copy link
Copy Markdown
Author

@krisztianfekete thanks for the reviews, I've added a few updates, let me know if should I take a look on anything else

@rtemperini

Copy link
Copy Markdown
Author

@krisztianfekete I merged main to clear the conflict with the new OTEL env forwarding. Could you take another look when you have a moment? The earlier review comments should still be the open questions.

@krisztianfekete krisztianfekete left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we go deeper on the details though, I think we can get to the same behaviour with a lot less code by using upstream building blocks.

Both runtimes ship a baggage -> spanattribute processor, each taking a filter/predicate, which is basically what MergeCallerContextAttributes + the attributes span processor are doing by hand.

If we then promote allowlisted A2A message.metadata keys into baggage at the executor boundary rather than reading them as a second source with its own precedence rule, we'd have one code path instead of two, and metadata-sourced values would survive subagent hops as well. That would also let us drop most of the policy layer to where OTel already puts it: value truncation is OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT, renaming can be done via a collector transformation, and hashing is really an collector or gw concern.

I'd rather not hardcode am unprefixed set, as to: is operator config, not caller input, so prefixing it doesn't really defend against anything, and right now it makes gen_ai.conversation.id unreachable even though our own ADK executor already sets it.

rtemperini and others added 2 commits September 7, 2026 10:17
Resolve conflicts from the v2 translator move and the rewritten
architecture docs so the allowlist still forwards after main.

Signed-off-by: rtemperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Use the upstream baggage-to-span processors instead of a second
request-scoped merge path, so allowlisted identity reaches every
span including generate_content and survives sub-agent hops.

Signed-off-by: rtemperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
Leftover {hash:...} allowlist entries must fail closed so a raw email cannot land on user.hash. Forward OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT into agent runtimes so the documented truncation control actually applies.

Signed-off-by: rtemperini <[email protected]>
Co-authored-by: Cursor <[email protected]>
@rtemperini

Copy link
Copy Markdown
Author

@krisztianfekete I reworked this around the upstream baggage-to-span processors and promote allowlisted metadata into baggage at the executor, so there is one path. The custom hash/prefix/merge layer is gone; leftover hash mappings fail closed. Could you take another look when you have a moment?

Keep TelemetryConfig, OtelEnvFromProcess, and applyOperatorOnlyEnv
together so compiled OTEL export and the operator-only allowlist both
survive.

Signed-off-by: rtemperini <[email protected]>
@rtemperini

Copy link
Copy Markdown
Author

Hi @krisztianfekete I've rebased and fixed conflicts with the latests main rebase, can you take a look on this when you have a minute, will be good to address this to be able to have the proper context attributes propagation. Thanks in advance

Leftover {hash:...} allowlist entries were dropped before their from
key entered the suppression set, so the executor still stamped the
raw value as a2a.message.metadata.<from>. Keep those sources covered
without re-implementing hashing.

Signed-off-by: rtemperini <[email protected]>
@rtemperini

Copy link
Copy Markdown
Author

Hi @krisztianfekete, I rebased onto the latest main and fixed the conflicts. The new telemetry stack is kept, and the caller context promotion sits on top of it. Can you take a look when you have a minute? Thanks

@rtemperini

rtemperini commented Sep 28, 2026 •

Copy link
Copy Markdown
Author

Hi @EItanya @supreme-gg-gg I rebased this again onto the latest main and fixed the conflicts. The checks that can run from the fork are green, can you take a look when you have a minute, or let me know if anything looks off? Thanks in advance

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale because of no activity in the last 15 days. It will be closed in the next 5 days unless it is tagged "no stalebot" or other activity occurs.

@github-actions github-actions Bot added the stale This issue or PR has become stale label Oct 15, 2026
Comment thread go/adk/pkg/a2a/executor.go Outdated
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Oct 19, 2026
@rtemperini

Copy link
Copy Markdown
Author

Thanks @moezdil, I removed the unrestricted metadata export and added regression coverage. The PR is also updated for current main. Could you and @krisztianfekete take another look?

Let me know if there is anything else to take a look and thanks again for the review

@rtemperini
rtemperini requested a review from moezdil October 19, 2026 09:49
@github-actions github-actions Bot removed the stale This issue or PR has become stale label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants