Skip to content

fix(tracing): stop set_tracing_processor_configs deadlocking on its own lock - #543

Open
michaelxu2288 wants to merge 2 commits into
scaleapi:mainfrom
michaelxu2288:fix/tracing-config-deadlock
Open

michaelxu2288 wants to merge 2 commits into
scaleapi:mainfrom
michaelxu2288:fix/tracing-config-deadlock

Conversation

@michaelxu2288

@michaelxu2288 michaelxu2288 commented Oct 1, 2026 •

Copy link
Copy Markdown

Problem

set_tracing_processor_configs() (exported from agentex.lib.core.tracing.tracing_processor_manager) never returns. TracingProcessorManager.set_processor_configs takes self.lock and then calls add_processor_config for each config, which takes the same lock again. self.lock is a threading.Lock, which is not reentrant, so the first config blocks forever:

from agentex.lib.core.tracing.tracing_processor_manager import set_tracing_processor_configs
set_tracing_processor_configs([SGPTracingProcessorConfig(...)])  # hangs

Called at ACP startup, the server never finishes booting; called before a worker starts, it never reaches worker.run(). Nothing inside the SDK calls the plural API today, which is how it went unnoticed, but it is exported next to add_tracing_processor_config.

Fix

self.lock becomes a threading.RLock. The batch still registers under one lock acquisition and the nested add_processor_config can re-enter it. add_processor_config itself is unchanged. Two-line diff.

set_processor_configs appends to the registered processors, the same as calling add_processor_config once per config. I kept that behaviour rather than making it replace the existing list; happy to change that if "set" was meant literally.

Verification

  • New tests/lib/core/tracing/test_tracing_processor_manager.py registers two configs from a thread with a 5 s join.
    • Before: AssertionError: set_processor_configs blocked on the manager's own lock.
    • After: passes, and both the sync and the async processor lists hold one processor per config.
  • uv run pytest -n 0 tests/lib/core/tracing/test_tracing_processor_manager.py tests/lib/core/tracing/test_span_queue.py tests/lib/core/tracing/processors: 89 passed.
  • ruff check and pyright clean on both files.

RetriggerConfidence Score: 4/5

The deadlock fix looks sound, but the test deadline still needs to meet the repository’s rule before merging.

Fix All in CursorFindings

  1. P2 Test deadline has no name ▶
Fix with agent prompt
### Issue 1
tests/lib/core/tracing/test_tracing_processor_manager.py:undefined-27
`_finishes` uses `5.0` as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The tracing processor manager now uses a lock that its own thread can enter again, so batch registration can finish. New tests check batch registration and the single-config path.

  • Batch registration adds matching sync and async processors for each config.
  • A timed thread check confirms the batch call completes.

Reviews (2) · Last reviewed commit: "test(tracing): name the registration dea..."

…wn lock

TracingProcessorManager.set_processor_configs takes self.lock and then
calls add_processor_config for each config, which takes the same lock
again. self.lock was a threading.Lock, which is not reentrant, so the
first call to the exported set_tracing_processor_configs() blocked
forever: on an ACP server it never finishes startup, on a worker it never
reaches worker.run().

Use an RLock so the batch still registers under one lock acquisition and
the nested add can re-enter it. add_processor_config is unchanged.

The new test registers two configs from a thread and fails on the old
lock (the thread is still blocked after 5 s); it passes with the RLock.
return manager


def _finishes(target: Any, timeout: float = 5.0) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test deadline has no name

_finishes uses 5.0 as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/lib/core/tracing/test_tracing_processor_manager.py
Line: 27

Comment:
**Test deadline has no name**

`_finishes` uses `5.0` as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/scale-ai/-/custom-context?memory=002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in ab7f15e: the deadline is REGISTRATION_DEADLINE_SECONDS, and an exception from the registration thread is re-raised after the join.

Comment thread tests/lib/core/tracing/test_tracing_processor_manager.py
Store the 5 s join deadline as REGISTRATION_DEADLINE_SECONDS, and re-raise
an exception from the registration thread after the join so a failing
registration reports its own error instead of a later list assertion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant