Skip to content

feat(AIC-3534): add a TypeSafe Jev judge handler - #149

Open
andrewklatzke wants to merge 3 commits into
mainfrom
aklatzke/AIC-3534/jev-handler
Open

andrewklatzke wants to merge 3 commits into
mainfrom
aklatzke/AIC-3534/jev-handler

Conversation

@andrewklatzke

Copy link
Copy Markdown
Contributor

Summary

  • Add launchdarkly-ai-typesafe, a judge handler that sends a variation's classifiers to Jev in one call.
  • Each label becomes its own judge_results entry, tracked on that label's eventKey, with the selected label as the reason.
  • A score is the expected rubric index divided by one less than the number of levels, then clamped to 0–1.

https://launchdarkly.atlassian.net/browse/AIC-3534

Test plan

  • pytest packages/typesafe/tests packages/client/tests/test_judges.py packages/client/tests/test_conversation.py packages/client/tests/test_judge_scoring.py packages/client/tests/test_schema.py
  • Run a TypeSafe judge variation and confirm one result per label, with the label value in response

Made with Cursor

andrewklatzke and others added 2 commits October 7, 2026 14:34
Judge variations can declare classifiers and record one score per label, with the selected label as the result reason.

Co-authored-by: Cursor <[email protected]>
The package is registered with release-please, and this job is what builds and publishes it.

Co-authored-by: Cursor <[email protected]>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 445c38b. Configure here.

Comment thread packages/typesafe/src/launchdarkly_ai_typesafe/handler.py Outdated
Comment thread examples/register.py
Comment thread packages/client/src/launchdarkly_ai_server/judges.py
A wildcard handler was taking the judge, the tracer name skipped evaluation recording, and Jev never received the original question.

Co-authored-by: Cursor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant