Skip to content

FIX: Match get_scores score_category against each stored category - #3622

Open
Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
microsoft:mainfrom
bIackr0se:fix/get-scores-category-filter
Open

Jaafer Rahmani (bIackr0se) wants to merge 1 commit into
microsoft:mainfrom
bIackr0se:fix/get-scores-category-filter

Conversation

@bIackr0se

@bIackr0se Jaafer Rahmani (bIackr0se) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

memory.get_scores_async(score_category="refusal") (and the deprecated get_scores) never returns a score. Score.score_category is a list, and ScoreEntry stores it in a JSON column, but _execute_get_scores compared the whole column with the argument (ScoreEntry.score_category == score_category). A stored ["refusal"] is never equal to "refusal", so every category query by name comes back empty (reproduced on SQLite; Azure SQL builds the same whole-column comparison). Only an argument equal to the whole stored value matched, such as the complete list in its stored order.

This PR builds the condition with _add_list_conditions(..., exact=True), the helper behind the seed list filters (harm_categories, authors, groups), in its whole-element mode. A query now returns the scores whose category list contains the requested name as a whole element, compared case-insensitively as that mode does: "hate" matches ["hate"] and ["violence", "hate"], but not ["hate_speech"]. A list argument keeps working and returns the scores that list every given category, in any order. The parameter type is now str | Sequence[str] | None, and the docstrings say how it matches.

Real run: two SubStringScorers score two stored assistant replies in IN_MEMORY memory, then the scores are read back:

                                                             main                          this PR
get_scores_async(score_category="refusal")                   []                            [['refusal']]
get_scores_async(score_category="privacy")                   []                            [['credentials', 'privacy']]
get_scores_async(score_category="unused")                    []                            []
get_scores_async(score_category=["privacy", "credentials"])  []                            [['credentials', 'privacy']]
get_scores_async(score_category=["credentials", "privacy"])  [['credentials', 'privacy']]  [['credentials', 'privacy']]

Tests and Documentation

Added test_get_scores_by_score_category to tests/unit/memory/memory_interface/test_interface_scores.py. It stores scores with ["hate"], ["violence", "hate"], ["hate_speech"], ["HaTe"], None and [] and checks whole-element matching, case folding, "speech" matching nothing, stored empty and None categories not matching the queried category names, a list argument, and the intersection with score_type. It fails on main (the first query returns []).

pytest tests/unit/memory: 1923 passed, 1 skipped. ruff check, ruff format --check (0.17.1) and ty check on the changed files pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant