Repository navigation
Adapted skills:check and skills:install and added reusable tests - #966
jana-selva wants to merge 11 commits into
Conversation
|
❌ The last analysis has failed. |
| build.assert_called_once_with(nox_session, config) | ||
|
|
||
|
|
||
| def test_build_docs_command(config): |
There was a problem hiding this comment.
These kinds of tests are require lots of maintenance, if the code is changed.
9ec363f to
b81a6eb
Compare
Ah, please add a test for |
Fixed |
|
| def test_ptb_skill_passes_shared_validation(): | ||
| assert validate_skill(PTB_SKILL_NAME) == () | ||
| @pytest.mark.parametrize("skill_name", _skills_with_eval_cases()) | ||
| class TestPackagedSkillEvalCases: |
There was a problem hiding this comment.
PackagedSkillEvalCases.model_validate() already enforces:
version == 1- non-empty
skill,id,category, andprompt - non-empty
cases - non-empty
must_include/must_not_include - non-blank string values
- required nested fields
- forbidden extra fields
So these tests add little value, so I'd remove them:
test_cases_have_required_fieldstest_cases_have_response_constraintseval_cases.casesintest_has_expected_metadataeval_cases.version == 1intest_has_expected_metadata
Instead, I'd recommend that you test that the Pydantic model behaves as it should, meaning that you verify that it rejects values that don't behave well.
@pytest.mark.parametrize(
"invalid_case",
[
{"version": 2, "skill": "skill", "cases": []},
{"version": 1, "skill": "", "cases": []},
{"version": 1, "skill": "skill", "cases": []},
{
"version": 1,
"skill": "skill",
"cases": [{"id": "x", "category": "x", "prompt": "x"}],
},
],
)
def test_rejects_invalid_eval_cases(invalid_case):
with pytest.raises(ValidationError):
PackagedSkillEvalCases.model_validate(invalid_case)| @pytest.mark.parametrize("skill_name", _skills_with_eval_cases()) | ||
| class TestPackagedSkillEvalCases: | ||
| @pytest.fixture(scope="module") | ||
| def eval_cases_by_skill(self): |
There was a problem hiding this comment.
Related to the previous comment, I'd split the fixture from the validation, so that the validation is in a test.
@pytest.mark.parametrize("skill_name", _skills_with_eval_cases())
def test_packaged_skill_eval_cases_are_valid(skill_name):
PackagedSkillEvalCases.model_validate(_load_eval_cases(skill_name))


Fixes #963
Checklist
Note: If any of the items in the checklist are not relevant to your PR, just check the box.
For any Pull Request
Is the following correct:
When Changes Were Made
Did you:
When Preparing a Release
Have you: