Skip to content

Adapted skills:check and skills:install and added reusable tests - #966

Open
jana-selva wants to merge 11 commits into
mainfrom
feature/963-multi-skill-support
Open

jana-selva wants to merge 11 commits into
mainfrom
feature/963-multi-skill-support

Conversation

@jana-selva

Copy link
Copy Markdown
Contributor

Fixes #963

Checklist

Note: If any of the items in the checklist are not relevant to your PR, just check the box.

For any Pull Request

Is the following correct:

  • the title of the Pull Request?
  • the title of the corresponding issue?
  • there are no other open Pull Requests for the same update/change?
  • that the issue which this Pull Request fixes ("Fixes...") is mentioned?

When Changes Were Made

Did you:

  • update the changelog?
  • update the cookiecutter-template?
  • update the implementation?
  • check coverage and add tests: unit tests and, if relevant, integration tests?
  • update the User Guide & other documentation?
  • resolve any failing CI criteria (incl. Sonar quality gate)?

When Preparing a Release

Have you:

  • thought about version number (major, minor, patch)?
  • checked Exasol packages for updates and resolved open vulnerabilities, if easily possible?

@jana-selva jana-selva changed the title Implement packaged skill support Adapted skills:check and skills:install and add reusable tests Sep 29, 2026
@jana-selva jana-selva changed the title Adapted skills:check and skills:install and add reusable tests Adapted skills:check and skills:install and added reusable tests Sep 29, 2026
@jana-selva
jana-selva deployed to manual-approval September 29, 2026 08:05 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval September 29, 2026 08:05 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval September 29, 2026 08:15 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval September 29, 2026 08:15 — with GitHub Actions Active
Comment thread doc/changes/unreleased.md Outdated
Comment thread doc/changes/unreleased.md Outdated
Comment thread doc/user_guide/features/agent_skills/index.rst
Comment thread doc/user_guide/features/agent_skills/index.rst
Comment thread doc/user_guide/features/agent_skills/index.rst
Comment thread exasol/toolbox/nox/tasks.py
Comment thread test/unit/nox/_skills_test.py Outdated
Comment thread test/unit/util/skill_test.py
Comment thread test/unit/util/skill_test.py
Comment thread test/unit/skills_test.py Outdated
@sonarqubecloud

Copy link
Copy Markdown

❌ The last analysis has failed.

See analysis details on SonarQube Cloud

Comment thread doc/developer_guide/agent_skills.rst
Comment thread test/unit/nox/_documentation_test.py Outdated
Comment thread test/unit/nox/_documentation_test.py Outdated
build.assert_called_once_with(nox_session, config)


def test_build_docs_command(config):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These kinds of tests are require lots of maintenance, if the code is changed.

Comment thread test/unit/nox/_documentation_test.py Outdated
Comment thread test/unit/nox/_documentation_test.py Outdated
Comment thread test/unit/nox/_documentation_test.py Outdated
@ArBridgeman
ArBridgeman force-pushed the feature/963-multi-skill-support branch from 9ec363f to b81a6eb Compare October 5, 2026 11:10
@ArBridgeman

Copy link
Copy Markdown
Collaborator

Quality Gate Failed Quality Gate failed

Failed conditions 75.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

Ah, please add a test for project:check

Comment thread test/unit/skills_test.py
Comment thread test/unit/skills_test.py Outdated
@jana-selva
jana-selva deployed to manual-approval October 7, 2026 08:46 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval October 7, 2026 08:46 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval October 7, 2026 08:53 — with GitHub Actions Active
@jana-selva
jana-selva deployed to manual-approval October 7, 2026 08:53 — with GitHub Actions Active
@jana-selva

Copy link
Copy Markdown
Contributor Author

Quality Gate Failed Quality Gate failed

Failed conditions 75.0% Coverage on New Code (required ≥ 80%)
See analysis details on SonarQube Cloud

Ah, please add a test for project:check

Fixed

@sonarqubecloud

sonarqubecloud Bot commented Oct 7, 2026

Copy link
Copy Markdown

Comment thread test/unit/skills_test.py
def test_ptb_skill_passes_shared_validation():
assert validate_skill(PTB_SKILL_NAME) == ()
@pytest.mark.parametrize("skill_name", _skills_with_eval_cases())
class TestPackagedSkillEvalCases:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PackagedSkillEvalCases.model_validate() already enforces:

  • version == 1
  • non-empty skill, id, category, and prompt
  • non-empty cases
  • non-empty must_include / must_not_include
  • non-blank string values
  • required nested fields
  • forbidden extra fields

So these tests add little value, so I'd remove them:

  • test_cases_have_required_fields
  • test_cases_have_response_constraints
  • eval_cases.cases in test_has_expected_metadata
  • eval_cases.version == 1 in test_has_expected_metadata

Instead, I'd recommend that you test that the Pydantic model behaves as it should, meaning that you verify that it rejects values that don't behave well.

@pytest.mark.parametrize(
    "invalid_case",
    [
        {"version": 2, "skill": "skill", "cases": []},
        {"version": 1, "skill": "", "cases": []},
        {"version": 1, "skill": "skill", "cases": []},
        {
            "version": 1,
            "skill": "skill",
            "cases": [{"id": "x", "category": "x", "prompt": "x"}],
        },
    ],
)
def test_rejects_invalid_eval_cases(invalid_case):
    with pytest.raises(ValidationError):
        PackagedSkillEvalCases.model_validate(invalid_case)

Comment thread test/unit/skills_test.py
@pytest.mark.parametrize("skill_name", _skills_with_eval_cases())
class TestPackagedSkillEvalCases:
@pytest.fixture(scope="module")
def eval_cases_by_skill(self):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to the previous comment, I'd split the fixture from the validation, so that the validation is in a test.

@pytest.mark.parametrize("skill_name", _skills_with_eval_cases())
def test_packaged_skill_eval_cases_are_valid(skill_name):
    PackagedSkillEvalCases.model_validate(_load_eval_cases(skill_name))

This branch was successfully deployed

1 active deployment
manual-approval — d5098b9b Deployed Oct 7, 2026 by jana-selva via Merge Gate / Extension / Approve Running Slow Tests? #2851
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adapt skills:check and skills:install and add reusable tests

2 participants