Adapted skills:check and skills:install and added reusable tests

doc/api/index.rst

@@ -7,5 +7,6 @@
     :maxdepth: 2
 
     base_config
+    skill_eval
     workflow_exceptions
     workflow_patcher_config

doc/api/skill_eval.rst

@@ -0,0 +1,10 @@
+.. _skill_eval:
+
+Skill Evaluation Cases
+----------------------
+
+These models define the structure of the ``eval_cases.yml`` file packaged with
+an agent skill.
+
+.. automodule:: exasol.toolbox.util.skill_eval
+   :members:

doc/changes/unreleased.md

@@ -5,4 +5,5 @@
 * #940: Added shared validation for packaged agent skills and the `skills:check` Nox session.
 * #938: Added the `skills:install` Nox session for installing the packaged PTB agent skill.
 * #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime
-  behavior
+  behavior
+* #963: Extended packaged skill checks and installation to support multiple skills.

doc/developer_guide/agent_skills.rst

@@ -0,0 +1,69 @@
+.. _developer_agent_skills:
+
+Testing Agent Skills
+====================
+
+Packaged Skills
+---------------
+
+The ``skills:check`` Nox session validates common structure and content rules
+for every packaged skill.
+
+The PTB pytest suite verifies skill packaging and installation with:
+
+* ``test/unit/skills_test.py``: packaging, installation, required files, and
+  ``eval_cases.yml`` validation.
+* ``test/unit/util/skill_test.py``: lower-level validation and
+  installation behavior.
+* ``test/integration/project-template/nox_test.py``: verifies that
+  ``skills:check`` and ``skills:install`` operate in a newly created project.
+* ``test/unit/nox/_skills_test.py``: tests Nox session behavior and failure
+  reporting.
+
+When adding a skill to the PTB:
+
+* Add its ``eval_cases.yml`` with representative prompts and expected response
+  characteristics.
+* Add assertions for any files beyond ``SKILL.md`` that the skill must package
+  and install.
+* Add deterministic, skill-specific assertions where the shared checks are
+  insufficient.
+
+These evaluation files and tests are PTB development resources; they are not
+required by downstream projects using the packaged skills.
+
+Writing ``eval_cases.yml``
+--------------------------
+
+An evaluation case should describe one distinct user goal. Keep the prompt
+specific enough that a good response can be recognized from observable
+evidence, and avoid requirements that merely repeat the skill's name or ask
+for generic quality.
+
+Use ``must_include`` for concrete evidence that should appear in a correct
+response, such as:
+
+* the names of the APIs or files that were inspected;
+* the tools or checks that must be used, when they are part of the skill's
+  intended behavior; and
+* the reported mismatch, affected API, severity, or other required result.
+
+Use ``must_not_include`` for concrete behavior the skill must avoid, such as
+inventing findings, treating one source as authoritative without comparison,
+or proposing implementation changes when the task asks only for an audit.
+The prohibited text should describe an actual failure mode, not a broad word
+that could legitimately occur in a response.
+
+For example, a signature and docstring audit can name specific existing
+functions in its prompt and require evidence from
+``inspect.signature()``, ``inspect.get_annotations()``,
+``typing.get_type_hints()``, and ``inspect.getdoc()``. Its ``must_include``
+values can then require the inspected function names, the introspection tools,
+and the concrete mismatch. This makes the case test the skill's intended
+audit behavior instead of only testing whether the response is generally
+relevant.
+
+Each case should have a unique ``id``, a meaningful ``category``, a focused
+``prompt``, and non-empty ``must_include`` and ``must_not_include`` lists.
+Cases should cover different behaviors rather than restating the same prompt
+with minor wording changes.

doc/developer_guide/developer_guide.rst

@@ -9,3 +9,4 @@
 
     ../design
     plugins
+    agent_skills

doc/user_guide/features/agent_skills/index.rst

@@ -6,6 +6,26 @@ Agent Skills
 The PTB can package agent skills for use by projects and provides shared
 validation for their common structure and content rules.
 
+Packaged skills maintained by the PTB
+-------------------------------------
+
+The following skill directories are provided by the PTB. Installing packaged
+skills can replace a project-local skill with the same name, so choose local
+skill names with this list in mind.
+
+.. list-table::
+   :widths: 30 70
+   :header-rows: 1
+
+   * - Skill directory
+     - Intended use
+   * - ``api-contract-audit``
+     - Audits a Python library for mismatches between type annotations,
+       docstrings, user-facing documentation, and runtime behavior.
+   * - ``exasol-python-toolbox``
+     - Guides agents in using PTB setup, Nox sessions, checks, workflows,
+       updates, releases, and configuration.
+
 Run the validation with:
 
 .. code-block:: shell
@@ -18,19 +38,20 @@ unfinished TODO markers or forbidden repository-specific metadata, and has no
 duplicated Markdown lines. Nox command examples are kept in the skill's
 ``references/nox-sessions.md`` file.
 
-These shared checks are intentionally separate from skill-specific tests. When
-adding a skill, add its expected files and behavior assertions to that skill's
-own test module, while ``skills:check`` covers the rules common to all skills.
+These shared checks are intentionally separate from skill-specific tests. The
+test suite validates the structure of every packaged skill and validates the
+structure of every available ``eval_cases.yml``. When adding a skill, add its
+expected files and behavior assertions to the shared skill test patterns.
 
-Installing the PTB skill
-------------------------
+Installing packaged skills
+---------------------------
 
-Projects can install the PTB skill packaged by their current PTB dependency with:
+Projects can install all skills packaged by their current PTB dependency with:
 
 .. code-block:: shell
 
     poetry run -- nox -s skills:install
 
-The session copies the packaged skill into
-``.agents/skills/exasol-python-toolbox``. Existing files in that skill directory
-are replaced so the installed copy stays aligned with the PTB version.
+The session copies each packaged skill into its own directory below
+``.agents/skills``. Existing files in those skill directories are replaced so
+the installed copies stay aligned with the PTB version.

exasol/toolbox/nox/_skills.py

@@ -22,6 +22,7 @@ def _format_skill_errors(skill_name: str, errors: tuple[str, ...]) -> str:
 def check_skills(session: Session) -> None:
     """Validate the common structure and content rules for packaged skills."""
     failures = {}
+    # Discover skills at runtime so newly packaged skills are installed too.
     for skill_name in get_packaged_skill_names():
         errors = validate_skill(skill_name)
         if errors:
@@ -35,9 +36,12 @@ def check_skills(session: Session) -> None:
 
 
 @nox.session(name="skills:install", python=False)
-def install_ptb_skill(session: Session) -> None:
-    """Install the PTB skill into the project's local agent skill directory."""
+def install_skills(session: Session) -> None:
+    """Install all packaged skills into the project's local agent directory."""
     from noxconfig import PROJECT_CONFIG
 
-    target = install_skill(target_directory=PROJECT_CONFIG.agent_skills_path)
-    session.log(f"Installed {target.name} skill to {target}")
+    for skill_name in get_packaged_skill_names():
+        target = install_skill(
+            skill_name, target_directory=PROJECT_CONFIG.agent_skills_path
+        )
+        session.log(f"Installed {target.name} skill to {target}")

exasol/toolbox/nox/tasks.py

@@ -10,7 +10,7 @@
     "integration_tests",
     "lint",
     "check_skills",
-    "install_ptb_skill",
+    "install_skills",
     "open_docs",
     "prepare_release",
     "type_check",
@@ -61,7 +61,7 @@ def check(session: Session) -> None:
     updated,
 )
 from exasol.toolbox.nox._release import prepare_release
-from exasol.toolbox.nox._skills import check_skills, install_ptb_skill
+from exasol.toolbox.nox._skills import check_skills, install_skills
 from exasol.toolbox.nox._shared import (
     Mode,
     _integration_test_context,

exasol/toolbox/skills/api-contract-audit/SKILL.md

@@ -1,4 +1,5 @@
 ---
+# Generated and maintained by the exasol-toolbox.
 name: api-contract-audit
 description: Audit a Python library's public API for inconsistencies between type annotations, docstrings, user-facing documentation/examples, and actual runtime behavior. Use when reviewing API changes, checking whether public methods accept undocumented parameter shapes, or validating that docs and type hints match enforcement in code.
 ---
@@ -87,4 +88,4 @@ If no findings are discovered, say that explicitly and mention any coverage limi
 
 - Do not rewrite the API contract on your own. If code, docs, and examples disagree, report the disagreement.
 - Do not stop at the first example. Check for the same pattern across sibling APIs.
-- Do not treat private helper inconsistencies as findings unless they affect public behavior.
+- Do not treat private helper inconsistencies as findings unless they affect public behavior.

exasol/toolbox/skills/exasol-python-toolbox/SKILL.md

@@ -1,4 +1,5 @@
 ---
+# Generated and maintained by the exasol-toolbox.
 name: exasol-python-toolbox
 description: Use this skill in Exasol Python projects that use exasol-toolbox/PTB. Use it for PTB setup, nox sessions, code checks, GitHub workflows, updates, releases, and PTB configuration. Use it when an agent must not replace PTB automation.
 ---

exasol/toolbox/skills/exasol-python-toolbox/references/nox-sessions.md

@@ -23,8 +23,8 @@ The sessions below match the PTB version that includes this skill.
 
 | Session | Use | Notes |
 | --- | --- | --- |
-| `skills:check` | Validate packaged PTB skills. | It checks common structure and content rules. |
-| `skills:install` | Install the PTB agent skill. | It updates `.agents/skills/exasol-python-toolbox` from the installed PTB package. |
+| `skills:check` | Validate packaged skills. | It checks common structure and content rules for every packaged skill. |
+| `skills:install` | Install packaged agent skills. | It updates each packaged skill below `.agents/skills` from the installed PTB package. |
 
 ## Test sessions
 

exasol/toolbox/util/skill_eval.py

@@ -0,0 +1,44 @@
+"""Models for validating packaged agent-skill evaluation cases."""
+
+from typing import (
+    Annotated,
+    Literal,
+)
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+)
+
+NonBlankString = Annotated[str, Field(pattern=r"\S")]
+
+
+class ExpectedResponse(BaseModel):
+    """Required and forbidden content for one evaluation response."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    must_include: list[NonBlankString] = Field(min_length=1)
+    must_not_include: list[NonBlankString] = Field(min_length=1)
+
+
+class EvalCase(BaseModel):
+    """One prompt and its expected response constraints."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    id: NonBlankString
+    category: NonBlankString
+    prompt: NonBlankString
+    expected: ExpectedResponse
+
+
+class PackagedSkillEvalCases(BaseModel):
+    """Schema for a packaged skill's ``eval_cases.yml`` file."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    version: Literal[1]
+    skill: NonBlankString
+    cases: list[EvalCase] = Field(min_length=1)

exasol/toolbox/util/skills.py

@@ -61,7 +61,7 @@ def get_packaged_skill_names() -> tuple[str, ...]:
         )
     except (FileNotFoundError, ModuleNotFoundError) as error:
         raise RuntimeError(
-            "Packaged PTB skills are unavailable. Reinstall exasol-toolbox "
+            "Packaged skills are unavailable. Reinstall exasol-toolbox "
             "with its package resources."
         ) from error
 

test/integration/project-template/nox_test.py

@@ -1,5 +1,47 @@
 from exasol.toolbox.util.version import Version
 
+EXPECTED_PACKAGED_SKILL_FILE_COUNTS = {
+    "api-contract-audit": 1,
+    "exasol-python-toolbox": 5,
+}
+EXPECTED_NOX_SESSIONS = {
+    "format:fix",
+    "format:check",
+    "project:check",
+    "test:unit",
+    "test:integration",
+    "test:coverage",
+    "lint:code",
+    "lint:typing",
+    "lint:security",
+    "lint:dependencies",
+    "docs:multiversion",
+    "docs:build",
+    "docs:open",
+    "docs:clean",
+    "links:list",
+    "links:check",
+    "changelog:updated",
+    "release:prepare",
+    "release:update",
+    "release:trigger",
+    "skills:check",
+    "skills:install",
+    "matrix:generate",
+    "artifacts:validate",
+    "artifacts:copy",
+    "sonar:check",
+    "dependency:licenses",
+    "dependency:audit",
+    "vulnerabilities:update",
+    "vulnerabilities:resolved",
+    "dependency:sbom",
+    "package:check",
+    "workflow:check",
+    "workflow:generate",
+    "workflow:audit",
+}
+
 
 class TestSpecificNoxTasks:
     """
@@ -84,3 +126,37 @@ def test_install_github_workflows(self, poetry_path, run_command):
 
         file_list = run_command(["ls", ".github/workflows"]).stdout.splitlines()
         assert len(file_list) == 14
+
+    def test_skills_install_and_check(self, poetry_path, run_command, new_project):
+        skills_install = self._command(poetry_path, "skills:install")
+        run_command(skills_install)
+
+        installed_skills = new_project / ".agents" / "skills"
+        installed_skill_names = {
+            path.name for path in installed_skills.iterdir() if path.is_dir()
+        }
+        assert set(EXPECTED_PACKAGED_SKILL_FILE_COUNTS) <= installed_skill_names
+
+        for (
+            skill_name,
+            expected_file_count,
+        ) in EXPECTED_PACKAGED_SKILL_FILE_COUNTS.items():
+            installed_file_count = sum(
+                path.is_file() for path in (installed_skills / skill_name).rglob("*")
+            )
+            assert installed_file_count == expected_file_count
+
+        skills_check = self._command(poetry_path, "skills:check")
+        output = run_command(skills_check)
+
+        assert output.returncode == 0
+
+    def test_exposed_nox_sessions(self, poetry_path, run_command):
+        output = run_command([poetry_path, "run", "--", "nox", "-l"])
+        sessions = {
+            line[2:].split(" ->", maxsplit=1)[0]
+            for line in output.stdout.splitlines()
+            if line.startswith(("* ", "- "))
+        }
+
+        assert sessions == EXPECTED_NOX_SESSIONS