SkillGym is an automatic pipeline that turns community-written agent skills into executable, verifiable training tasks. A builder-reviewer agent system constructs skill-critical tasks across four reasoning structures (procedural, abductive, constraint satisfaction, and partial order), each with a reference solution and an executable verifier. Finetuning on trajectories collected from these tasks improves how LLM agents use skills, across model families and sizes.
All models are finetuned with supervised finetuning (SFT) on SkillGym trajectories for 2 epochs.
| Model | Base model | Size | Link |
|---|---|---|---|
| SkillGym-MiniCPM5-2B | MiniCPM5 | 2B | Not released yet |
| SkillGym-Qwen3.5-4B | Qwen3.5 | 4B | reasonwang/SkillGym-Qwen3.5-4B |
| SkillGym-Ministral-3-8B | Ministral-3 | 8B | reasonwang/SkillGym-Ministral-3-8B |
| SkillGym-Qwen3.5-9B | Qwen3.5 | 9B | reasonwang/SkillGym-Qwen3.5-9B |
| SkillGym-Qwen3.5-27B | Qwen3.5 | 27B | reasonwang/SkillGym-Qwen3.5-27B |
| SkillGym-Qwen3.5-122B-A10B | Qwen3.5 | 122B (10B active) | reasonwang/SkillGym-Qwen3.5-122B-A10B |
The SkillGym test set contains 400 tasks: 200 on skills seen during training (held-in) and 200 on skills excluded from training (held-out), covering all four reasoning structures. Each task ships with an instruction, an initial workspace, a container environment, the source skill, and an executable verifier.
- Download: reasonwang/skillgym-test
- Tasks are in Harbor task format, organized as
tasks/<profile>/<held_in|held_out>/<task>/, with 50 held-in and 50 held-out tasks per profile.manifest.jsonllists every task with its profile, split, skill, and domain. See the dataset card for the layout of a task package.
The SFT data contains 19,053 successful agent trajectories on SkillGym tasks, one JSONL file per agent harness (AgentFly, mini-swe-agent, and Terminus-2).
- Download: reasonwang/skillgym-sft
Scores are on a 0-100 scale, higher is better. SkillGym is task success on our test set. SkillEval and SkillsBench report the mean and standard deviation over three runs. Skill-Use-Bench reports the skill-use (SU) score. All models are evaluated with skills available.
| Model | Size | SkillGym | SkillEval | SkillsBench | Skill-Use-Bench |
|---|---|---|---|---|---|
| MiniCPM5 | 2B | 33.2 | 63.0 ± 0.6 | 10.8 ± 2.7 | 24.1 |
| + SkillGym SFT | 2B | 33.2 | 68.1 ± 2.0 | 6.1 ± 0.8 | 44.7 |
| Ministral-3 | 8B | 17.5 | 55.7 ± 3.1 | 3.9 ± 2.7 | 4.3 |
| + SkillGym SFT | 8B | 49.5 | 77.5 ± 0.4 | 16.9 ± 1.3 | 55.1 |
| Qwen3.5 | 4B | 33.8 | 62.8 ± 1.5 | 10.1 ± 1.3 | 8.8 |
| + SkillGym SFT | 4B | 47.0 | 70.8 ± 1.6 | 14.3 ± 1.0 | 48.7 |
| Qwen3.5 | 9B | 41.3 | 65.7 ± 1.1 | 14.8 ± 3.5 | 14.8 |
| + SkillGym SFT | 9B | 59.5 | 74.7 ± 1.2 | 22.4 ± 4.2 | 49.6 |
| Qwen3.5 | 27B | 55.3 | 76.2 ± 1.1 | 32.6 ± 2.3 | 28.3 |
| + SkillGym SFT | 27B | 62.8 | 80.1 ± 1.5 | 47.4 ± 5.1 | 74.2 |
| Qwen3.5 | 122B/10B | 53.0 | 69.8 ± 1.1 | 30.1 ± 2.5 | 16.8 |
| + SkillGym SFT | 122B/10B | 65.0 | 80.0 ± 0.3 | 53.6 ± 5.2 | 72.2 |
See the paper for reference models, held-out results, and analysis.
@article{wang2026skillgym,
title = {{SkillGym}: Training Skill-Use Agents with Automatic Verifiable Environment Generation},
author = {Wang, Renxi and Hee, Mingshan and Koto, Fajri and Baldwin, Timothy and Li, Haonan},
journal = {arXiv preprint arXiv:2609.37539},
year = {2026}
}Questions and issues are welcome through GitHub issues, or by email to [email protected].
