AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Agent Skills, Custom-Fitted to Small AI Models, Aim to Cut Cost of Reliable Automation

Agent skills are text packages that encode procedural workflows, tool-use conventions, and output contracts for AI agents. They make assistants more reliable, but the strongest models that generate them are closed-source and expensive; skills written for those models often overwhelm small open-source models, causing hallucinated arguments, skipped verification, or failure inside an agent loop. Researchers from Shanghai Jiao Tong University, Shanghai AI Laboratory, Harbin Institute of Technology, and Tsinghua introduce SKILLER, a natural-language-driven reinforcement learning framework that rewrites skills for compact models without changing the models’ weights.

SKILLER treats the skill text itself as the policy to optimize. A frontier model, GPT-5.4, acts as critic and actor, while the target model—Qwen3.5-9B or Qwen3.5-4B—runs in an agent harness as the environment. The benchmark’s official verifier supplies rewards, usually task success or test pass rate. After each attempt, the critic compares the small model’s trajectory with a reference successful trajectory and verifier diagnostics, locates the earliest causal error, and suggests bounded edits. The actor applies them with insert, replace, create, or delete operations. A replay memory stores failure signatures and accepted edits. Feedback flows as structured natural language; no neural weights are updated.

An example from springboot-tdd in SWE-Skills-Bench shows the mechanism. The first skill version was a generic seven-phase TDD workflow. Later versions added a root directory anchor, a limited exploration budget, and requirements to verify exact database targets, method signatures, imports, and cross-file dependencies before editing. The authors explain that a locally plausible Java class can still fail when its imports, repository methods, SQL profile, or tool call does not match the surrounding project. The evolved skill converted vague advice into interface-level preconditions.

Across five benchmarks—SkillsBench, SkillLearnBench, SWE-Skills-Bench, GAIA, and EarthBench—SKILLER outperformed three open-source baselines (AutoSkill, EvoSkill, SkillX) and the closed-source Manus skill system on most comparisons. For Qwen3.5-9B, the authors report absolute gains of 4.3 to 20.4 percentage points over the leading baseline on benchmarks where SKILLER led; for Qwen3.5-4B, gains reached 13.3 percentage points where it led. On SWE-Skills-Bench, the 4B model with SKILLER reached a 66.7% pass rate, surpassing the 9B model using Manus skills (62.4%) or human-authored skills (52.0%). On SkillsBench single-skill tasks, the 9B model scored 73.91%, versus 60.87% for SkillX and 57.97% for Manus.

The approach carries costs and limits. Skill generation uses GPT-5.4 offline; the authors report $8.95 for Qwen3.5-9B across the five benchmarks, more than AutoSkill’s $2.53 or EvoSkill’s $1.95, but less than SkillX’s $14.55, with the highest average score. Results were not uniformly best: on SkillsBench with the 4B model, SkillX scored 43.48% to SKILLER’s 42.03%. The authors also note that procedural constraints cannot substitute for missing factual knowledge or complex mathematical reasoning in multi-hop retrieval.

The broader significance is economic. If small models can run reliably under carefully matched skills, organizations may deploy agents on consumer-grade hardware instead of renting expensive frontier APIs. The paper does not claim SKILLER closes the capability gap everywhere; it argues that for structured, repetitive workflows, tightly constrained behavioral policies can be worth more than extra parameter scale.