AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A Harness Made to Order: New System Generates an AI Agent’s Scaffolding on the Fly

An AI agent’s usefulness depends on more than its underlying model. The “harness” wrapped around that model—the code that decides what the agent remembers, how it plans, which tools it calls, and when to stop—can be just as important. Today those harnesses are usually hand-built for particular tasks, a process that does not scale. In a new paper, researchers at LV-NUS Lab present JIT-Agent, a model trained to write a task-specific harness on the fly for an arbitrary off-the-shelf large language model.

The central idea is to treat harness design as a learnable skill rather than an engineering chore. The researchers define a harness as four interchangeable modules—memory, planning, action, and capability orchestration—governed by a fixed protocol with validation rules. At inference, JIT-Agent receives the task, the protocol, a registry of available tools, and a few previously built harnesses, then emits executable code that an existing model (the “backbone”) follows. Training happens in three stages: imitation of teacher-written harnesses, supervised repair of broken generations, and a reinforcement-learning stage called Evo-GDPO that rewards candidate harnesses for outperforming the current archive on reward, latency, and cost. The generator itself is a 27-billion-parameter model built from Qwen3.6-27B.

Consider a travel request specifying dates, passengers, hotel requirements, meals, and daily routes. JIT-Agent generated a harness called Turnstile: a TravelPlanning module compiles the request into a typed checklist, a DataStoreMemory module assigns each required fact its own evidence bucket, and a DynamicToolPolicy module exposes search tools for missing buckets but withholds the final-answer tool until an is_complete() check succeeds. The harness enforces evidence collection before itinerary synthesis, rather than letting the model improvise.

Across nine benchmarks spanning deep research, daily work, planning, and office tasks, the researchers report consistent gains. JIT-Agent’s harness raised GLM-5.2’s nine-benchmark average from 74.1 to 81.8—a gain of 7.7 percentage points—and DeepSeek-V4-Flash’s average from 66.7 to 75.5, up 8.8 points. Equipped with a JIT harness, DeepSeek-V4-Flash scored 9.1 points above GPT-5.6 on DeepSearchQA, 8.7 points above it on PinchBench, and 4.3 points above it on OdysseyBench. The largest single gain was 20.2 percentage points for GLM-5.2 on DeepPlanning-Travel. In controlled tests holding the backbone fixed, JIT-generated harnesses had the lowest token use and API cost in all six settings, cutting per-case cost by 14.9–54.1% relative to the cheapest fixed harness, with an average reduction of 36.0%. On xBench-DS with DeepSeek-V4-Flash, token use fell from 527,000 to 212,000 and cost from $0.075 to $0.039 while the score rose from 78.0 to 82.0.

The gains were not universal. With DeepSeek-V4-Flash on AgentIF, JIT-Agent scored 63.8 versus Claude Code’s 66.9; with Qwen3.6-Flash on DeepSearchQA, it scored 70.3 versus NanoBot’s 74.2, though it used far fewer tokens in both cases. The authors also note that their four-module protocol is deliberately compact compared with production runtimes, and future systems may keep a stable core while generating selected components.

The broader significance, as the authors frame it, is that “harness intelligence” may be a trainable, transferable axis of agent improvement—one that can compound with model scaling. The results do not show that scaffolding can substitute for model quality; they suggest that automatically matching the scaffold to the task deserves attention alongside the model itself.