AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Reinforcement Learning Turns a 122B Model Into a More Reliable Terminal Agent

Long-horizon terminal tasks require AI agents to issue shell commands, read output, recover from failures and keep working over hundreds of turns. Training such agents with reinforcement learning is hard because the systems that generate and train on trajectories can disagree about tokens or expert routing, while a failed task can yield little reward. A Tencent-led team describes T1, a 122-billion-parameter mixture-of-experts model with 10 billion active parameters, post-trained from Qwen3.5-122B-A10B to operate a real shell in a cloud sandbox for up to 300-plus tool-call turns per task.

T1 is trained with PPO, a reinforcement-learning algorithm, rewarded by each task’s execution verifier. To align training with generation, the authors introduce TITO, or token-in, token-out: the trainer consumes the exact token identifiers the sampler emitted, repairs turn boundaries under auditable cases, and masks tool output so it provides context without gradient. A second mechanism, rollout routing replay, records which experts the sampler selected at every MoE layer and replays those selections during training. Together, the researchers report, these cut the training-to-inference log-probability gap from 0.021 to 0.013, with zero token drift in the loss region.

The reward is dense. Instead of scoring only solved or unsolved, T1 counts the absolute number of passing assertions on a fixed scale, S=20. Passing 10 assertions on a hard task earns 0.5, while passing 2 of 4 on an easy task earns 0.1, preserving partial progress. A warm-started critic helps convert that signal into credit across a long trajectory.

A case study illustrates why execution rewards matter. In polyglot-c-py, both T1 and its supervised fine-tuning checkpoint write a correct Python/C polyglot that prints Fibonacci numbers. But the verifier requires /app/polyglot to contain exactly main.py.c. The SFT model compiles a test binary, cmain, and leaves it behind, failing with [‘main.py.c’, ‘cmain’]. T1 runs the same test, deletes the binary, and passes. The difference is one cleanup command—but it reflects whether the agent treats the final machine state as part of the deliverable.

On Terminal-Bench 2.1, T1 reaches 64.0% resolved, up from 49.4% for the SFT checkpoint and 43.8% for the base model. Reinforcement learning adds 14.6 percentage points over SFT. Under the same harness, the authors report, this is above GPT-5.4 (54.8%) and DeepSeek-V4-Flash (56.9%), and close to Claude Opus 4.7 (66.1%). The authors caution that harness choices move scores and T1 was trained for this harness. On Long-Horizon Terminal Bench, T1 scores 27.9 average reward versus 23.6 for SFT and 18.9 for base; on Terminal-Bench Hard, it resolves 38.0% versus 28.3% and 20.0%.

The gains come with costs. T1 uses 94.4 turns on average in one evaluation, compared with 31.5 for SFT and 41.1 for base, and case studies show unsolved tasks can spend hundreds of turns before timing out. Oversampling cancels the slowest trials, which may drop the hardest tasks. Improvements are uneven, with the Hard difficulty group still at about 33%. The study covers one model family, one agent harness and one benchmark family, and its dense-reward comparison is campaign-level rather than a single-axis ablation. The work suggests execution-based reinforcement learning can improve terminal agency when token and routing fidelity are controlled—not that it solves autonomous software engineering.