Out in the Wild: New Benchmark Exposes the Evolving Limits of AI Terminal Agents
AI agents are rapidly moving from simple code generators to autonomous system administrators, capable of executing complex instructions directly inside a computer’s terminal. Yet, testing these digital assistants has historically relied on expert-crafted, puzzle-like scenarios that rarely match the messy reality of day-to-day software development.
To bridge this gap, a team of researchers from University College London, Nanjing University, and Tencent has introduced TERMINALWORLD, an automated data engine that mines “in-the-wild” human terminal recordings to build a highly realistic benchmark. By reverse-engineering 80,870 public recordings from the developer sharing platform asciinema, the team generated 1,530 validated tasks spanning 18 categories, including container orchestration, database operations, and system administration.
To understand how TERMINALWORLD works, consider a developer who recorded themselves securing a server. The raw recording might contain typos, retries, and confusing system outputs. TERMINALWORLD’s engine uses a Large Language Model (LLM) to extract the core intent—for example: “Block all IPs with over 10 failed SSH logins in auth.log, and save them to /app/result.txt.”
The engine then uses an AI agent to build a custom Docker container, replaying a cleaned-up version of the human’s commands to verify they work. Finally, it dynamically tests the environment by comparing the system’s state before and after the commands run, ensuring that any future agent being tested is graded on whether it actually achieves the end goal, rather than just copying the human’s steps.
When the researchers tested top-tier AI models (including Claude 4.7, GPT-5.5, and Gemini 3.1 Pro) on a verified subset of 200 tasks, the results revealed that real-world operations remain a significant hurdle. Even the best-performing model, Claude Opus 4.7, successfully resolved only 62.5% of the tasks.
The evaluations exposed what researchers called an “efficiency paradox.” When human developers get stuck in a terminal, they typically stop to rethink. In contrast, failing AI agents went on expensive, compute-heavy wild goose chases—consuming 3.3 times more tokens and 1.4 times more time on failed attempts than successful ones. Lacking reliable planning and stopping criteria, the agents endlessly brute-forced commands in the open-ended terminal environment.
Interestingly, when the agents did succeed, they rarely mimicked humans. The command-set overlap between agents and humans was just 21.4%. For example, in a network packet analysis task where the original human used the network tool ettercap to extract credentials, the AI agent instead chose to parse the packet directly using a custom Python script with tshark. Because TERMINALWORLD evaluates the final state of the file system rather than the exact commands typed, it successfully recognized the agent’s unique path as a valid solution.
Ultimately, the study revealed that traditional, human-curated benchmarks are poor predictors of real-world capability. As terminal tools and developer practices shift, TERMINALWORLD’s automated, scalable pipeline offers a living testbed that can evolve alongside the very technologies these AI agents are being trained to manage.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.