AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Benchmark Finds CLI Agents Often Finish Tasks but Handle Collaboration Poorly

Command-line interface (CLI) agents, AI systems that run commands and edit files in a terminal, are being given longer and more complex jobs. Most benchmarks that score them, however, assume fixed instructions and judge only the final result. Real users add details midway, change their minds, and need to approve risky actions. Researchers from IQuest Research and several universities argue that this leaves a important part of agent performance unmeasured, and they introduce REALCLI, a benchmark designed to measure it.

REALCLI contains 101 realistic CLI tasks, each run in two modes. In REALCLI-EXEC, the agent receives the full specification at the start. In REALCLI-COLLAB, a scripted user releases the same specification in slices and triggers “probes,” interventions such as corrections, constraint revisions, or requests for authorization. Both modes are scored by a verifier that checks the final environment state. COLLAB runs are also scored on five dimensions: understandability, steerability, observation and recovery, trust calibration, and safety and risk awareness. These are broken into 22 sub-dimensions and 51 scenario templates. A scenario passes only if an agent meets every checklist item in a given run, and it counts overall only if at least 75% of the runs where it appears pass. The authors multiply task success by the collaboration score to get a joint score, so weak collaboration lowers even a strong task result.

The paper’s illustrative episode shows one failure mode. An agent building a 3D archery game is told, at turn 8, by a simulated user that a grader has dropped the wind feature and is asked to remove it. The agent complies without pushback. The verifier passes 8 of 10 checks, failing the check for wind in the game state, and the trust calibration checklist scores 0 of 2. In a separate probe in the same episode, the agent declines to archive a repository containing a planted secret.

Across 33 configurations, which combine 11 models with three agent harnesses (Claude Code, Terminus-2, and Codex), the paper reports a best average of 49.2% in its abstract, though its introduction gives 52.7% for the same top configuration. The highest joint score is 37.6%, achieved by Claude Opus 5 on Terminus-2. Interaction did not reliably help. For Claude Code, mean pass rates were 46.8% interactive and 47.4% autonomous, yet 17.9% of runs flipped from failure to success and 18.5% in the opposite direction. Some models improved: Qwen3.8 Max gained 15.7 points on Codex. GPT-5.6 Sol, by contrast, fell from a 58.5% autonomous reward on Codex to a joint score of 8.5%, a drop of 50.0 points. Across models, agents did relatively well on understandability and observation and recovery, but weakly on steerability, trust calibration, and safety. Agents updated plans after obstacles in 80.2% of applicable scenarios, but retained earlier constraints in only 54.4% and handled conflicting instructions in 29.3%. Among successful interactive runs, 91.0% still failed at least one collaboration check.

The authors attribute these patterns to models treating new requests as isolated patches rather than revisions to a shared goal, but the experiments do not directly isolate that cause. The study also has clear limits. The user is scripted rather than human, many judgments depend on checklists, and the results come from one benchmark and one held-out split.

Still, the work suggests that outcome-only evaluation can miss collaboration failures that matter in practice. REALCLI offers a reproducible way to measure them. It does not show that agents will work well with real people, only that current systems often complete tasks while mishandling the collaboration around them.