AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Crop-and-Zoom Tools Can Make AI Vision Models Look Smarter Than They Are

Multimodal AI systems increasingly are built to “think with images,” interleaving reasoning with crop-and-zoom operations to inspect fine details. Yet these tool-using models often gain little over direct inference while spending far more tokens, and sometimes fail questions a plain answer would have gotten right. Researchers from Shanghai AI Lab and Shanghai Jiao Tong University asked whether returned visual evidence actually causes the final answer, or merely accompanies it. Their causal audit, posted as a preprint, finds that aggregate accuracy gains hide an illusion: most tool calls are either disconnected from the image content or scheduled incoherently.

The researchers modeled visual tool-use as a causal graph. In a rollout, a model emits an action, receives an observation, and eventually answers. The intended path runs observation to answer, but an “action-induced shortcut” also exists: the mere act of calling a zoom tool can shift the prediction regardless of what image is returned. To separate these paths, they intervened at three levels. Policy-level: compare tool-enabled versus tool-free inference. Trajectory-level: replace every returned crop with a random same-shape crop during rollout. Step-level: replace a real crop with a counterfactual crop and measure the “Visual Evidence Gain” (VEG), the change in answer preference caused specifically by visual content.

This distinction is not academic. On one V* flag-color question, Mini-o3 answered correctly when run directly; with tools enabled, it repeatedly cropped irrelevant regions and produced a wrong answer or no valid answer.

Across six open-source models and five benchmarks, net effects were uneven. DeepEyes showed no policy-level gain, while Mini-o3 and Qwen3-VL-8B had the largest, such as Qwen3-VL-8B rising from 84.2% to 91.1% accuracy on V* (+6.9 percentage points). Corrupting all returned crops with random same-shape images was devastating for some models—Mini-o3 fell from 87.8% to 23.6% (–64.2 percentage points), often exhausting its tool budget—but nearly harmless for DeepEyes and Thyme.

Step-level analysis exposed two failure modes. In “Calling Without Looking,” returned observations have no causal effect; DeepEyes’ best crop contributed essentially nothing, and 81% of its calls had near-zero VEG. In “Looking Without Planning,” observations are informative but the call schedule is poor: Mini-o3’s evidence value decays with each call, and it keeps zooming after already being confident. The researchers classified every trajectory and decomposed the overall gain. Only the “Calibrated” minority—rollouts where images genuinely shaped the answer—was positive in every model decomposition. For Qwen3-VL-8B, that subset contributed +7.6 percentage points, while other groups offset it to a net +6.9.

The study has limits. Step-level VEG requires white-box access to token probabilities, so closed-source models such as OpenAI’s o3/o4-mini could not be audited. Only crop-and-zoom was tested, not OCR, segmentation, or search. The authors also stress that their “RL-trap” explanation—that outcome-only reinforcement learning rewards tool calls without rewarding genuine perception—is a hypothesis, not a proven mechanism.

Still, the work offers a sharper yardstick: the relevant question is not whether a model calls visual tools, but whether the images it receives causally change what it thinks.