AI Papers Reader

Personalized digests of latest AI research

View on GitHub

AI Agents Gain Strategic Vision with 'ToolScope' Framework

Researchers introduce a training-free multimodal agent that overcomes critical limitations in complex visual reasoning by integrating global planning with dynamic, queryable visual memory.

[City, Date]—Large Language Models (LLMs) equipped with vision capabilities often struggle when tackling complex, multi-step tasks like detailed visual question answering (VQA). These Multimodal LLMs (MLLMs) frequently fail due to two core issues: getting lost in localized reasoning traps and forgetting critical visual details as the inference process extends—a problem known as visual context degradation.

To solve this, researchers from Renmin University of China have developed ToolScope, a novel, training-free agentic framework designed to unify high-level strategy with fine-grained multimodal perception. ToolScope acts as an expert planner, allowing MLLMs to approach long-horizon VQA tasks with the precision of a human expert.

ToolScope’s architecture comprises three dedicated components: the Global Navigator, the Agentic Executor, and the Response Synthesizer.

The Global Navigator functions as the agent’s “telescope,” analyzing the input image and question to create a high-level strategic plan and pre-selecting the most appropriate external tools (Search, Code, or Perceive). This top-down planning prevents the model from wasting time on irrelevant steps.

The Agentic Executor then executes this plan iteratively, using external tools to gather information. The key innovation here is the specialized Perceive tool. Unlike standard MLLMs that process an image only once at the start, the Perceive tool treats the image as a “queryable perceptual memory.” This allows the agent to dynamically “zoom in” and formulate visual sub-questions on demand, preventing visual context degradation.

Preventing Visual Forgetting

A crucial example highlighted by the researchers illustrates the power of the Perceive tool. When asked, “How old was the author when the novel shown in the picture was first published?” an unguided MLLM might recognize the book is from the Harry Potter series, search for the general series publication date, and give an incorrect age (32).

ToolScope, however, uses its global planning to recognize the need for specific visual and external knowledge. It first invokes the <perceive> tool to explicitly ask, “What is the name of the novel in the picture?”—confirming it is Harry Potter and the Chamber of Secrets. It then uses the <search> tool to retrieve the exact release date (June 2, 1998) and correctly calculates the author’s age as 33.

The framework is also adept at integrating other tools: if a question requires solving a geometric problem shown in an image, the Navigator selects the Code tool, prompting the Executor to write and run Python code using the Pythagorean theorem for a precise calculation.

In rigorous evaluations across four diverse VQA benchmarks, including ScienceQA and the mathematical reasoning dataset MathVista, ToolScope demonstrated strong generalization capabilities. It achieved an average accuracy improvement of up to +6.69% over existing baselines and maintained superior performance across various MLLM backbones, including Qwen2.5-VL and InternVL3, confirming its robust and scalable design.