AI Papers Reader

Personalized digests of latest AI research

View on GitHub

The Coordination Cure: How Teaching AI When to Look and When to Think Fixes Visual Reasoning

Imagine trying to solve a complex geometry problem. Your eyes must dance back and forth: first, you look at a diagram to identify a triangle; next, you look down at your notepad to write out an equation; then, you look back at the diagram to locate the next angle. If you try to write the math without remembering your last step, or if you guess an angle’s value without actually looking at the shape, your proof falls apart.

Multimodal Large Language Models (MLLMs)—AI systems designed to “see” images and “read” text—suffer from exactly this kind of coordination breakdown. When tackling multi-step visual reasoning, these systems frequently fail to alternate between extracting visual evidence and synthesizing textual logic. This leads to frustrating errors: an AI might hallucinate that an angle is 80 degrees because it did not look closely at the diagram, or it might write a mathematically incoherent sentence that contradicts its own reasoning history.

To solve this, a research team from the University of Trento, BAAI, Singapore Management University, and IQuest Research has introduced DyCo-RL (Dynamic Coordination Reinforcement Learning). Rather than just grading the AI on its final answer, this new training framework acts as a real-time tutor, teaching the AI exactly when to look at an image and when to focus on its own written thoughts.

Existing training methods treat all parts of an AI’s generated response equally, ignoring the fact that different words (or “tokens”) serve different purposes. DyCo-RL solves this by first identifying the “functional role” of each token during the reasoning process.

To do this, the framework measures how the model shifts its attention from one step to the next using a geometric metric called the Fisher–Rao geodesic distance. If the model suddenly restructures its focus on the image, DyCo-RL flags that token as “visually-oriented” (a “looker”). If it reorganizes its focus on the preceding text, the token is deemed “text-oriented” (a “thinker”).

Once these roles are assigned, DyCo-RL evaluates how well the AI’s actual attention allocation matches its designated role. If a “looker” token failed to sufficiently examine the image, or if a “thinker” token ignored the previous math steps, the system attenuates the learning reward during optimization. Conversely, when the AI coordinates its focus perfectly, DyCo-RL amplifies the positive reinforcement signal.

This elegant, plug-and-play module was tested on state-of-the-art open models, including Qwen2.5-VL (3B and 7B variants). Across seven rigorous benchmarks spanning complex mathematics and visual understanding, DyCo-RL consistently boosted reasoning accuracy. Crucially, the researchers found that their method successfully broke the rigid “perceive first, reason later” pipeline of traditional AIs, allowing the models to dynamically re-ground their logic in the visual details throughout the entire problem-solving chain.

By transforming cross-modal coordination from a byproduct of training into an explicit learning objective, DyCo-RL paves the way for a new generation of reliable, hallucination-resistant AI assistants capable of genuinely thinking through what they see.