AI Papers Reader

Personalized digests of latest AI research

View on GitHub

For CT Spatial Questions, a Modular Agent Outperforms End-to-End Vision-Language Models

Vision-language models (VLMs), which process images and text together, have shown promise in radiology, but spatial reasoning remains a weak spot. That matters because clinical reports depend on statements such as “the mass is left of the aorta” or “the lesion lies above the liver.” Axial CT slices — cross-sectional images of the body — make the challenge concrete: on the MIRP RQ1 benchmark, which asks binary questions about pairwise spatial relations between anatomical structures, direct VLM prompting has hovered near chance. In a new workshop paper, researchers at Ulm University and the Technical University of Munich propose a modular medical agent that replaces end-to-end guesswork with explicit, auditable steps: parse the question, locate the anatomical structures, then verify the spatial relation using deterministic geometry.

The system keeps a familiar conversational interface, and both it and the direct baselines receive the same prompt: an image, a question, and an instruction to answer 1 for true or 0 for false. When asked, “Is the liver left of the spleen?” the VLM-based controller routes the question to a specialized pathway, where a parser converts it into a structured tuple—(liver, left of, spleen). A YOLO-based object detector—a real-time object-detection model—trained on 61 anatomical classes, returns the highest-confidence bounding box for each organ. A deterministic module then compares the horizontal or vertical ordering of the boxes’ center points to decide whether the relation is true. Crucially, no neural network directly predicts the final yes/no answer.

On the MIRP RQ1 benchmark’s held-out test set of 938 image-question pairs, the best hybrid configuration, Qwen2-VL 7b paired with the agent, reached 94.1% accuracy and 94.2% F1—a 42.5 percentage-point improvement over the same model prompted directly, which scored 51.6%. F1 is a standard measure that balances precision and recall. MedGemma 4b plus the agent reached 91.6% accuracy and 91.3 F1, versus 51.8% for direct MedGemma. Both direct baselines were close to chance, in line with earlier leaderboard results. The authors attribute the hybrid’s advantage to explicit spatial verification rather than a stronger language model.

The modular design also makes failures explainable. Among the 55 incorrect answers from the best hybrid, the authors assigned 50.9% to imprecise localization by the YOLO detector, 27.3% to parsing or query-extraction errors, and 21.8% to missing detections. In a mislocalization failure, a small error in an organ’s detected center can flip an above/below or left/right judgment. No errors were attributed to routing, mapping organ names to detector classes, geometric ambiguity, or formatting.

The current system has clear limits. It verifies only pairwise, two-dimensional spatial relations in axial CT slices; it does not handle volumetric reasoning, distance-sensitive relations, containment, overlap, or consistency across multiple structures. Its accuracy is also bounded by the detector’s quality, and the authors describe spatial verification as a controlled building block rather than a full radiology assistant. Still, by exposing each reasoning stage, the framework makes spatial claims traceable to image evidence—an important property for any future AI system meant to support, rather than replace, clinical judgment.