AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Cancer Benchmark Tests Whether AI Can Follow a Patient Through Care

Medical AI systems now perform well on licensing-style exam questions and single-image tasks, but those tests rarely ask whether a model can keep one patient’s care consistent over time. A new study from researchers at The Hong Kong Polytechnic University, Sun Yat-sen University and collaborators argues that this gap matters most in specialized oncology, where a correct guideline answer, a correct staging reading and a correct treatment decision must all hold for the same patient.

The team introduces NPCBench, a benchmark for nasopharyngeal carcinoma (NPC), a cancer of the upper throat that is especially common in Southern China and Southeast Asia. The benchmark is modeled on specialist training. It moves from guideline recall to reading pathology slides and MRI scans, planning treatment for a single patient state, revising decisions as a patient’s condition changes, and answering open-ended consultation questions. Its central component is a “CarePath” cohort of 100 patients with 417 scored tasks: 100 pathology assessments, 100 MRI tumor-stage (T-stage) assessments and 217 treatment decisions. The full benchmark contains 4,777 items.

The key metric is strict success. A patient counts only if the model’s pathology label, MRI T-stage and every treatment answer are all correct. The authors call the resulting shortfall a “composition gap”: strong scores on individual tasks do not guarantee joint correctness for one patient.

The results show that gap. Gemini-3.1 reached 89.9% decision-level (Care) accuracy but only 11.0% strict success. Claude-4.6 had lower Care accuracy, at 80.6%, but the highest strict success, at 23.0% (exact 95% confidence interval 15.2–32.5%). Across the 21 multimodal models tested, strict success reached at most 23%.

Visual grounding emerged as a particular weakness. On a 100-patient anatomical panel, complete 18-structure invasion correctness reached at most 5.0%. In one controlled comparison, a fixed staging rule scored 99.0% when given reference findings. When the same rule used Gemini-3.1’s own visual findings, accuracy fell to 49.0% with known anatomy and 36.0% with fully predicted evidence. The authors attribute the drop mainly to flawed invasion recognition, not to the staging logic.

Patient history also cut both ways. For GPT-5.1, giving full chronological history rather than a reorganized snapshot produced a 1.0 percentage-point gain in turn-level accuracy alongside a 1.0 point decline in all-turn patient success. Most such differences had confidence intervals spanning zero.

In consultations, GPT-5.1 scored highest at 58.4 on a 0–100 rubric. Yet the primary GPT-OSS-120B judge flagged major errors in 14.6% of its responses. Automated judges also diverged from physicians: in an expanded audit, the primary judge agreed with physicians on 70.5% of criteria, with 51.8% sensitivity and 85.8% specificity.

The study has clear limits. Records come mainly from one center, 75 CarePath episodes use model-selected pathology assignments, and the expanded judge audit covers only six cases. Multiple-choice cues also inflate guideline scores: removing answer options lowered accuracy by 7.0 points for Gemini-3.1, 12.7 points for GPT-4.1 and 15.5 points for GPT-5.1, in a 71-question physician-audited sample.

The authors argue that evidence-decision consistency should become a central target for medical AI evaluation. The results do not show whether any model would improve patient outcomes. They establish a testbed, which the authors say needs multicenter and prospective studies before its value to clinical care can be judged.