From single slices to whole studies: A scoping review of frontier LLM evaluation on CT and MRI and the case for study-level assessment

Abstract

Large multimodal and vision–language models are increasingly evaluated on computed tomography (CT) and magnetic resonance imaging (MRI), yet a radiology study is a multi-series, multi-plane, multi-sequence volumetric object read with prior examinations and clinical context, not a single picture. This scoping review maps the 2025–2026 evidence base on frontier and specialist model evaluation for CT and MRI, characterises the granularity at which models are assessed, and examines whether reported scores represent real value for patients and radiologists. Following PRISMA-ScR, we searched arXiv, medRxiv, bioRxiv, PubMed/MEDLINE, IEEE Xplore, major conference proceedings, challenge registries, and dataset repositories, and charted benchmark, dataset, model, and evaluation papers against a four-level input-granularity taxonomy (single slice; single volume; curated multi-series; whole study with priors and context) and realism-gap dimensions. Evaluation clusters at the single-slice and single-volume levels; whole-study evaluation is rare and, where present, relies on degraded imaging with the radiology report available as an input. Frontier models receive volumetric data as sampled slices or montages constrained by per-request image caps and consistently underperform specialist models and radiologists on volumetric and measurement-dependent tasks, while widely reported metrics map weakly onto clinical value and slice-level data leakage can inflate accuracy by tens of percentage points. The deployable-at-scale option, API-accessible frontier models, is therefore the least rigorously evaluated on whole studies. We propose desiderata and a twelve-item checklist for study-level, workflow-faithful evaluation: whole-study input with priors, the report withheld, consensus- or outcome-based reference standards, task framing that mirrors the read, and explicit safety and deployability reporting.