A radiology-specific prompting framework may improve the ability of large language models (LLMs) to detect errors in radiology reports, with the largest gains involving complex interpretation and omission or insertion errors.
Investigators evaluated the Radiological Chain-of-Thought (RadCoT) framework using a quality-assurance repository from Beijing Friendship Hospital, Capital Medical University. The dataset included 1,170 clinician-validated errors identified from January 2021 through December 2024 and mapped to 900 error-containing reports. Investigators added 300 error-free reports, yielding 1,200 reports evenly distributed across radiography, ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI). Seven LLMs, including commercial and open-source models, were evaluated with standard prompting and the 6-step RadCoT framework.
The primary performance measures were micro-averaged precision, recall, and F1 score (a measure balancing precision and recall) at the error-instance level. Secondary analyses examined performance by error type and imaging modality and assessed inference time. RadCoT guided each model through sequential checks of anatomy, measurements, consistency across report sections, findings-impression alignment, clinical completeness, and terminology. The authors also compared RadCoT with generic chain-of-thought prompting in GPT-4o and Llama-3.3-70B.
Across the 7 models, RadCoT increased mean micro-averaged F1 from 0.77 with standard prompting to 0.85. Mean precision increased from 0.80 to 0.88, and mean recall increased from 0.75 to 0.83. GPT-4o achieved the highest F1 with RadCoT, increasing from 0.88 to 0.93, while Claude-3.5 Sonnet increased from 0.86 to 0.92.
The largest improvements occurred in error categories requiring higher-level reasoning. Mean F1 for interpretation errors increased from 0.57 to 0.75, and F1 for omission or insertion errors increased from 0.64 to 0.82. Findings-impression discrepancies improved from 0.69 to 0.81. Gains were smaller for numerical and typographical errors, which had higher performance with standard prompting.
Improvements were also greater for CT and MRI reports. F1 increased from 0.68 to 0.80 for CT and from 0.66 to 0.78 for MRI, compared with increases from 0.82 to 0.88 for radiography and 0.77 to 0.83 for ultrasound. However, the authors cautioned that the larger gains in CT and MRI could not be attributed to report complexity alone because RadCoT’s cross-referencing and cross-sectional verification steps may have provided an additional advantage for multiplanar examinations.
Open-source models showed similar directional improvements. With RadCoT, F1 reached 0.90 for Qwen-2.5-72B and 0.89 for Llama-3.3-70B. The latter approached the 0.88 F1 achieved by GPT-4o with standard prompting, with no statistically significant difference detected between the 2 model-prompting combinations in this dataset. In the focused comparison with generic chain-of-thought prompting, RadCoT increased F1 from 0.90 to 0.93 for GPT-4o and from 0.86 to 0.89 for Llama-3.3-70B, indicating an incremental performance gain from the radiology-specific scaffold in those models.
The performance gains required additional processing time. RadCoT increased inference time by 68%. For GPT-4o, mean processing time increased from approximately 15 seconds to 22 seconds per report, while open-source models required approximately 10 to 13 seconds per report with RadCoT. This latency may be compatible with retrospective auditing and review before sign-off but less suitable for real-time dictation.
The authors also modeled performance at error prevalences more representative of clinical deployment because the evaluation dataset was deliberately enriched for errors. At an assumed 2% error prevalence, the estimated positive predictive value was 0.15, with approximately 88 false-positive flags per 1,000 reports. Positive predictive value was estimated to increase to 0.68 at an assumed 20% prevalence. The authors cautioned that performance in the error-enriched evaluation dataset could therefore overestimate operational utility in routine settings where errors are less common.
The study had several limitations, including its single-institution design and relatively modest sample of errors, which may limit generalizability and leave rare or nuanced errors underrepresented. The evaluation was also restricted to report text rather than imaging data, meaning RadCoT could not identify perceptual misses on images or independently determine diagnostic correctness against the underlying examination. Additionally, explicit chain-of-thought output may not faithfully represent a model’s internal computation, and the investigators noted that prospective, multi-institutional studies are needed to determine performance in real-world clinical settings.
Radiology-specific structured prompting may strengthen LLM-based report quality assurance, including with locally deployed open-source models, but the authors positioned RadCoT as a complementary first-pass tool requiring human review rather than an autonomous quality-assurance system. Even with RadCoT, complex interpretation errors were missed in approximately 1 in 4 cases.
“RadCoT is best viewed as a first-pass QA assistant rather than a replacement for human review,” wrote lead study author Jia Li, of the Department of Radiology, Beijing Friendship Hospital, Capital Medical University, and colleagues.
Disclosures:The authors reported no conflicts of interest.
Source: European Radiology Experimental
