Objective:

Evaluate whether a radiology-specific prompting framework improves large language models’ detection of errors in radiology reports.

Approach:

• Dataset: Researchers assessed 1,200 reports from a single-hospital quality-assurance repository: 900 reports containing 1,170 clinician-validated errors and 300 error-free reports. Reports covered radiography, ultrasound, CT, and MRI.
• Comparison: Seven commercial and open-source LLMs were tested with standard prompting and the six-step Radiological Chain-of-Thought (RadCoT) framework. Outcomes included error-level micro-averaged precision, recall, and F1, as well as performance by error type and modali…

Key Findings:

• Across seven models, mean micro-averaged F1 increased from 0.77 with standard prompting to 0.85 with RadCoT; mean precision rose from 0.80 to 0.88 and recall from 0.75 to 0.83.
• The largest gains were for interpretation errors (F1 0.57 to 0.75), omission or insertion errors (0.64 to 0.82), and findings-impression discrepancies (0.69 to 0.81). Gains were smaller for numerical and typographical errors.
• F1 improved more for CT (0.68 to 0.80) and MRI (0.66 to 0.78) than for radiography (0.82 to 0.88) or ultrasound (0.77 to 0.83).
• With RadCoT, GPT-4o reached an F1 of 0.93 and Claude-3.5 Sonnet 0.92. Open-source models also improved, reaching 0.90 for Qwen-2.5-72B and 0.89 for Llama-3.3-70B.
• RadCoT increased inference time by 68%. GPT-4o processing time rose from about 15 to 22 seconds per report; open-source models took about 10 to 13 seconds per report with RadCoT.
• At an assumed 2% error prevalence, modeled positive predictive value was 0.15, with about 88 false-positive flags per 1,000 reports; at 20% prevalence, estimated positive predictive value was 0.68.

Interpretation:

The authors describe RadCoT as a potential first-pass quality-assurance aid, not a replacement for human review. They note that the error-enriched evaluation dataset may overestimate utility in routine settings and that complex interpretation errors were still missed in approximately one in four cases.

Limitations:

• Single-institution study with a relatively modest error sample, potentially limiting generalizability and representation of rare or nuanced errors.
• Evaluation used report text only; the models could not detect perceptual misses on images or independently verify diagnostic correctness against the examination.
• Explicit chain-of-thought output may not faithfully represent a model’s internal computation.
• The authors called for prospective, multi-institutional studies to assess real-world performance.

Conclusion:

The study found improved report-error detection with RadCoT across evaluated models, alongside increased processing time. The authors positioned the framework as a complementary tool requiring human review.

Sources:

• European Radiology Experimental (source named in article context)