Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports

TL;DR

This paper introduces a novel evaluation method for radiology reports that combines radiologist expertise with advanced LLM techniques, significantly improving the accuracy and robustness of AI report assessments.

Contribution

It proposes an innovative approach integrating radiologist standards with LLMs using ICIL and CoT reasoning, surpassing existing metrics in report evaluation accuracy.

Findings

GPT-4 (5-shot) outperforms METEOR by 0.19 in evaluation score

Regressed GPT-4 aligns more closely with radiologist judgments

The method demonstrates robustness validated through iterative testing

Abstract

In radiology, Artificial Intelligence (AI) has significantly advanced report generation, but automatic evaluation of these AI-produced reports remains challenging. Current metrics, such as Conventional Natural Language Generation (NLG) and Clinical Efficacy (CE), often fall short in capturing the semantic intricacies of clinical contexts or overemphasize clinical details, undermining report clarity. To overcome these issues, our proposed method synergizes the expertise of professional radiologists with Large Language Models (LLMs), like GPT-3.5 and GPT-4 1. Utilizing In-Context Instruction Learning (ICIL) and Chain of Thought (CoT) reasoning, our approach aligns LLM evaluations with radiologist standards, enabling detailed comparisons between human and AI generated reports. This is further enhanced by a Regression model that aggregates sentence evaluation scores. Experimental results show that our "Detailed GPT-4 (5-shot)" model achieves a 0.48 score, outperforming the METEOR metric by 0.19, while our "Regressed GPT-4" model shows even greater alignment with expert evaluations, exceeding the best existing metric by a 0.35 margin. Moreover, the robustness of our explanations has been validated through a thorough iterative strategy. We plan to publicly release annotations from radiology experts, setting a new standard for accuracy in future assessments. This underscores the potential of our approach in enhancing the quality assessment of AI-driven medical reports.