Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation

TL;DR

This paper introduces X-REM, a retrieval-based method that uses image-text matching scores to generate more accurate and clinically relevant radiology reports from chest X-ray images, outperforming previous methods.

Contribution

The paper presents a novel image-text matching approach for radiology report retrieval that captures fine-grained interactions, improving report accuracy and clinical relevance.

Findings

X-REM outperforms prior models in natural language metrics

X-REM generates more zero-error reports in human evaluation

X-REM reduces average error severity in generated reports

Abstract

Automated generation of clinically accurate radiology reports can improve patient care. Previous report generation methods that rely on image captioning models often generate incoherent and incorrect text due to their lack of relevant domain knowledge, while retrieval-based attempts frequently retrieve reports that are irrelevant to the input image. In this work, we propose Contrastive X-Ray REport Match (X-REM), a novel retrieval-based radiology report generation module that uses an image-text matching score to measure the similarity of a chest X-ray image and radiology report for report retrieval. We observe that computing the image-text matching score with a language-image model can effectively capture the fine-grained interaction between image and text that is often lost when using cosine similarity. X-REM outperforms multiple prior radiology report generation modules in terms of both natural language and clinical metrics. Human evaluation of the generated reports suggests that X-REM increased the number of zero-error reports and decreased the average error severity compared to the baseline retrieval approach. Our code is available at: https://github.com/rajpurkarlab/X-REM