Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation

TL;DR

This paper presents CVLM, a cyclic manipulation method that improves interpretability of AI-generated X-ray reports by identifying key image features influencing report content, thereby enhancing transparency and reliability.

Contribution

Introduction of CVLM, a cyclic manipulation module that clarifies image feature influence on report generation, improving interpretability over existing explanation methods.

Findings

CVLM outperforms existing explanation methods in identifying precise features.

It enhances transparency and trustworthiness of AI-generated medical reports.

Empirical results confirm improved reliability of report interpretation.

Abstract

Despite significant advancements in automated report generation, the opaqueness of text interpretability continues to cast doubt on the reliability of the content produced. This paper introduces a novel approach to identify specific image features in X-ray images that influence the outputs of report generation models. Specifically, we propose Cyclic Vision-Language Manipulator CVLM, a module to generate a manipulated X-ray from an original X-ray and its report from a designated report generator. The essence of CVLM is that cycling manipulated X-rays to the report generator produces altered reports aligned with the alterations pre-injected into the reports for X-ray generation, achieving the term "cyclic manipulation". This process allows direct comparison between original and manipulated X-rays, clarifying the critical image features driving changes in reports and enabling model users to assess the reliability of the generated texts. Empirical evaluations demonstrate that CVLM can identify more precise and reliable features compared to existing explanation methods, significantly enhancing the transparency and applicability of AI-generated reports.