Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation

TL;DR

This paper introduces KERP, a novel approach combining knowledge-based and learning-based methods for generating accurate, structured, and explainable medical reports from images, achieving state-of-the-art results.

Contribution

The paper proposes a new framework that decomposes report generation into abnormality graph learning and natural language modeling, integrating prior medical knowledge with a Graph Transformer.

Findings

Achieves state-of-the-art results on two benchmarks.

Provides accurate abnormality descriptions and explainable regions.

Improves classification accuracy and human evaluation performance.

Abstract

Generating long and semantic-coherent reports to describe medical images poses great challenges towards bridging visual and linguistic modalities, incorporating medical domain knowledge, and generating realistic and accurate descriptions. We propose a novel Knowledge-driven Encode, Retrieve, Paraphrase (KERP) approach which reconciles traditional knowledge- and retrieval-based methods with modern learning-based methods for accurate and robust medical report generation. Specifically, KERP decomposes medical report generation into explicit medical abnormality graph learning and subsequent natural language modeling. KERP first employs an Encode module that transforms visual features into a structured abnormality graph by incorporating prior medical knowledge; then a Retrieve module that retrieves text templates based on the detected abnormalities; and lastly, a Paraphrase module that rewrites the templates according to specific cases. The core of KERP is a proposed generic implementation unit---Graph Transformer (GTR) that dynamically transforms high-level semantics between graph-structured data of multiple domains such as knowledge graphs, images and sequences. Experiments show that the proposed approach generates structured and robust reports supported with accurate abnormality description and explainable attentive regions, achieving the state-of-the-art results on two medical report benchmarks, with the best medical abnormality and disease classification accuracy and improved human evaluation performance.