Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation

TL;DR

This paper introduces a novel approach for radiology report generation that mimics radiologists' working patterns by exploring and distilling posterior and prior knowledge to improve report accuracy and reduce data biases.

Contribution

The paper proposes the PPKED framework with three modules to explore and distill visual and textual knowledge, significantly enhancing report generation performance.

Findings

Outperforms previous state-of-the-art models on MIMIC-CXR and IU-Xray datasets.

Effectively reduces visual and textual data biases in report generation.

Demonstrates the importance of integrating prior medical knowledge and experience.

Abstract

Automatically generating radiology reports can improve current clinical practice in diagnostic radiology. On one hand, it can relieve radiologists from the heavy burden of report writing; On the other hand, it can remind radiologists of abnormalities and avoid the misdiagnosis and missed diagnosis. Yet, this task remains a challenging job for data-driven neural networks, due to the serious visual and textual data biases. To this end, we propose a Posterior-and-Prior Knowledge Exploring-and-Distilling approach (PPKED) to imitate the working patterns of radiologists, who will first examine the abnormal regions and assign the disease topic tags to the abnormal regions, and then rely on the years of prior medical knowledge and prior working experience accumulations to write reports. Thus, the PPKED includes three modules: Posterior Knowledge Explorer (PoKE), Prior Knowledge Explorer (PrKE) and Multi-domain Knowledge Distiller (MKD). In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias. The explored knowledge is distilled by the MKD to generate the final reports. Evaluated on MIMIC-CXR and IU-Xray datasets, our method is able to outperform previous state-of-the-art models on these two datasets.