On the Automatic Generation of Medical Imaging Reports

TL;DR

This paper presents a multi-task learning framework with co-attention and hierarchical LSTM models to automatically generate detailed medical imaging reports, addressing challenges like heterogeneous information and long report length.

Contribution

It introduces a novel multi-task framework with co-attention and hierarchical LSTM for improved medical report generation from images.

Findings

Effective report generation demonstrated on two datasets.

Improved localization of abnormal regions.

Enhanced handling of long, detailed reports.

Abstract

Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time- consuming and tedious for experienced physicians. To address these issues, we study the automatic generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings and tags. Second, abnormal regions in medical images are difficult to identify. Third, the re- ports are typically long, containing multiple sentences. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on two publicly available datasets.