Scene Graph Aided Radiology Report Generation

TL;DR

This paper introduces SGRRG, a novel framework that enhances radiology report generation by integrating scene graphs to incorporate detailed medical knowledge, leading to more accurate and clinically relevant reports.

Contribution

The paper proposes a new end-to-end scene graph aided network for radiology report generation, combining scene graph encoding with visual features for improved medical report accuracy.

Findings

Outperforms previous methods in report quality

Better captures abnormal findings

Enhances medical knowledge integration

Abstract

Radiology report generation (RRG) methods often lack sufficient medical knowledge to produce clinically accurate reports. The scene graph contains rich information to describe the objects in an image. We explore enriching the medical knowledge for RRG via a scene graph, which has not been done in the current RRG literature. To this end, we propose the Scene Graph aided RRG (SGRRG) network, a framework that generates region-level visual features, predicts anatomical attributes, and leverages an automatically generated scene graph, thus achieving medical knowledge distillation in an end-to-end manner. SGRRG is composed of a dedicated scene graph encoder responsible for translating the scene graph, and a scene graph-aided decoder that takes advantage of both patch-level and region-level visual information. A fine-grained, sentence-level attention method is designed to better dis-till the scene graph information. Extensive experiments demonstrate that SGRRG outperforms previous state-of-the-art methods in report generation and can better capture abnormal findings.