Scene Graph Aided Radiology Report Generation
Jun Wang, Lixing Zhu, Abhir Bhalerao, and Yulan He

TL;DR
This paper introduces SGRRG, a novel framework that enhances radiology report generation by integrating scene graphs to incorporate detailed medical knowledge, leading to more accurate and clinically relevant reports.
Contribution
The paper proposes a new end-to-end scene graph aided network for radiology report generation, combining scene graph encoding with visual features for improved medical report accuracy.
Findings
Outperforms previous methods in report quality
Better captures abnormal findings
Enhances medical knowledge integration
Abstract
Radiology report generation (RRG) methods often lack sufficient medical knowledge to produce clinically accurate reports. The scene graph contains rich information to describe the objects in an image. We explore enriching the medical knowledge for RRG via a scene graph, which has not been done in the current RRG literature. To this end, we propose the Scene Graph aided RRG (SGRRG) network, a framework that generates region-level visual features, predicts anatomical attributes, and leverages an automatically generated scene graph, thus achieving medical knowledge distillation in an end-to-end manner. SGRRG is composed of a dedicated scene graph encoder responsible for translating the scene graph, and a scene graph-aided decoder that takes advantage of both patch-level and region-level visual information. A fine-grained, sentence-level attention method is designed to better dis-till the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Biomedical Text Mining and Ontologies · Radiomics and Machine Learning in Medical Imaging
MethodsKnowledge Distillation
