IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer
Keqiang Fan, Xiaohao Cai, Mahesan Niranjan

TL;DR
This paper introduces IIHT, a hierarchical transformer framework that improves medical report generation by combining image features with disease indicators, enhancing clinical accuracy and report fluency.
Contribution
The paper proposes a novel image-to-indicator hierarchical transformer that integrates disease indicators into report generation, addressing data imbalance and sequence correlation issues in medical data.
Findings
Outperforms state-of-the-art methods on multiple metrics
Enables real-world indicator modification by radiologists
Achieves fluent and clinically accurate report generation
Abstract
Automated medical report generation has become increasingly important in medical analysis. It can produce computer-aided diagnosis descriptions and thus significantly alleviate the doctors' work. Inspired by the huge success of neural machine translation and image captioning, various deep learning methods have been proposed for medical report generation. However, due to the inherent properties of medical data, including data imbalance and the length and correlation between report sequences, the generated reports by existing methods may exhibit linguistic fluency but lack adequate clinical accuracy. In this work, we propose an image-to-indicator hierarchical transformer (IIHT) framework for medical report generation. It consists of three modules, i.e., a classifier module, an indicator expansion module and a generator module. The classifier module first extracts image features from the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Natural Language Processing Techniques
