Lesion Guided Explainable Few Weak-shot Medical Report Generation

TL;DR

This paper introduces a lesion-guided, explainable few-shot medical report generation framework that effectively generates reports for unseen diseases by learning correlations between seen and novel classes through visual and semantic features.

Contribution

The proposed framework is the first to enable explainable report generation for unseen diseases using lesion-centric features and multi-view embeddings in a few-shot setting.

Findings

Outperforms existing methods on FFA-IR dataset for novel disease report generation.

Effectively detects abnormal regions and generates explainable reports.

Learns correlations between seen and unseen disease classes.

Abstract

Medical images are widely used in clinical practice for diagnosis. Automatically generating interpretable medical reports can reduce radiologists' burden and facilitate timely care. However, most existing approaches to automatic report generation require sufficient labeled data for training. In addition, the learned model can only generate reports for the training classes, lacking the ability to adapt to previously unseen novel diseases. To this end, we propose a lesion guided explainable few weak-shot medical report generation framework that learns correlation between seen and novel classes through visual and semantic feature alignment, aiming to generate medical reports for diseases not observed in training. It integrates a lesion-centric feature extractor and a Transformer-based report generation module. Concretely, the lesion-centric feature extractor detects the abnormal regions and learns correlations between seen and novel classes with multi-view (visual and lexical) embeddings. Then, features of the detected regions and corresponding embeddings are concatenated as multi-view input to the report generation module for explainable report generation, including text descriptions and corresponding abnormal regions detected in the images. We conduct experiments on FFA-IR, a dataset providing explainable annotations, showing that our framework outperforms others on report generation for novel diseases.