C^2M-DoT: Cross-modal consistent multi-view medical report generation with domain transfer network

TL;DR

This paper introduces C^2M-DoT, a novel framework for multi-view medical report generation that leverages cross-view information and domain transfer to improve report accuracy and applicability in clinical settings.

Contribution

The paper proposes a multi-view contrastive learning framework combined with a domain transfer network to generate semantically consistent medical reports from multi-view images.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Effective in single-view and multi-view scenarios

Ablation confirms each component's importance

Abstract

In clinical scenarios, multiple medical images with different views are usually generated simultaneously, and these images have high semantic consistency. However, most existing medical report generation methods only consider single-view data. The rich multi-view mutual information of medical images can help generate more accurate reports, however, the dependence of multi-view models on multi-view data in the inference stage severely limits their application in clinical practice. In addition, word-level optimization based on numbers ignores the semantics of reports and medical images, and the generated reports often cannot achieve good performance. Therefore, we propose a cross-modal consistent multi-view medical report generation with a domain transfer network (C^2M-DoT). Specifically, (i) a semantic-based multi-view contrastive learning medical report generation framework is adopted to utilize cross-view information to learn the semantic representation of lesions; (ii) a domain transfer network is further proposed to ensure that the multi-view report generation model can still achieve good inference performance under single-view input; (iii) meanwhile, optimization using a cross-modal consistency loss facilitates the generation of textual reports that are semantically consistent with medical images. Extensive experimental studies on two public benchmark datasets demonstrate that C^2M-DoT substantially outperforms state-of-the-art baselines in all metrics. Ablation studies also confirmed the validity and necessity of each component in C^2M-DoT.