Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation

TL;DR

This paper introduces an unsupervised knowledge graph auto-encoder for medical report generation that effectively utilizes independent image and report data, reducing reliance on paired datasets and improving performance.

Contribution

The paper presents a novel unsupervised model, KGAE, that leverages a knowledge graph to generate medical reports without requiring paired image-report training data.

Findings

KGAE can generate medical reports without paired training data.

KGAE outperforms state-of-the-art models after fine-tuning with paired data.

The model works in semi-supervised and supervised settings.

Abstract

Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approaches mainly adopt a supervised manner and heavily rely on coupled image-report pairs. However, in the medical domain, building a large-scale image-report paired dataset is both time-consuming and expensive. To relax the dependency on paired data, we propose an unsupervised model Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images and reports in training. KGAE consists of a pre-constructed knowledge graph, a knowledge-driven encoder and a knowledge-driven decoder. The knowledge graph works as the shared latent space to bridge the visual and textual domains; The knowledge-driven encoder projects medical images and reports to the corresponding coordinates in this latent space and the knowledge-driven decoder generates a medical report given a coordinate in this space. Since the knowledge-driven encoder and decoder can be trained with independent sets of images and reports, KGAE is unsupervised. The experiments show that the unsupervised KGAE generates desirable medical reports without using any image-report training pairs. Moreover, KGAE can also work in both semi-supervised and supervised settings, and accept paired images and reports in training. By further fine-tuning with image-report pairs, KGAE consistently outperforms the current state-of-the-art models on two datasets.