MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks

TL;DR

MedRAT introduces a novel approach for unpaired medical report generation from X-ray images by leveraging auxiliary tasks like contrastive learning and classification to align images and reports without requiring paired data.

Contribution

The paper presents MedRAT, a new model that effectively generates medical reports from images using unpaired datasets, avoiding external knowledge bases and pre-processing steps.

Findings

MedRAT outperforms previous state-of-the-art methods.

The approach demonstrates effective report generation without paired data.

Auxiliary tasks successfully align images and reports in the embedding space.

Abstract

Medical report generation from X-ray images is a challenging task, particularly in an unpaired setting where paired image-report data is unavailable for training. To address this challenge, we propose a novel model that leverages the available information in two distinct datasets, one comprising reports and the other consisting of images. The core idea of our model revolves around the notion that combining auto-encoding report generation with multi-modal (report-image) alignment can offer a solution. However, the challenge persists regarding how to achieve this alignment when pair correspondence is absent. Our proposed solution involves the use of auxiliary tasks, particularly contrastive learning and classification, to position related images and reports in close proximity to each other. This approach differs from previous methods that rely on pre-processing steps, such as using external information stored in a knowledge graph. Our model, named MedRAT, surpasses previous state-of-the-art methods, demonstrating the feasibility of generating comprehensive medical reports without the need for paired data or external tools.