Unsupervised Multimodal Representation Learning across Medical Images and Reports

TL;DR

This paper explores unsupervised and supervised joint embedding techniques for medical images and reports, demonstrating that limited supervision can achieve results comparable to fully-supervised methods on the MIMIC-CXR dataset.

Contribution

It establishes baseline joint embedding results for medical images and reports, highlighting the effectiveness of limited supervision in multimodal representation learning.

Findings

Limited supervision achieves comparable results to full supervision in document retrieval.

Baseline joint embeddings established on MIMIC-CXR dataset.

Both local and global retrieval methods evaluated.

Abstract

Joint embeddings between medical imaging modalities and associated radiology reports have the potential to offer significant benefits to the clinical community, ranging from cross-domain retrieval to conditional generation of reports to the broader goals of multimodal representation learning. In this work, we establish baseline joint embedding results measured via both local and global retrieval methods on the soon to be released MIMIC-CXR dataset consisting of both chest X-ray images and the associated radiology reports. We examine both supervised and unsupervised methods on this task and show that for document retrieval tasks with the learned representations, only a limited amount of supervision is needed to yield results comparable to those of fully-supervised methods.