3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology

TL;DR

This paper introduces a new benchmark and comprehensive study for 3D medical image retrieval in radiology, aiming to improve diagnostic efficiency and reduce radiologist workload.

Contribution

It presents the first benchmark dataset for 3D medical image retrieval across four anatomies and evaluates various search strategies using this benchmark.

Findings

Aggregated 2D slices, 3D volumes, and multi-modal embeddings vary in retrieval effectiveness.

The benchmark facilitates standardized evaluation and comparison of retrieval methods.

Publicly available dataset and code support future research in the field.

Abstract

The increasing use of medical imaging in healthcare settings presents a significant challenge due to the increasing workload for radiologists, yet it also offers opportunity for enhancing healthcare outcomes if effectively leveraged. 3D image retrieval holds potential to reduce radiologist workloads by enabling clinicians to efficiently search through diagnostically similar or otherwise relevant cases, resulting in faster and more precise diagnoses. However, the field of 3D medical image retrieval is still emerging, lacking established evaluation benchmarks, comprehensive datasets, and thorough studies. This paper attempts to bridge this gap by introducing a novel benchmark for 3D Medical Image Retrieval (3D-MIR) that encompasses four different anatomies imaged with computed tomography. Using this benchmark, we explore a diverse set of search strategies that use aggregated 2D slices, 3D volumes, and multi-modal embeddings from popular multi-modal foundation models as queries. Quantitative and qualitative assessments of each approach are provided alongside an in-depth discussion that offers insight for future research. To promote the advancement of this field, our benchmark, dataset, and code are made publicly available.