Content-Based Image Retrieval for Multi-Class Volumetric Radiology Images: A Benchmark Study

TL;DR

This paper establishes a benchmark for content-based retrieval of 3D medical images, comparing different embeddings and methods to improve accuracy across multiple organs and structures.

Contribution

It introduces a comprehensive benchmark for multi-organ 3D medical image retrieval using pre-trained models and re-ranking techniques, enabling objective evaluation of CBIR approaches.

Findings

Achieved a retrieval recall of 1.0 for diverse anatomical regions.

Compared supervised and unsupervised embeddings for medical image retrieval.

Demonstrated the effectiveness of a late interaction re-ranking method.

Abstract

While content-based image retrieval (CBIR) has been extensively studied in natural image retrieval, its application to medical images presents ongoing challenges, primarily due to the 3D nature of medical images. Recent studies have shown the potential use of pre-trained vision embeddings for CBIR in the context of radiology image retrieval. However, a benchmark for the retrieval of 3D volumetric medical images is still lacking, hindering the ability to objectively evaluate and compare the efficiency of proposed CBIR approaches in medical imaging. In this study, we extend previous work and establish a benchmark for region-based and localized multi-organ retrieval using the TotalSegmentator dataset (TS) with detailed multi-organ annotations. We benchmark embeddings derived from pre-trained supervised models on medical images against embeddings derived from pre-trained unsupervised models on non-medical images for 29 coarse and 104 detailed anatomical structures in volume and region levels. For volumetric image retrieval, we adopt a late interaction re-ranking method inspired by text matching. We compare it against the original method proposed for volume and region retrieval and achieve a retrieval recall of 1.0 for diverse anatomical regions with a wide size range. The findings and methodologies presented in this paper provide insights and benchmarks for further development and evaluation of CBIR approaches in the context of medical imaging.