Content-based 3D Image Retrieval and a ColBERT-inspired Re-ranking for Tumor Flagging and Staging

TL;DR

This paper introduces a novel content-based 3D medical image retrieval framework that utilizes a ColBERT-inspired re-ranking method, C-MIR, to improve tumor localization, flagging, and staging without relying on pre-segmented data.

Contribution

The study presents C-MIR, a volumetric re-ranking approach inspired by ColBERT, tailored for 3D medical images, and demonstrates its effectiveness across multiple tumor types and database configurations.

Findings

C-MIR effectively localizes regions of interest without pre-segmentation.

C-MIR improves tumor flagging accuracy, especially for colon and lung tumors (p<0.05).

The approach offers a computationally efficient alternative to data-intensive methods.

Abstract

The increasing volume of medical images poses challenges for radiologists in retrieving relevant cases. Content-based image retrieval (CBIR) systems offer potential for efficient access to similar cases, yet lack standardized evaluation and comprehensive studies. Building on prior studies for tumor characterization via CBIR, this study advances CBIR research for volumetric medical images through three key contributions: (1) a framework eliminating reliance on pre-segmented data and organ-specific datasets, aligning with large and unstructured image archiving systems, i.e. PACS in clinical practice; (2) introduction of C-MIR, a novel volumetric re-ranking method adapting ColBERT's contextualized late interaction mechanism for 3D medical imaging; (3) comprehensive evaluation across four tumor sites using three feature extractors and three database configurations. Our evaluations highlight the significant advantages of C-MIR. We demonstrate the successful adaptation of the late interaction principle to volumetric medical images, enabling effective context-aware re-ranking. A key finding is C-MIR's ability to effectively localize the region of interest, eliminating the need for pre-segmentation of datasets and offering a computationally efficient alternative to systems relying on expensive data enrichment steps. C-MIR demonstrates promising improvements in tumor flagging, achieving improved performance, particularly for colon and lung tumors (p<0.05). C-MIR also shows potential for improving tumor staging, warranting further exploration of its capabilities. Ultimately, our work seeks to bridge the gap between advanced retrieval techniques and their practical applications in healthcare, paving the way for improved diagnostic processes.