RadiomicsRetrieval: A Customizable Framework for Medical Image Retrieval Using Radiomics Features

TL;DR

RadiomicsRetrieval is a flexible 3D medical image retrieval framework that combines radiomics features with deep learning embeddings, enabling efficient, location-aware, and customizable searches in volumetric medical images.

Contribution

It introduces a novel framework that integrates radiomics descriptors with deep learning embeddings for 3D medical image retrieval, supporting flexible, location-aware querying with minimal user prompts.

Findings

Radiomics features improve retrieval specificity.

Anatomical positional embedding enhances location-based searches.

Framework supports diverse clinical query scenarios.

Abstract

Medical image retrieval is a valuable field for supporting clinical decision-making, yet current methods primarily support 2D images and require fully annotated queries, limiting clinical flexibility. To address this, we propose RadiomicsRetrieval, a 3D content-based retrieval framework bridging handcrafted radiomics descriptors with deep learning-based embeddings at the tumor level. Unlike existing 2D approaches, RadiomicsRetrieval fully exploits volumetric data to leverage richer spatial context in medical images. We employ a promptable segmentation model (e.g., SAM) to derive tumor-specific image embeddings, which are aligned with radiomics features extracted from the same tumor via contrastive learning. These representations are further enriched by anatomical positional embedding (APE). As a result, RadiomicsRetrieval enables flexible querying based on shape, location, or partial feature sets. Extensive experiments on both lung CT and brain MRI public datasets demonstrate that radiomics features significantly enhance retrieval specificity, while APE provides global anatomical context essential for location-based searches. Notably, our framework requires only minimal user prompts (e.g., a single point), minimizing segmentation overhead and supporting diverse clinical scenarios. The capability to query using either image embeddings or selected radiomics attributes highlights its adaptability, potentially benefiting diagnosis, treatment planning, and research on large-scale medical imaging repositories. Our code is available at https://github.com/nainye/RadiomicsRetrieval.