Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Query

TL;DR

This paper presents RegionMIR, a contrastive pretraining method that improves medical image retrieval by focusing on anatomical regions, achieving higher accuracy and demonstrating generalizability across different anatomies.

Contribution

The paper introduces a novel region-based contrastive pretraining approach for medical image retrieval that emphasizes anatomical localization and improves retrieval accuracy.

Findings

Achieved 94.12% accuracy in anatomy classification, up from 92.24%.

Demonstrated effective retrieval of images with similar anatomical regions.

Showed generalizability across multiple anatomies with different morphology.

Abstract

We introduce a novel Region-based contrastive pretraining for Medical Image Retrieval (RegionMIR) that demonstrates the feasibility of medical image retrieval with similar anatomical regions. RegionMIR addresses two major challenges for medical image retrieval i) standardization of clinically relevant searching criteria (e.g., anatomical, pathology-based), and ii) localization of anatomical area of interests that are semantically meaningful. In this work, we propose an ROI image retrieval image network that retrieves images with similar anatomy by extracting anatomical features (via bounding boxes) and evaluate similarity between pairwise anatomy-categorized features between the query and the database of images using contrastive learning. ROI queries are encoded using a contrastive-pretrained encoder that was fine-tuned for anatomy classification, which generates an anatomical-specific latent space for region-correlated image retrieval. During retrieval, we compare the anatomically encoded query to find similar features within a feature database generated from training samples, and retrieve images with similar regions from training samples. We evaluate our approach on both anatomy classification and image retrieval tasks using the Chest ImaGenome Dataset. Our proposed strategy yields an improvement over state-of-the-art pretraining and co-training strategies, from 92.24 to 94.12 (2.03%) classification accuracy in anatomies. We qualitatively evaluate the image retrieval performance demonstrating generalizability across multiple anatomies with different morphology.