HOMIE: Histopathology Omni-modal Embedding for Pathology Composed Retrieval

TL;DR

HOMIE introduces a novel framework that adapts multimodal large language models for pathology retrieval, addressing domain and task mismatches, and establishes a new benchmark for evaluating complex clinical queries.

Contribution

The paper presents HOMIE, a systematic approach transforming general MLLMs into pathology-specific retrieval models, and introduces the PCR Benchmark for evaluating composed retrieval in pathology.

Findings

HOMIE matches SOTA on traditional retrieval tasks.

HOMIE outperforms baselines on the PCR benchmark.

The framework effectively addresses domain and task mismatches.

Abstract

The integration of Artificial Intelligence (AI) into pathology faces a fundamental challenge: black-box predictive models lack transparency, while generative approaches risk clinical hallucination. A case-based retrieval paradigm offers a more interpretable alternative for clinical adoption. However, current SOTA models are constrained by dual-encoder architectures that cannot process the composed modality of real-world clinical queries. We formally define the task of Pathology Composed Retrieval (PCR). However, progress in this newly defined task is blocked by two critical challenges: (1) Multimodal Large Language Models (MLLMs) offer the necessary deep-fusion architecture but suffer from a critical Task Mismatch and Domain Mismatch. (2) No benchmark exists to evaluate such compositional queries. To solve these challenges, we propose HOMIE, a systematic framework that transforms a general MLLM into a specialized retrieval expert. HOMIE resolves the dual mismatch via a two-stage process: a retrieval-adaptation stage to solve the task mismatch, and a pathology-specific tuning stage, featuring a progressive knowledge curriculum, pathology specfic stain and native resolution processing, to solve the domain mismatch. We also introduce the PCR Benchmark, a benchmark designed to evaluate composed retrieval in pathology. Experiments show that HOMIE, trained only on public data, matches SOTA performance on traditional retrieval tasks and outperforms all baselines on the newly defined PCR task.