Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis

TL;DR

This paper introduces OmniPath, a large vision-language model tailored for pathology analysis that enhances feature extraction across scales, overcoming resolution limits of existing models, and achieves superior diagnostic accuracy and efficiency.

Contribution

The paper presents two novel strategies for feature extraction in large vision-language models, specifically designed for pathology images, improving both detail capture and inference speed.

Findings

Outperforms existing models in diagnostic accuracy

Demonstrates significant efficiency improvements

Trained on a large dataset of 490K pathology samples

Abstract

Pathological diagnosis is vital for determining disease characteristics, guiding treatment, and assessing prognosis, relying heavily on detailed, multi-scale analysis of high-resolution whole slide images (WSI). However, existing large vision-language models (LVLMs) are limited by input resolution constraints, hindering their efficiency and accuracy in pathology image analysis. To overcome these issues, we propose two innovative strategies: the mixed task-guided feature enhancement, which directs feature extraction toward lesion-related details across scales, and the prompt-guided detail feature completion, which integrates coarse- and fine-grained features from WSI based on specific prompts without compromising inference speed. Leveraging a comprehensive dataset of 490K samples from diverse pathology tasks, we trained the pathology-specialized LVLM, OmniPath. Extensive experiments demonstrate that this model significantly outperforms existing methods in diagnostic accuracy and efficiency, providing an interactive, clinically aligned approach for auxiliary diagnosis in a wide range of pathology applications.