WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image

TL;DR

This paper introduces WSI-LLaVA, a multimodal large language model designed for comprehensive analysis of whole slide images in pathology, supported by a new large-scale benchmark and specialized evaluation metrics.

Contribution

The paper presents WSI-LLaVA, a novel framework for gigapixel WSI understanding, along with WSI-Bench, a large-scale morphology-aware benchmark, and new metrics for pathological assessment.

Findings

WSI-LLaVA outperforms existing models in morphological analysis.

The model shows a strong correlation between morphological understanding and diagnostic accuracy.

Introduction of WSI-Bench and specialized WSI metrics enhances evaluation of WSI models.

Abstract

Recent advancements in computational pathology have produced patch-level Multi-modal Large Language Models (MLLMs), but these models are limited by their inability to analyze whole slide images (WSIs) comprehensively and their tendency to bypass crucial morphological features that pathologists rely on for diagnosis. To address these challenges, we first introduce WSI-Bench, a large-scale morphology-aware benchmark containing 180k VQA pairs from 9,850 WSIs across 30 cancer types, designed to evaluate MLLMs' understanding of morphological characteristics crucial for accurate diagnosis. Building upon this benchmark, we present WSI-LLaVA, a novel framework for gigapixel WSI understanding that employs a three-stage training approach: WSI-text alignment, feature space alignment, and task-specific instruction tuning. To better assess model performance in pathological contexts, we develop two specialized WSI metrics: WSI-Precision and WSI-Relevance. Experimental results demonstrate that WSI-LLaVA outperforms existing models across all capability dimensions, with a significant improvement in morphological analysis, establishing a clear correlation between morphological understanding and diagnostic accuracy.