ViDRiP-LLaVA: A Dataset and Benchmark for Diagnostic Reasoning from Pathology Videos

TL;DR

ViDRiP-LLaVA introduces a large multimodal model and dataset for diagnostic reasoning in pathology videos, combining diverse image scenarios and chain-of-thought explanations to support clinical decision-making.

Contribution

It is the first to integrate multiple pathology video scenarios with a new dataset and benchmark, advancing AI diagnostic reasoning in computational pathology.

Findings

Established a new benchmark for pathology video analysis

Transferred knowledge from single-image datasets to improve video understanding

Demonstrated the effectiveness of multimodal reasoning in diagnostic tasks

Abstract

We present ViDRiP-LLaVA, the first large multimodal model (LMM) in computational pathology that integrates three distinct image scenarios, including single patch images, automatically segmented pathology video clips, and manually segmented pathology videos. This integration closely mirrors the natural diagnostic process of pathologists. By generating detailed histological descriptions and culminating in a definitive sign-out diagnosis, ViDRiP-LLaVA bridges visual narratives with diagnostic reasoning. Central to our approach is the ViDRiP-Instruct dataset, comprising 4278 video and diagnosis-specific chain-of-thought instructional pairs sourced from educational histopathology videos on YouTube. Although high-quality data is critical for enhancing diagnostic reasoning, its creation is time-intensive and limited in volume. To overcome this challenge, we transfer knowledge from existing single-image instruction datasets to train on weakly annotated, keyframe-extracted clips, followed by fine-tuning on manually segmented videos. ViDRiP-LLaVA establishes a new benchmark in pathology video analysis and offers a promising foundation for future AI systems that support clinical decision-making through integrated visual and diagnostic reasoning. Our code, data, and model are publicly available at: https://github.com/QuIIL/ViDRiP-LLaVA.