Efficient Whole Slide Pathology VQA via Token Compression

TL;DR

This paper introduces TCP-LLaVA, a novel multimodal large language model architecture that efficiently performs whole-slide image visual question answering by compressing visual and textual tokens, reducing resource use while improving accuracy.

Contribution

The paper presents the first MLLM architecture for WSI VQA using token compression, significantly lowering computational costs and enhancing performance.

Findings

Outperforms existing MLLM baselines in VQA accuracy.

Reduces training resource consumption substantially.

Effective token compression improves efficiency and accuracy.

Abstract

Whole-slide images (WSIs) in pathology can reach up to 10,000 x 10,000 pixels, posing significant challenges for multimodal large language model (MLLM) due to long context length and high computational demands. Previous methods typically focus on patch-level analysis or slide-level classification using CLIP-based models with multi-instance learning, but they lack the generative capabilities needed for visual question answering (VQA). More recent MLLM-based approaches address VQA by feeding thousands of patch tokens directly into the language model, which leads to excessive resource consumption. To address these limitations, we propose Token Compression Pathology LLaVA (TCP-LLaVA), the first MLLM architecture to perform WSI VQA via token compression. TCP-LLaVA introduces a set of trainable compression tokens that aggregate visual and textual information through a modality compression module, inspired by the [CLS] token mechanism in BERT. Only the compressed tokens are forwarded to the LLM for answer generation, significantly reducing input length and computational cost. Experiments on ten TCGA tumor subtypes show that TCP-LLaVA outperforms existing MLLM baselines in VQA accuracy while reducing training resource consumption by a substantial margin.