Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling

TL;DR

This paper introduces HiVE-MIL, a hierarchical vision-language framework that models multi-scale tissue structures and aligns visual-textual data to improve few-shot classification of whole slide images in pathology.

Contribution

It proposes a novel graph-based hierarchical model with a two-stage filtering and hierarchical contrastive loss for better multimodal and multi-scale representation in WSI classification.

Findings

Outperforms traditional MIL and recent VLM-based MIL methods.

Achieves up to 4.1% improvement in macro F1 score in 16-shot settings.

Demonstrates effective modeling of hierarchical tissue structures and multimodal alignment.

Abstract

Vision-language models (VLMs) have recently been integrated into multiple instance learning (MIL) frameworks to address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the same modalities across scales (e.g., 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph consisting of (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism that removes weakly correlated patch-text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% in macro F1 under 16-shot settings. Our results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code is available at https://github.com/bryanwong17/HiVE-MIL.