HookMIL: Revisiting Context Modeling in Multiple Instance Learning for Computational Pathology

TL;DR

HookMIL introduces a novel, efficient, and context-aware multiple instance learning framework for computational pathology that leverages learnable hook tokens and multimodal priors to improve performance and interpretability.

Contribution

The paper proposes HookMIL, a new MIL framework using learnable hook tokens with multimodal initialization and a diversity loss, enhancing efficiency and contextual modeling in pathology analysis.

Findings

Achieves state-of-the-art results on four pathology datasets.

Improves computational efficiency with linear attention complexity.

Enhances interpretability through structured contextual aggregation.

Abstract

Multiple Instance Learning (MIL) has enabled weakly supervised analysis of whole-slide images (WSIs) in computational pathology. However, traditional MIL approaches often lose crucial contextual information, while transformer-based variants, though more expressive, suffer from quadratic complexity and redundant computations. To address these limitations, we propose HookMIL, a context-aware and computationally efficient MIL framework that leverages compact, learnable hook tokens for structured contextual aggregation. These tokens can be initialized from (i) key-patch visual features, (ii) text embeddings from vision-language pathology models, and (iii) spatially grounded features from spatial transcriptomics-vision models. This multimodal initialization enables Hook Tokens to incorporate rich textual and spatial priors, accelerating convergence and enhancing representation quality. During training, Hook tokens interact with instances through bidirectional attention with linear complexity. To further promote specialization, we introduce a Hook Diversity Loss that encourages each token to focus on distinct histopathological patterns. Additionally, a hook-to-hook communication mechanism refines contextual interactions while minimizing redundancy. Extensive experiments on four public pathology datasets demonstrate that HookMIL achieves state-of-the-art performance, with improved computational efficiency and interpretability. Codes are available at https://github.com/lingxitong/HookMIL.