Classifying Whole Slide Images: What Matters?

TL;DR

This study investigates various design choices in whole slide image classification, revealing that local patch features are more crucial than global context, and focused pre-training enhances performance.

Contribution

It systematically evaluates the impact of global versus local information and pre-training datasets, challenging assumptions about the importance of global context in WSI classification.

Findings

Global context does not improve accuracy and can worsen performance.

Simple multi-instance learning approaches perform nearly as well as complex models.

Focused pre-training on relevant cancer types enhances feature extraction.

Abstract

Recently there have been many algorithms proposed for the classification of very high resolution whole slide images (WSIs). These new algorithms are mostly focused on finding novel ways to combine the information from small local patches extracted from the slide, with an emphasis on effectively aggregating more global information for the final predictor. In this paper we thoroughly explore different key design choices for WSI classification algorithms to investigate what matters most for achieving high accuracy. Surprisingly, we found that capturing global context information does not necessarily mean better performance. A model that captures the most global information consistently performs worse than a model that captures less global information. In addition, a very simple multi-instance learning method that captures no global information performs almost as well as models that capture a lot of global information. These results suggest that the most important features for effective WSI classification are captured at the local small patch level, where cell and tissue micro-environment detail is most pronounced. Another surprising finding was that unsupervised pre-training on a larger set of 33 cancers gives significantly worse performance compared to pre-training on a smaller dataset of 7 cancers (including the target cancer). We posit that pre-training on a smaller, more focused dataset allows the feature extractor to make better use of the limited feature space to better discriminate between subtle differences in the input patch.