Pay Attention with Focus: A Novel Learning Scheme for Classification of Whole Slide Images

TL;DR

This paper introduces a novel two-stage deep learning approach for classifying whole slide images by extracting representative patches, encoding them, and applying an attention-based scheme to improve diagnosis accuracy.

Contribution

It proposes a new attention-weighted averaging scheme with a trainable focal factor for WSI classification, addressing large image size challenges.

Findings

Model achieves robust classification performance.

Attention scheme improves focus on relevant patches.

Effective for primary diagnosis of WSIs.

Abstract

Deep learning methods such as convolutional neural networks (CNNs) are difficult to directly utilize to analyze whole slide images (WSIs) due to the large image dimensions. We overcome this limitation by proposing a novel two-stage approach. First, we extract a set of representative patches (called mosaic) from a WSI. Each patch of a mosaic is encoded to a feature vector using a deep network. The feature extractor model is fine-tuned using hierarchical target labels of WSIs, i.e., anatomic site and primary diagnosis. In the second stage, a set of encoded patch-level features from a WSI is used to compute the primary diagnosis probability through the proposed Pay Attention with Focus scheme, an attention-weighted averaging of predicted probabilities for all patches of a mosaic modulated by a trainable focal factor. Experimental results show that the proposed model can be robust, and effective for the classification of WSIs.