Learning Nuclei Representations with Masked Image Modelling

TL;DR

This paper demonstrates that masked image modeling can effectively learn rich, nuclei-level semantic representations in histopathology images, improving downstream cell classification accuracy.

Contribution

The study introduces a novel nuclei-focused patching and positional encoding scheme for masked image modeling in medical images, enhancing nuclei representation learning.

Findings

Achieved over 5% improvement in cell classification accuracy on PanNuke.

Effectively captures nuclei-level semantic information in histopathology images.

Generalizes well to downstream classification tasks.

Abstract

Masked image modelling (MIM) is a powerful self-supervised representation learning paradigm, whose potential has not been widely demonstrated in medical image analysis. In this work, we show the capacity of MIM to capture rich semantic representations of Haemotoxylin & Eosin (H&E)-stained images at the nuclear level. Inspired by Bidirectional Encoder representation from Image Transformers (BEiT), we split the images into smaller patches and generate corresponding discrete visual tokens. In addition to the regular grid-based patches, typically used in visual Transformers, we introduce patches of individual cell nuclei. We propose positional encoding of the irregular distribution of these structures within an image. We pre-train the model in a self-supervised manner on H&E-stained whole-slide images of diffuse large B-cell lymphoma, where cell nuclei have been segmented. The pre-training objective is to recover the original discrete visual tokens of the masked image on the one hand, and to reconstruct the visual tokens of the masked object instances on the other. Coupling these two pre-training tasks allows us to build powerful, context-aware representations of nuclei. Our model generalizes well and can be fine-tuned on downstream classification tasks, achieving improved cell classification accuracy on PanNuke dataset by more than 5% compared to current instance segmentation methods.