ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysis

TL;DR

ConSlide introduces a novel continual learning framework for Whole Slide Image analysis, effectively handling large images, hierarchical structures, and catastrophic forgetting through hierarchical transformers, rehearsal, and asynchronous updates.

Contribution

The paper presents ConSlide, a new continual learning approach with hierarchical transformers, a rehearsal method, and asynchronous updates tailored for WSI analysis.

Findings

Outperforms state-of-the-art methods on four public WSI datasets.

Achieves a better balance between overall performance and forgetting.

Effectively models hierarchical structure and large image data in continual learning.

Abstract

Whole slide image (WSI) analysis has become increasingly important in the medical imaging community, enabling automated and objective diagnosis, prognosis, and therapeutic-response prediction. However, in clinical practice, the ever-evolving environment hamper the utility of WSI analysis models. In this paper, we propose the FIRST continual learning framework for WSI analysis, named ConSlide, to tackle the challenges of enormous image size, utilization of hierarchical structure, and catastrophic forgetting by progressive model updating on multiple sequential datasets. Our framework contains three key components. The Hierarchical Interaction Transformer (HIT) is proposed to model and utilize the hierarchical structural knowledge of WSI. The Breakup-Reorganize (BuRo) rehearsal method is developed for WSI data replay with efficient region storing buffer and WSI reorganizing operation. The asynchronous updating mechanism is devised to encourage the network to learn generic and specific knowledge respectively during the replay stage, based on a nested cross-scale similarity learning (CSSL) module. We evaluated the proposed ConSlide on four public WSI datasets from TCGA projects. It performs best over other state-of-the-art methods with a fair WSI-based continual learning setting and achieves a better trade-off of the overall performance and forgetting on previous task