Towards Explainable End-to-End Prostate Cancer Relapse Prediction from H&E Images Combining Self-Attention Multiple Instance Learning with a Recurrent Neural Network

TL;DR

This paper introduces eCaReNet, an explainable deep learning model for prostate cancer relapse prediction from histopathology images that achieves high accuracy without requiring detailed annotations, and provides interpretability through attention mechanisms.

Contribution

The study presents a novel end-to-end model combining self-attention multiple instance learning with RNNs for relapse prediction, enabling interpretability without strong annotations.

Findings

Achieved a cumulative dynamic AUC of 0.78 on validation, comparable to expert pathologists.

Model outputs include survival curves, risk scores, and patient grouping.

Attention weights highlight malignant patches as more influential, offering interpretability.

Abstract

Clinical decision support for histopathology image data mainly focuses on strongly supervised annotations, which offers intuitive interpretability, but is bound by expert performance. Here, we propose an explainable cancer relapse prediction network (eCaReNet) and show that end-to-end learning without strong annotations offers state-of-the-art performance while interpretability can be included through an attention mechanism. On the use case of prostate cancer survival prediction, using 14,479 images and only relapse times as annotations, we reach a cumulative dynamic AUC of 0.78 on a validation set, being on par with an expert pathologist (and an AUC of 0.77 on a separate test set). Our model is well-calibrated and outputs survival curves as well as a risk score and group per patient. Making use of the attention weights of a multiple instance learning layer, we show that malignant patches have a higher influence on the prediction than benign patches, thus offering an intuitive interpretation of the prediction. Our code is available at www.github.com/imsb-uke/ecarenet.