From Explanations to Segmentation: Using Explainable AI for Image Segmentation

TL;DR

This paper proposes a novel weakly supervised image segmentation method that leverages explainable AI techniques, specifically Layer-wise Relevance Propagation, to generate pixel-wise masks from image-level labels, reducing the need for extensive manual labeling.

Contribution

It introduces a new approach that extracts segmentation masks from classification models using explainability methods, achieving comparable results to traditional segmentation networks with less labeling effort.

Findings

Achieves similar segmentation quality to U-Net.

Enables training with only image-level labels.

Reduces manual pixel-wise annotation effort.

Abstract

The new era of image segmentation leveraging the power of Deep Neural Nets (DNNs) comes with a price tag: to train a neural network for pixel-wise segmentation, a large amount of training samples has to be manually labeled on pixel-precision. In this work, we address this by following an indirect solution. We build upon the advances of the Explainable AI (XAI) community and extract a pixel-wise binary segmentation from the output of the Layer-wise Relevance Propagation (LRP) explaining the decision of a classification network. We show that we achieve similar results compared to an established U-Net segmentation architecture, while the generation of the training data is significantly simplified. The proposed method can be trained in a weakly supervised fashion, as the training samples must be only labeled on image-level, at the same time enabling the output of a segmentation mask. This makes it especially applicable to a wider range of real applications where tedious pixel-level labelling is often not possible.