Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation

TL;DR

This paper introduces a multi-scale prototype learning approach for semantic segmentation that enhances interpretability by explicitly modeling diverse prototypical parts at multiple scales and grouping them sparsely.

Contribution

It proposes a novel multi-scale prototype layer and a sparse grouping mechanism to improve interpretability and understanding of multi-scale object representations in segmentation.

Findings

Increases model sparsity and interpretability.

Narrows performance gap with non-interpretable models.

Demonstrates effectiveness on Pascal VOC, Cityscapes, and ADE20K.

Abstract

Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. The model selects real patches seen during training as prototypes and constructs the dense prediction map based on the similarity between parts of the test image and the prototypes. This improves interpretability since the user can inspect the link between the predicted output and the patterns learned by the model in terms of prototypical information. In this paper, we propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. First, we introduce a prototype layer that explicitly learns diverse prototypical parts at several scales, leading to multi-scale representations in the prototype activation output. Then, we propose a sparse grouping mechanism that produces multi-scale sparse groups of these scale-specific prototypical parts. This provides a deeper understanding of the interactions between multi-scale object representations while enhancing the interpretability of the segmentation model. The experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models. Code is available at github.com/eceo-epfl/ScaleProtoSeg.