VS-CAM: Vertex Semantic Class Activation Mapping to Interpret Vision Graph Neural Network

TL;DR

This paper introduces VS-CAM, a novel visualization method for graph neural networks that produces more accurate and semantically meaningful heatmaps compared to traditional CNN-based CAMs.

Contribution

The paper proposes VS-CAM, a new visualization technique specifically designed for GCNs, which improves interpretability by generating precise semantic heatmaps.

Findings

VS-CAM produces heatmaps with regions that better match objects.

Quantitative results show VS-CAM outperforms existing methods.

Qualitative analysis confirms improved semantic clarity.

Abstract

Graph convolutional neural network (GCN) has drawn increasing attention and attained good performance in various computer vision tasks, however, there lacks a clear interpretation of GCN's inner mechanism. For standard convolutional neural networks (CNNs), class activation mapping (CAM) methods are commonly used to visualize the connection between CNN's decision and image region by generating a heatmap. Nonetheless, such heatmap usually exhibits semantic-chaos when these CAMs are applied to GCN directly. In this paper, we proposed a novel visualization method particularly applicable to GCN, Vertex Semantic Class Activation Mapping (VS-CAM). VS-CAM includes two independent pipelines to produce a set of semantic-probe maps and a semantic-base map, respectively. Semantic-probe maps are used to detect the semantic information from semantic-base map to aggregate a semantic-aware heatmap. Qualitative results show that VS-CAM can obtain heatmaps where the highlighted regions match the objects much more precisely than CNN-based CAM. The quantitative evaluation further demonstrates the superiority of VS-CAM.