ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition

TL;DR

This paper introduces ST-ABN, a novel method for visual explanation in video recognition that considers both spatial and temporal information, enhancing interpretability and recognition accuracy.

Contribution

The paper proposes ST-ABN, a spatio-temporal attention mechanism that improves visual explanation and recognition performance in video analysis.

Findings

Enables visual explanation considering spatial and temporal info

Improves recognition accuracy on Something-Something datasets

Demonstrates better interpretability of video decision processes

Abstract

It is difficult for people to interpret the decision-making in the inference process of deep neural networks. Visual explanation is one method for interpreting the decision-making of deep learning. It analyzes the decision-making of 2D CNNs by visualizing an attention map that highlights discriminative regions. Visual explanation for interpreting the decision-making process in video recognition is more difficult because it is necessary to consider not only spatial but also temporal information, which is different from the case of still images. In this paper, we propose a visual explanation method called spatio-temporal attention branch network (ST-ABN) for video recognition. It enables visual explanation for both spatial and temporal information. ST-ABN acquires the importance of spatial and temporal information during network inference and applies it to recognition processing to improve recognition performance and visual explainability. Experimental results with Something-Something datasets V1 \& V2 demonstrated that ST-ABN enables visual explanation that takes into account spatial and temporal information simultaneously and improves recognition performance.