Object-ABN: Learning to Generate Sharp Attention Maps for Action Recognition

TL;DR

Object-ABN enhances attention maps for action recognition by integrating instance segmentation, resulting in sharper, more interpretable visual explanations and improved classification accuracy on benchmark datasets.

Contribution

It introduces a novel mask loss and additional techniques to produce clearer attention maps, advancing the interpretability and performance of attention-based action recognition models.

Findings

Generated attention maps are significantly sharper and more interpretable.

The method improves classification accuracy on UCF101 and SSv2 datasets.

Quantitative metrics confirm the enhanced quality of attention maps.

Abstract

In this paper we propose an extension of the Attention Branch Network (ABN) by using instance segmentation for generating sharper attention maps for action recognition. Methods for visual explanation such as Grad-CAM usually generate blurry maps which are not intuitive for humans to understand, particularly in recognizing actions of people in videos. Our proposed method, Object-ABN, tackles this issue by introducing a new mask loss that makes the generated attention maps close to the instance segmentation result. Further the PC loss and multiple attention maps are introduced to enhance the sharpness of the maps and improve the performance of classification. Experimental results with UCF101 and SSv2 shows that the generated maps by the proposed method are much clearer qualitatively and quantitatively than those of the original ABN.