Attention in Attention: Modeling Context Correlation for Efficient Video Classification

TL;DR

This paper introduces an efficient attention-in-attention (AIA) mechanism that models the correlation between different context types in video classification, improving feature refinement with minimal computational overhead.

Contribution

It proposes a novel AIA module that incorporates context correlation into attention learning, enhancing video feature refinement without significant computational cost.

Findings

AIA improves classification accuracy on standard benchmarks.

The method introduces less than 0.02% additional computational cost.

Extensive experiments validate the effectiveness of AIA in various backbones.

Abstract

Attention mechanisms have significantly boosted the performance of video classification neural networks thanks to the utilization of perspective contexts. However, the current research on video attention generally focuses on adopting a specific aspect of contexts (e.g., channel, spatial/temporal, or global context) to refine the features and neglects their underlying correlation when computing attentions. This leads to incomplete context utilization and hence bears the weakness of limited performance improvement. To tackle the problem, this paper proposes an efficient attention-in-attention (AIA) method for element-wise feature refinement, which investigates the feasibility of inserting the channel context into the spatio-temporal attention learning module, referred to as CinST, and also its reverse variant, referred to as STinC. Specifically, we instantiate the video feature contexts as dynamics aggregated along a specific axis with global average and max pooling operations. The workflow of an AIA module is that the first attention block uses one kind of context information to guide the gating weights calculation of the second attention that targets at the other context. Moreover, all the computational operations in attention units act on the pooled dimension, which results in quite few computational cost increase ($<$0.02\%). To verify our method, we densely integrate it into two classical video network backbones and conduct extensive experiments on several standard video classification benchmarks. The source code of our AIA is available at \url{https://github.com/haoyanbin918/Attention-in-Attention}.