SSAN: Separable Self-Attention Network for Video Representation Learning

TL;DR

The paper introduces SSAN, a novel video representation learning model that sequentially models spatial and temporal correlations using a separable self-attention module, leading to improved performance on multiple datasets.

Contribution

It proposes a separable self-attention module that models spatial and temporal dependencies sequentially, enhancing video understanding.

Findings

Outperforms state-of-the-art on Something-Something and Kinetics-400.

Achieves better results with shallower networks and fewer modalities.

Improves video retrieval performance on MSR-VTT and Youcook2.

Abstract

Self-attention has been successfully applied to video representation learning due to the effectiveness of modeling long range dependencies. Existing approaches build the dependencies merely by computing the pairwise correlations along spatial and temporal dimensions simultaneously. However, spatial correlations and temporal correlations represent different contextual information of scenes and temporal reasoning. Intuitively, learning spatial contextual information first will benefit temporal modeling. In this paper, we propose a separable self-attention (SSA) module, which models spatial and temporal correlations sequentially, so that spatial contexts can be efficiently used in temporal modeling. By adding SSA module into 2D CNN, we build a SSA network (SSAN) for video representation learning. On the task of video action recognition, our approach outperforms state-of-the-art methods on Something-Something and Kinetics-400 datasets. Our models often outperform counterparts with shallower network and fewer modalities. We further verify the semantic learning ability of our method in visual-language task of video retrieval, which showcases the homogeneity of video representations and text embeddings. On MSR-VTT and Youcook2 datasets, video representations learnt by SSA significantly improve the state-of-the-art performance.