Semantic Adversarial Network with Multi-scale Pyramid Attention for Video Classification

TL;DR

This paper introduces a novel two-stream deep learning framework for video classification that leverages multi-scale pyramid attention and semantic adversarial learning to improve accuracy using only RGB frames, avoiding optical flow.

Contribution

It proposes a new framework combining MPA and SAL modules to enhance feature extraction and semantic understanding in video classification tasks.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively captures global and local features from RGB frames.

Reduces reliance on computationally expensive optical flow.

Abstract

Two-stream architecture have shown strong performance in video classification task. The key idea is to learn spatio-temporal features by fusing convolutional networks spatially and temporally. However, there are some problems within such architecture. First, it relies on optical flow to model temporal information, which are often expensive to compute and store. Second, it has limited ability to capture details and local context information for video data. Third, it lacks explicit semantic guidance that greatly decrease the classification performance. In this paper, we proposed a new two-stream based deep framework for video classification to discover spatial and temporal information only from RGB frames, moreover, the multi-scale pyramid attention (MPA) layer and the semantic adversarial learning (SAL) module is introduced and integrated in our framework. The MPA enables the network capturing global and local feature to generate a comprehensive representation for video, and the SAL can make this representation gradually approximate to the real video semantics in an adversarial manner. Experimental results on two public benchmarks demonstrate our proposed methods achieves state-of-the-art results on standard video datasets.