TEINet: Towards an Efficient Architecture for Video Recognition

TL;DR

TEINet introduces an efficient, plug-in temporal module for 2D CNNs that enhances motion features and captures temporal context, significantly improving video action recognition while reducing computational costs.

Contribution

The paper proposes the TEI Module, a novel two-stage temporal modeling approach that is easily integrated into existing 2D CNNs to improve efficiency and effectiveness in video recognition.

Findings

Achieves competitive accuracy on multiple benchmarks.

Reduces computational cost compared to 3D CNNs.

Demonstrates flexible and effective temporal feature learning.

Abstract

Efficiency is an important issue in designing video architectures for action recognition. 3D CNNs have witnessed remarkable progress in action recognition from videos. However, compared with their 2D counterparts, 3D convolutions often introduce a large amount of parameters and cause high computational cost. To relieve this problem, we propose an efficient temporal module, termed as Temporal Enhancement-and-Interaction (TEI Module), which could be plugged into the existing 2D CNNs (denoted by TEINet). The TEI module presents a different paradigm to learn temporal features by decoupling the modeling of channel correlation and temporal interaction. First, it contains a Motion Enhanced Module (MEM) which is to enhance the motion-related features while suppress irrelevant information (e.g., background). Then, it introduces a Temporal Interaction Module (TIM) which supplements the temporal contextual information in a channel-wise manner. This two-stage modeling scheme is not only able to capture temporal structure flexibly and effectively, but also efficient for model inference. We conduct extensive experiments to verify the effectiveness of TEINet on several benchmarks (e.g., Something-Something V1&V2, Kinetics, UCF101 and HMDB51). Our proposed TEINet can achieve a good recognition accuracy on these datasets but still preserve a high efficiency.