Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

TL;DR

This paper introduces Pseudo-3D Residual Networks (P3D ResNet), a novel architecture that efficiently captures spatio-temporal features in videos by recycling 2D CNN components, achieving superior performance on multiple benchmarks.

Contribution

The paper proposes a new P3D ResNet architecture that combines 2D CNN modules to simulate 3D convolutions, reducing computational costs and improving video representation learning.

Findings

P3D ResNet outperforms 3D CNN and 2D CNN on Sports-1M dataset by 5.3% and 1.8%.

Pre-trained P3D ResNet demonstrates superior generalization across five benchmarks.

The architecture enhances structural diversity and deep learning capacity for video tasks.

Abstract

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3\times3\times3$ convolutions with $1\times3\times3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3\times1\times1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.