Motion Representation Using Residual Frames with 3D CNN

TL;DR

This paper introduces a novel method using residual frames as input to 3D CNNs for action recognition, achieving high accuracy without the need for computationally expensive optical flow streams.

Contribution

The authors propose replacing RGB frames with residual frames in 3D CNNs, significantly improving accuracy and eliminating the reliance on optical flow for motion feature extraction.

Findings

35.6% and 26.6% top-1 accuracy improvements on UCF101 and HMDB51.

Achieved state-of-the-art results in training from scratch.

Residual frames outperform RGB in capturing motion features.

Abstract

Recently, 3D convolutional networks (3D ConvNets) yield good performance in action recognition. However, optical flow stream is still needed to ensure better performance, the cost of which is very high. In this paper, we propose a fast but effective way to extract motion features from videos utilizing residual frames as the input data in 3D ConvNets. By replacing traditional stacked RGB frames with residual ones, 35.6% and 26.6% points improvements over top-1 accuracy can be obtained on the UCF101 and HMDB51 datasets when ResNet-18 models are trained from scratch. And we achieved the state-of-the-art results in this training mode. Analysis shows that better motion features can be extracted using residual frames compared to RGB counterpart. By combining with a simple appearance path, our proposal can be even better than some methods using optical flow streams.