A Real-time Action Representation with Temporal Encoding and Deep Compression

TL;DR

This paper introduces T-C3D, a real-time 3D convolutional neural network for video action recognition that combines hierarchical temporal encoding and deep compression to achieve high accuracy and speed in practical applications.

Contribution

The paper presents a novel real-time convolutional architecture, T-C3D, integrating residual 3D CNN, temporal encoding, and deep compression for efficient and effective action recognition.

Findings

Achieves 5.4% higher accuracy on UCF101 benchmark.

Runs twice as fast as comparable real-time methods.

Model size is less than 5MB, enabling deployment on resource-limited devices.

Abstract

Deep neural networks have achieved remarkable success for video-based action recognition. However, most of existing approaches cannot be deployed in practice due to the high computational cost. To address this challenge, we propose a new real-time convolutional architecture, called Temporal Convolutional 3D Network (T-C3D), for action representation. T-C3D learns video action representations in a hierarchical multi-granularity manner while obtaining a high process speed. Specifically, we propose a residual 3D Convolutional Neural Network (CNN) to capture complementary information on the appearance of a single frame and the motion between consecutive frames. Based on this CNN, we develop a new temporal encoding method to explore the temporal dynamics of the whole video. Furthermore, we integrate deep compression techniques with T-C3D to further accelerate the deployment of models via reducing the size of the model. By these means, heavy calculations can be avoided when doing the inference, which enables the method to deal with videos beyond real-time speed while keeping promising performance. Our method achieves clear improvements on UCF101 action recognition benchmark against state-of-the-art real-time methods by 5.4% in terms of accuracy and 2 times faster in terms of inference speed with a less than 5MB storage model. We validate our approach by studying its action representation performance on four different benchmarks over three different tasks. Extensive experiments demonstrate comparable recognition performance to the state-of-the-art methods. The source code and the pre-trained models are publicly available at https://github.com/tc3d.