ConvNet Architecture Search for Spatiotemporal Feature Learning

TL;DR

This paper introduces an empirical search for ConvNet architectures tailored for spatiotemporal video features, resulting in a 3D Residual ConvNet that surpasses previous models in accuracy and efficiency.

Contribution

It presents a novel architecture search method specifically for spatiotemporal ConvNets, leading to a more effective and efficient 3D Residual ConvNet for video analysis.

Findings

Outperforms C3D on multiple video benchmarks

Twice as fast at inference compared to previous models

Half the model size of comparable architectures

Abstract

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.