Comparison of Spatiotemporal Networks for Learning Video Related Tasks

TL;DR

This paper compares different spatiotemporal neural network architectures on a new MNIST-based video dataset, revealing how design choices affect learning of video-related tasks like classification and speed estimation.

Contribution

It introduces a new dataset and provides an empirical analysis of how various spatiotemporal models perform across different video tasks.

Findings

Model performance varies significantly with task and architecture.

Design choices greatly influence the ability to learn spatiotemporal features.

Complex relationships exist between spatial and temporal processing in video models.

Abstract

Many methods for learning from video sequences involve temporally processing 2D CNN features from the individual frames or directly utilizing 3D convolutions within high-performing 2D CNN architectures. The focus typically remains on how to incorporate the temporal processing within an already stable spatial architecture. This work constructs an MNIST-based video dataset with parameters controlling relevant facets of common video-related tasks: classification, ordering, and speed estimation. Models trained on this dataset are shown to differ in key ways depending on the task and their use of 2D convolutions, 3D convolutions, or convolutional LSTMs. An empirical analysis indicates a complex, interdependent relationship between the spatial and temporal dimensions with design choices having a large impact on a network's ability to learn the appropriate spatiotemporal features.