Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction

TL;DR

This paper introduces 3DRotNet, a self-supervised learning method that predicts video rotations to learn meaningful spatiotemporal features, significantly improving action recognition accuracy on small datasets.

Contribution

It proposes a novel self-supervised pretext task of rotation prediction for videos, enabling effective learning of spatiotemporal features without labeled data.

Findings

Boosts UCF101 accuracy by 20.4% with pretraining

Enhances HMDB51 accuracy by 16.7% with pretraining

Outperforms existing self-supervised methods in video understanding

Abstract

The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper, we propose 3DRotNet: a fully self-supervised approach to learn spatiotemporal features from unlabeled videos. A set of rotations are applied to all videos, and a pretext task is defined as prediction of these rotations. When accomplishing this task, 3DRotNet is actually trained to understand the semantic concepts and motions in videos. In other words, it learns a spatiotemporal video representation, which can be transferred to improve video understanding tasks in small datasets. Our extensive experiments successfully demonstrate the effectiveness of the proposed framework on action recognition, leading to significant improvements over the state-of-the-art self-supervised methods. With the self-supervised pre-trained 3DRotNet from large datasets, the recognition accuracy is boosted up by 20.4% on UCF101 and 16.7% on HMDB51 respectively, compared to the models trained from scratch.