Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning

TL;DR

This paper introduces a novel self-supervised learning method called Video Playback Rate Perception (PRP) that enhances spatio-temporal video representations by leveraging playback rate classification and reconstruction, improving performance on action recognition and retrieval tasks.

Contribution

The paper proposes a new self-supervised approach, PRP, combining discriminative and generative models to better capture multi-scale temporal features in videos.

Findings

PRP outperforms existing self-supervised models on key video tasks.

The method effectively captures both long-term and short-term temporal features.

Experimental results demonstrate significant performance improvements.

Abstract

In self-supervised spatio-temporal representation learning, the temporal resolution and long-short term characteristics are not yet fully explored, which limits representation capabilities of learned models. In this paper, we propose a novel self-supervised method, referred to as video Playback Rate Perception (PRP), to learn spatio-temporal representation in a simple-yet-effective way. PRP roots in a dilated sampling strategy, which produces self-supervision signals about video playback rates for representation model learning. PRP is implemented with a feature encoder, a classification module, and a reconstructing decoder, to achieve spatio-temporal semantic retention in a collaborative discrimination-generation manner. The discriminative perception model follows a feature encoder to prefer perceiving low temporal resolution and long-term representation by classifying fast-forward rates. The generative perception model acts as a feature decoder to focus on comprehending high temporal resolution and short-term representation by introducing a motion-attention mechanism. PRP is applied on typical video target tasks including action recognition and video retrieval. Experiments show that PRP outperforms state-of-the-art self-supervised models with significant margins. Code is available at github.com/yuanyao366/PRP