Learning Transferable Spatiotemporal Representations from Natural Script Knowledge

TL;DR

This paper introduces a novel pretraining task called TVTS that leverages natural speech transcripts to improve the transferability of spatiotemporal video representations, outperforming existing methods on various benchmarks.

Contribution

It proposes a new self-supervised pretext task using natural speech transcripts to enhance spatiotemporal video understanding without relying on descriptive captions.

Findings

+13.6% gains over VideoMAE on SSV2 with linear probing

Effective use of natural speech transcripts for video pretraining

Strong out-of-the-box performance on diverse benchmarks

Abstract

Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years. Despite some progress, existing methods are mostly limited to highly curated datasets (e.g., K400) and exhibit unsatisfactory out-of-the-box representations. We argue that it is due to the fact that they only capture pixel-level knowledge rather than spatiotemporal semantics, which hinders further progress in video understanding. Inspired by the great success of image-text pre-training (e.g., CLIP), we take the first step to exploit language semantics to boost transferable spatiotemporal representation learning. We introduce a new pretext task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR scripts by attending to learned video representations. We do not rely on descriptive captions and learn purely from video, i.e., leveraging the natural transcribed speech knowledge to provide noisy but useful semantics over time. Our method enforces the vision model to contextualize what is happening over time so that it can re-organize the narrative transcripts, and can seamlessly apply to large-scale uncurated video data in the real world. Our method demonstrates strong out-of-the-box spatiotemporal representations on diverse benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing. The code is available at https://github.com/TencentARC/TVTS.