Learning Video Representations using Contrastive Bidirectional Transformer

TL;DR

This paper introduces a self-supervised contrastive learning method based on a bidirectional transformer architecture for video representations, improving performance on various downstream tasks through cross-modal training.

Contribution

It extends the BERT model to real-valued video feature sequences using noise contrastive estimation and incorporates cross-modal training with speech-derived text features.

Findings

Significant performance improvements on video classification, captioning, and segmentation.

Effective learning from sequences of visual features and speech-derived text.

Cross-modal training enhances representation quality.

Abstract

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.