UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

TL;DR

UniVL is a comprehensive pre-training model that unifies understanding and generation tasks in video-language processing, achieving state-of-the-art results across multiple benchmarks.

Contribution

The paper introduces UniVL, a novel unified pre-training framework with multiple objectives and strategies, enabling effective multimodal understanding and generation.

Findings

Achieves state-of-the-art results on five downstream tasks.

Effectively learns strong video-text representations.

Demonstrates the benefit of unified pre-training for multimodal tasks.

Abstract

With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.