Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

TL;DR

This paper introduces HD-VILA, a high-resolution, diversified video-language pre-training model that leverages a large-scale dataset to significantly improve performance across multiple visual and language understanding tasks.

Contribution

The paper presents a novel high-resolution, diversified dataset and a hybrid Transformer model for joint video and language pre-training, achieving state-of-the-art results.

Findings

40.4% relative increase in zero-shot MSR-VTT text-to-video retrieval R@1

55.4% improvement in high-resolution LSMDC retrieval

Effective in text-to-visual editing and super-resolution tasks

Abstract

We study joint video and language (VL) pre-training to enable cross-modality learning and benefit plentiful downstream VL tasks. Existing works either extract low-quality video features or learn limited text embedding, while neglecting that high-resolution videos and diversified semantics can significantly improve cross-modality learning. In this paper, we propose a novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-VILA) for many visual tasks. In particular, we collect a large dataset with two distinct properties: 1) the first high-resolution dataset including 371.5k hours of 720p videos, and 2) the most diversified dataset covering 15 popular YouTube categories. To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts. Our pre-training model achieves new state-of-the-art results in 10 VL understanding tasks and 2 more novel text-to-visual generation tasks. For example, we outperform SOTA models with relative increases of 40.4% R@1 in zero-shot MSR-VTT text-to-video retrieval task and 55.4% in high-resolution dataset LSMDC. The learned VL embedding is also effective in generating visually pleasing and semantically relevant results in text-to-visual editing and super-resolution tasks.