Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training

TL;DR

This paper introduces Auto-captions on GIF, a large-scale dataset created from web data for pre-training models in video understanding and captioning, demonstrating its effectiveness across multiple downstream tasks.

Contribution

The paper presents a new large-scale, automatically generated video-sentence dataset and evaluates a Transformer-based pre-training approach for vision-language tasks.

Findings

Effective pre-training for video captioning

Strong generalization on MSR-VTT

Comparison with existing datasets shows advantages

Abstract

In this work, we present Auto-captions on GIF, which is a new large-scale pre-training dataset for generic video understanding. All video-sentence pairs are created by automatically extracting and filtering video caption annotations from billions of web pages. Auto-captions on GIF dataset can be utilized to pre-train the generic feature representation or encoder-decoder structure for video captioning, and other downstream tasks (e.g., sentence localization in videos, video question answering, etc.) as well. We present a detailed analysis of Auto-captions on GIF dataset in comparison to existing video-sentence datasets. We also provide an evaluation of a Transformer-based encoder-decoder structure for vision-language pre-training, which is further adapted to video captioning downstream task and yields the compelling generalizability on MSR-VTT. The dataset is available at \url{http://www.auto-video-captions.top/2020/dataset}.