VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

TL;DR

This paper introduces VAST, a large-scale omni-modality video dataset and foundation model that integrates vision, audio, subtitles, and text, enabling improved multi-modal understanding and tasks in videos.

Contribution

The paper presents VAST-27M, a novel large-scale omni-modality dataset, and a foundation model that jointly processes vision, audio, subtitles, and text for comprehensive video understanding.

Findings

VAST achieves 22 new state-of-the-art results on cross-modality benchmarks.

The dataset effectively supports training multi-modal video understanding models.

The VAST model demonstrates strong performance across retrieval, captioning, and QA tasks.

Abstract

Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.