Tarsier: Recipes for Training and Evaluating Large Video Description Models

TL;DR

Tarsier introduces a large-scale video-language model with a novel training approach that significantly outperforms existing open-source models in generating detailed video descriptions and achieves state-of-the-art results across multiple benchmarks.

Contribution

The paper presents Tarsier, a new large-scale video description model with a two-stage training process and introduces the DREAM-1K benchmark for evaluating video description quality.

Findings

Tarsier outperforms existing open-source models by 51.4% in human evaluations.

Tarsier achieves state-of-the-art results on nine public benchmarks.

Tarsier2 further improves performance with a 4.8% advantage over GPT-4o.

Abstract

Generating fine-grained video descriptions is a fundamental challenge in video understanding. In this work, we introduce Tarsier, a family of large-scale video-language models designed to generate high-quality video descriptions. Tarsier employs CLIP-ViT to encode frames separately and then uses an LLM to model temporal relationships. Despite its simple architecture, we demonstrate that with a meticulously designed two-stage training procedure, the Tarsier models exhibit substantially stronger video description capabilities than any existing open-source model, showing a $+51.4\%$ advantage in human side-by-side evaluation over the strongest model. Additionally, they are comparable to state-of-the-art proprietary models, with a $+12.3\%$ advantage against GPT-4V and a $-6.7\%$ disadvantage against Gemini 1.5 Pro. When upgraded to Tarsier2 by building upon SigLIP and Qwen2-7B, it further improves significantly with a $+4.8\%$ advantage against GPT-4o. Besides video description, Tarsier proves to be a versatile generalist model, achieving new state-of-the-art results across nine public benchmarks, including multi-choice VQA, open-ended VQA, and zero-shot video captioning. Our second contribution is the introduction of a new benchmark -- DREAM-1K (https://tarsier-vlm.github.io/) for evaluating video description models, consisting of a new challenging dataset featuring videos from diverse sources and varying complexity, along with an automatic method specifically designed to assess the quality of fine-grained video descriptions. We make our models and evaluation benchmark publicly available at https://github.com/bytedance/tarsier.