Scalable Vision Language Model Training via High Quality Data Curation

TL;DR

This paper presents SAIL-VL, a scalable vision language model series that achieves state-of-the-art performance through high-quality data curation, extensive pretraining, and effective supervised fine-tuning, setting new benchmarks in visual understanding.

Contribution

Introduction of SAIL-VL, a vision language model series that leverages high-quality data construction, large-scale pretraining, and advanced fine-tuning techniques for superior performance.

Findings

SAIL-Caption dataset has the highest quality among open-source datasets.

2B SAIL-VL model achieves top scores on 18 VLM benchmarks.

Scaling data size and complexity improves model performance significantly.

Abstract

In this paper, we introduce SAIL-VL (ScAlable Vision Language Model TraIning via High QuaLity Data Curation), an open-source vision language model (VLM) series achieving state-of-the-art (SOTA) performance in 2B and 8B parameters. The following three key improvements contribute to SAIL-VL's leading performance: (1) Scalable high-quality visual understanding data construction: We implement a data construction pipeline to enable hundred-million-scale high-quality recaption data annotation. The resulted dataset SAIL-Caption is validated to be of the highest data quality compared with opensource datasets. (2) Scalable Pretraining with High-Quality Visual Understanding Data: We scale SAIL-VL's pretraining budget up to 655B tokens and show that even a 2B VLM benefits from scaled up training data sizes, exhibiting logarithmic data size scaling laws in benchmark performance. (3) Scalable SFT via data quantity and complexity scaling: We curate a high-quality SFT dataset collection with leading data quantity scaling effectiveness and demonstrate that training with progressively higher-complexity data surpasses baseline one-stage training by a large margin. SAIL-VL series models achieve the highest average score in 18 widely used VLM benchmarks in our evaluation, with the 2B model takes the top position over VLMs of comparable sizes on OpenCompass 2024 (https://rank.opencompass.org.cn/leaderboard-multimodal), demonstrating robust visual comprehension abilities. SAIL-VL series models are released at HuggingFace (https://huggingface.co/BytedanceDouyinContent).