Baichuan-Omni-1.5 Technical Report

TL;DR

Baichuan-Omni-1.5 is a multimodal model that integrates understanding and audio generation, trained on 500 billion high-quality multimodal data, and outperforms many existing models in omni-modal tasks.

Contribution

The paper introduces a comprehensive multimodal training pipeline, a novel audio tokenizer, and a multi-stage training strategy for an advanced omni-modal model.

Findings

Achieves comparable performance to leading models on multimodal benchmarks.

Outperforms contemporary models in omni-modal understanding and generation.

Utilizes a large-scale high-quality multimodal dataset.

Abstract

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.