M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance

TL;DR

M2-omni is an open-source omni-MLLM that supports multiple modalities with competitive performance to GPT-4o, utilizing a unified modeling framework and novel training strategies for balanced, comprehensive cross-modal understanding and generation.

Contribution

The paper introduces M2-omni, a versatile omni-MLLM with a unified multimodal framework and innovative training strategies to handle data disparities and enhance cross-modal capabilities.

Findings

Achieves performance comparable to GPT-4o across modalities.

Supports arbitrary combinations of audio, video, image, and text inputs.

Maintains strong language understanding while handling multimodal tasks.

Abstract

We present M2-omni, a cutting-edge, open-source omni-MLLM that achieves competitive performance to GPT-4o. M2-omni employs a unified multimodal sequence modeling framework, which empowers Large Language Models(LLMs) to acquire comprehensive cross-modal understanding and generation capabilities. Specifically, M2-omni can process arbitrary combinations of audio, video, image, and text modalities as input, generating multimodal sequences interleaving with audio, image, or text outputs, thereby enabling an advanced and interactive real-time experience. The training of such an omni-MLLM is challenged by significant disparities in data quantity and convergence rates across modalities. To address these challenges, we propose a step balance strategy during pre-training to handle the quantity disparities in modality-specific data. Additionally, a dynamically adaptive balance strategy is introduced during the instruction tuning stage to synchronize the modality-wise training progress, ensuring optimal convergence. Notably, we prioritize preserving strong performance on pure text tasks to maintain the robustness of M2-omni's language understanding capability throughout the training process. To our best knowledge, M2-omni is currently a very competitive open-source model to GPT-4o, characterized by its comprehensive modality and task support, as well as its exceptional performance. We expect M2-omni will advance the development of omni-MLLMs, thus facilitating future research in this domain.