JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

TL;DR

JavisGPT is a novel unified multimodal large language model that achieves coherent audio-video comprehension and generation through a specialized architecture and extensive instruction tuning, outperforming existing models in complex scenarios.

Contribution

The paper introduces JavisGPT, the first unified model for joint audio-video understanding and generation, with a new architecture and large-scale instruction dataset.

Findings

Outperforms existing MLLMs on JAV benchmarks.

Achieves temporally coherent video-audio understanding.

Effective three-stage training pipeline enhances multimodal capabilities.

Abstract

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.