EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

TL;DR

EMOVA is a novel omni-modal model that enables large language models to perceive, generate, and interact with images, speech, and emotions end-to-end, achieving state-of-the-art results in vision-language and speech tasks.

Contribution

The paper introduces EMOVA, a comprehensive omni-modal framework that integrates vision, speech, and emotion understanding in large language models, with novel modules for semantic-acoustic disentanglement and style control.

Findings

Achieves state-of-the-art on vision-language benchmarks.

Supports omni-modal spoken dialogue with vivid emotions.

Enhances vision-language and speech abilities through semantic-acoustic alignment.

Abstract

GPT-4o, an omni-modal model that enables vocal conversations with diverse emotions and tones, marks a milestone for omni-modal foundation models. However, empowering Large Language Models to perceive and generate images, texts, and speeches end-to-end with publicly available data remains challenging for the open-source community. Existing vision-language models rely on external tools for speech processing, while speech-language models still suffer from limited or totally without vision-understanding capabilities. To address this gap, we propose the EMOVA (EMotionally Omni-present Voice Assistant), to enable Large Language Models with end-to-end speech abilities while maintaining the leading vision-language performance. With a semantic-acoustic disentangled speech tokenizer, we surprisingly notice that omni-modal alignment can further enhance vision-language and speech abilities compared with the bi-modal aligned counterparts. Moreover, a lightweight style module is introduced for the flexible speech style controls including emotions and pitches. For the first time, EMOVA achieves state-of-the-art performance on both the vision-language and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue with vivid emotions.