OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction

TL;DR

OmniCharacter introduces a novel speech-language personality interaction model for immersive role-playing agents, enabling consistent vocal and dialogue traits with low latency, supported by a new extensive dataset and outperforming existing models.

Contribution

The paper presents OmniCharacter, the first seamless speech-language personality interaction model for RPAs, along with a new dataset OmniCharacter-10K, enhancing realism and immersion in AI role-playing scenarios.

Findings

Better response content and style compared to existing models

Achieves low latency of 289ms in responses

Supports diverse characters with rich contextual dialogues

Abstract

Role-Playing Agents (RPAs), benefiting from large language models, is an emerging interactive AI system that simulates roles or characters with diverse personalities. However, existing methods primarily focus on mimicking dialogues among roles in textual form, neglecting the role's voice traits (e.g., voice style and emotions) as playing a crucial effect in interaction, which tends to be more immersive experiences in realistic scenarios. Towards this goal, we propose OmniCharacter, a first seamless speech-language personality interaction model to achieve immersive RPAs with low latency. Specifically, OmniCharacter enables agents to consistently exhibit role-specific personality traits and vocal traits throughout the interaction, enabling a mixture of speech and language responses. To align the model with speech-language scenarios, we construct a dataset named OmniCharacter-10K, which involves more distinctive characters (20), richly contextualized multi-round dialogue (10K), and dynamic speech response (135K). Experimental results showcase that our method yields better responses in terms of both content and style compared to existing RPAs and mainstream speech-language models, with a response latency as low as 289ms. Code and dataset are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/OmniCharacter.