ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing

TL;DR

This paper introduces ActorMind, a framework for speech role-playing that emulates human actor reasoning, supported by a hierarchical benchmark, ActorMindBench, to advance human-machine interaction and sociological research.

Contribution

It presents a novel multi-agent reasoning framework and a comprehensive benchmark for speech role-playing, addressing the gap in speech-based role-playing models.

Findings

ActorMind improves speech role-playing performance.

ActorMindBench provides extensive hierarchical data for evaluation.

The framework emulates human actor reasoning processes.

Abstract

Role-playing has garnered rising attention as it provides a strong foundation for human-machine interaction and facilitates sociological research. However, current work is confined to textual modalities, neglecting speech, which plays a predominant role in daily life, thus limiting genuine role-playing. To bridge this gap, we conceptualize and benchmark speech role-playing through ActorMindBench, and we present a corresponding reasoning framework, called ActorMind. Specifically, (1) Speech Role-Playing enables models to deliver spontaneous responses with personalized verbal traits based on their role, the scene, and spoken dialogue. (2) ActorMindBench is a hierarchical benchmark comprises Utterance-Level content with 7,653 utterances, Scene-Level content with 313 scenes, and Role-Level content with 6 roles. (3) ActorMind is an off-the-shelf, multi-agent, chain-of-though style reasoning framework that emulates how human actors perform in theaters. Concretely, ActorMind first reads its assigned role description via Eye Agent, then comprehends emotional cues within contextual spoken dialogues through Ear Agent. Subsequently, Brain Agent generates a descriptive emotional state, and finally, Mouth Agent delivers the scripts infused with corresponding emotion state. Experimental results demonstrate the effectiveness of ActorMind in enhancing speech role-playing.