SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems

TL;DR

SpeechAgents introduces a multi-modal multi-agent system leveraging large language models to simulate human communication with rich emotions, rhythm, and scalability, enabling applications like drama and audio novel creation.

Contribution

The paper presents SpeechAgents, a novel multi-modal LLM-based multi-agent system for simulating human communication, along with Multi-Agent Tuning and a new benchmark for evaluation.

Findings

Effective simulation of human dialogue with authentic rhythm and emotions

Scalability to 25 agents in complex communication tasks

Successful application in drama creation and audio novels

Abstract

Human communication is a complex and diverse process that not only involves multiple factors such as language, commonsense, and cultural backgrounds but also requires the participation of multimodal information, such as speech. Large Language Model (LLM)-based multi-agent systems have demonstrated promising performance in simulating human society. Can we leverage LLM-based multi-agent systems to simulate human communication? However, current LLM-based multi-agent systems mainly rely on text as the primary medium. In this paper, we propose SpeechAgents, a multi-modal LLM based multi-agent system designed for simulating human communication. SpeechAgents utilizes multi-modal LLM as the control center for individual agent and employes multi-modal signals as the medium for exchanged messages among agents. Additionally, we propose Multi-Agent Tuning to enhance the multi-agent capabilities of LLM without compromising general abilities. To strengthen and evaluate the effectiveness of human communication simulation, we build the Human-Communication Simulation Benchmark. Experimental results demonstrate that SpeechAgents can simulate human communication dialogues with consistent content, authentic rhythm, and rich emotions and demonstrate excellent scalability even with up to 25 agents, which can apply to tasks such as drama creation and audio novels generation. Code and models will be open-sourced at https://github. com/0nutation/SpeechAgents