Multi-Party Conversational Agents: A Survey

TL;DR

This survey reviews recent advances in multi-party conversational agents, focusing on modeling social dynamics, semantic understanding, and future dialogue prediction, emphasizing the importance of Theory of Mind and multi-modal systems.

Contribution

It provides a comprehensive overview of methods and challenges in MPCAs, highlighting the role of Theory of Mind and multi-modal understanding as key research directions.

Findings

Theory of Mind is crucial for MPCAs

Multi-modal understanding is promising but underexplored

Recent progress includes large language models and classical ML methods

Abstract

Multi-party Conversational Agents (MPCAs) are systems designed to engage in dialogue with more than two participants simultaneously. Unlike traditional two-party agents, designing MPCAs faces additional challenges due to the need to interpret both utterance semantics and social dynamics. This survey explores recent progress in MPCAs by addressing three key questions: 1) Can agents model each participants' mental states? (State of Mind Modeling); 2) Can they properly understand the dialogue content? (Semantic Understanding); and 3) Can they reason about and predict future conversation flow? (Agent Action Modeling). We review methods ranging from classical machine learning to Large Language Models (LLMs) and multi-modal systems. Our analysis underscores Theory of Mind (ToM) as essential for building intelligent MPCAs and highlights multi-modal understanding as a promising yet underexplored direction. Finally, this survey offers guidance to future researchers on developing more capable MPCAs.