MPCEval: A Benchmark for Multi-Party Conversation Generation

TL;DR

MPCEval introduces a comprehensive, reference-free benchmarking suite for multi-party conversation generation, addressing unique challenges and providing nuanced evaluation metrics that reveal detailed model behaviors.

Contribution

It presents MPCEval, a novel evaluation framework with quantitative metrics for multi-party dialogue, enabling detailed analysis beyond single-score assessments.

Findings

Models show systematic participation and content progression patterns.

Evaluation objectives influence model assessment outcomes.

Single-score metrics obscure fundamental differences in multi-party conversations.

Abstract

Multi-party conversation generation, such as smart reply and collaborative assistants, is an increasingly important capability of generative AI, yet its evaluation remains a critical bottleneck. Compared to two-party dialogue, multi-party settings introduce distinct challenges, including complex turn-taking, role-dependent speaker behavior, long-range conversational structure, and multiple equally valid continuations. Accordingly, we introduce MPCEval, a task-aware evaluation and benchmarking suite for multi-party conversation generation. MPCEval decomposes generation quality into speaker modeling, content quality, and speaker--content consistency, and explicitly distinguishes local next-turn prediction from global full-conversation generation. It provides novel, quantitative, reference-free, and reproducible metrics that scale across datasets and models. We apply MPCEval to diverse public and real-world datasets and evaluate modern generation methods alongside human-authored conversations. The results reveal systematic, dimension-specific model characteristics in participation balance, content progression and novelty, and speaker--content consistency, demonstrating that evaluation objectives critically shape model assessment and that single-score evaluation obscures fundamental differences in multi-party conversational behavior. The implementation of MPCEval and the associated evaluation code are publicly available at https://github.com/Owen-Yang-18/MPCEval.