MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation

TL;DR

MARS-Bench is a new benchmark designed to evaluate large language models' robustness in multi-turn, real-world dialogue scenarios, highlighting their strengths and weaknesses in handling complex, long conversations.

Contribution

The paper introduces MARS-Bench, a realistic multi-turn dialogue benchmark based on sports commentary, to assess LLMs' performance on complex dialogue tasks and analyze their limitations.

Findings

Closed-source LLMs outperform open-source models.

Explicit reasoning improves robustness in long dialogues.

LLMs struggle with motivation transfer and cross-turn dependency.

Abstract

Large Language Models (\textbf{LLMs}), e.g. ChatGPT, have been widely adopted in real-world dialogue applications. However, LLMs' robustness, especially in handling long complex dialogue sessions, including frequent motivation transfer, sophisticated cross-turn dependency, is criticized all along. Nevertheless, no existing benchmarks can fully reflect these weaknesses. We present \textbf{MARS-Bench}, a \textbf{M}ulti-turn \textbf{A}thletic \textbf{R}eal-world \textbf{S}cenario Dialogue \textbf{Bench}mark, designed to remedy the gap. MARS-Bench is constructed from play-by-play text commentary so to feature realistic dialogues specifically designed to evaluate three critical aspects of multi-turn conversations: Ultra Multi-turn, Interactive Multi-turn, and Cross-turn Tasks. Extensive experiments on MARS-Bench also reveal that closed-source LLMs significantly outperform open-source alternatives, explicit reasoning significantly boosts LLMs' robustness on handling long complex dialogue sessions, and LLMs indeed face significant challenges when handling motivation transfer and sophisticated cross-turn dependency. Moreover, we provide mechanistic interpretability on how attention sinks due to special tokens lead to LLMs' performance degradation when handling long complex dialogue sessions based on attention visualization experiment in Qwen2.5-7B-Instruction.