LLMs Get Lost In Multi-Turn Conversation

TL;DR

This paper reveals that large language models perform significantly worse in multi-turn conversations compared to single-turn settings, often getting lost and failing to recover from early mistakes, which impacts their usefulness in interactive tasks.

Contribution

The study provides large-scale empirical evidence showing performance drops in multi-turn LLM conversations and analyzes the causes of unreliability and assumption-driven errors.

Findings

Average 39% performance drop in multi-turn settings

Performance degradation due to unreliability and assumption errors

LLMs often get lost and fail to recover in extended conversations

Abstract

Large Language Models (LLMs) are conversational interfaces. As such, LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange. Although analysis of LLM conversation logs has confirmed that underspecification occurs frequently in user instructions, LLM evaluation has predominantly focused on the single-turn, fully-specified instruction setting. In this work, we perform large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. Analysis of 200,000+ simulated conversations decomposes the performance degradation into two components: a minor loss in aptitude and a significant increase in unreliability. We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that *when LLMs take a wrong turn in a conversation, they get lost and do not recover*.