BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues

TL;DR

This paper introduces a novel LLM-based evaluation method for multi-turn dialogues, demonstrating GPT-4's superior ability to generate human-like conversations and providing a benchmark for assessing LLM conversational skills.

Contribution

It proposes an LLM-based evaluation approach for multi-turn dialogues and benchmarks GPT-4's performance against other models using real-world dialogue data.

Findings

GPT-4 generates high-quality, human-like multi-turn dialogues.

Discriminator-based methods struggle to distinguish GPT-4 dialogues from human ones.

Other LLMs have limitations in instruction-following and maintaining dialogue quality.

Abstract

Interacting with human via high-quality multi-turn dialogues is a key feature of large language models (LLMs). However, human-based evaluation of such capability involves intensive manual labor. This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting, through an LLM-based approach. We start from real-world human dialogues and keep the very first utterances as the ChatSEED. Then we prompt LLMs to generate a full multi-turn dialogue (tens of utterances) based on the ChatSEED, utterance by utterance. Finally, we adopt state-of-the-art LLMs (GPT-4, \etc) as the judge to evaluate the generated dialogues. With different evaluation protocols, we come to substantially identical conclusions. We find that GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts. It's difficult for a discriminator to distinguish between GPT-4 generated dialogues and human dialogues. In contrast, other LLMs struggle to generate multi-turn dialogues of satisfactory quality due to poor instruction-following capability, tendency to generate lengthy utterances, or limited general capability. All data and codes will be provided in https://github.com/open-compass/BotChat/ and we hope they can serve as a valuable resource for evaluating multi-turn chatting capabilities of LLMs.