ReviewInstruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Models

TL;DR

ReviewInstruct introduces an iterative multi-agent framework that synthesizes diverse, high-quality multi-turn dialogues for large language models, significantly improving their contextual coherence and instruction quality.

Contribution

It proposes a novel review-driven multi-agent method for generating multi-turn dialogue data, enhancing diversity and difficulty for LLM fine-tuning.

Findings

Achieves 2.9% improvement on MMLU-Pro

Achieves 2% improvement on MT-Bench

Demonstrates the effectiveness of review stages and multiple reviewers

Abstract

The effectiveness of large language models (LLMs) in conversational AI is hindered by their reliance on single-turn supervised fine-tuning (SFT) data, which limits contextual coherence in multi-turn dialogues. Existing methods for generating multi-turn dialogue data struggle to ensure both diversity and quality in instructions. To address this, we propose Review-Instruct, a novel framework that synthesizes multi-turn conversations through an iterative "Ask-Respond-Review" process involving three agent roles: a Candidate, multiple Reviewers, and a Chairman. The framework iteratively refines instructions by incorporating Reviewer feedback, enhancing dialogue diversity and difficulty. We construct a multi-turn dataset using the Alpaca dataset and fine-tune the LLaMA2-13B model. Evaluations on MT-Bench, MMLU-Pro, and Auto-Arena demonstrate significant improvements, achieving absolute gains of 2.9\% on MMLU-Pro and 2\% on MT-Bench compared to prior state-of-the-art models based on LLaMA2-13B. Ablation studies confirm the critical role of the Review stage and the use of multiple Reviewers in boosting instruction diversity and difficulty. Our work highlights the potential of review-driven, multi-agent frameworks for generating high-quality conversational data at scale.