Improving Model Alignment Through Collective Intelligence of Open-Source LLMS

TL;DR

This paper introduces MoAA, a novel method leveraging collective intelligence of multiple language models to generate high-quality data for aligning open-source LLMs, improving performance and enabling self-improvement without external supervision.

Contribution

We propose MoAA, a scalable approach that combines various models' strengths to enhance data quality for model alignment, surpassing single-model methods and enabling self-improvement.

Findings

Significant performance gains on benchmark tasks.

Models fine-tuned on MoAA data outperform initial capabilities.

Demonstrates self-improvement without external data.

Abstract

Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality human-labeled data. Constructing such datasets is often expensive and hard to scale, and may face potential limitations on diversity and generalization. To address these challenges, we introduce Mixture of Agents Alignment (MoAA), that leverages the collective strengths of various language models to provide high-quality data for model alignment. By employing MoAA, we enhance both supervised fine-tuning and preference optimization, leading to improved performance compared to using a single model alone to generate alignment data (e.g. using GPT-4o alone). Evaluation results show that our approach can improve win rate of LLaMA-3.1-8B-Instruct from 19.5 to 48.3 on Arena-Hard and from 22.33 to 57.23 on AlpacaEval2, highlighting a promising direction for model alignment through this new scalable and diverse synthetic data recipe. Furthermore, we demonstrate that MoAA enables a self-improvement pipeline, where models finetuned on MoA-generated data surpass their own initial capabilities, providing evidence that our approach can push the frontier of open-source LLMs without reliance on stronger external supervision. Data and code will be released.