OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

TL;DR

OpenChat introduces a novel framework for fine-tuning open-source language models using mixed-quality data by leveraging class-conditioned reinforcement learning, achieving superior performance without costly preference labeling.

Contribution

The paper proposes C-RLFT, a class-conditioned reinforcement learning approach that effectively utilizes mixed-quality data for open-source language model fine-tuning, avoiding expensive human preference labels.

Findings

OpenChat-13b outperforms other 13b open-source models on benchmarks.

The model generalizes well, surpassing the base model on AGIEval.

C-RLFT is lightweight and avoids costly human preference labeling.

Abstract

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.