Evolutionary Contrastive Distillation for Language Model Alignment

TL;DR

This paper introduces Evolutionary Contrastive Distillation, a method that enhances large language models' ability to follow complex instructions by generating and training on synthetic contrastive data, leading to significant performance improvements.

Contribution

It proposes a novel data generation technique that creates challenging contrastive examples for instruction tuning, improving model alignment with complex tasks.

Findings

7B model surpasses current SOTA 7B models in instruction following

Method achieves competitive performance with open-source 70B models

Contrastive data improves complex instruction adherence

Abstract

The ability of large language models (LLMs) to execute complex instructions is essential for their real-world applications. However, several recent studies indicate that LLMs struggle with challenging instructions. In this paper, we propose Evolutionary Contrastive Distillation (ECD), a novel method for generating high-quality synthetic preference data designed to enhance the complex instruction-following capability of language models. ECD generates data that specifically illustrates the difference between a response that successfully follows a set of complex instructions and a response that is high-quality, but nevertheless makes some subtle mistakes. This is done by prompting LLMs to progressively evolve simple instructions to more complex instructions. When the complexity of an instruction is increased, the original successful response to the original instruction becomes a "hard negative" response for the new instruction, mostly meeting requirements of the new instruction, but barely missing one or two. By pairing a good response with such a hard negative response, and employing contrastive learning algorithms such as DPO, we improve language models' ability to follow complex instructions. Empirically, we observe that our method yields a 7B model that exceeds the complex instruction-following performance of current SOTA 7B models and is competitive even with open-source 70B models.