Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

TL;DR

This paper introduces CoDIT, a contrastive decoding method that enhances instruction-tuning datasets by disentangling instruction-following capabilities from pre-trained knowledge, leading to improved model performance.

Contribution

The paper proposes CoDIT, a novel contrastive decoding approach that isolates instruction-following behavior in generated responses, improving instruction-tuning datasets and model performance.

Findings

Models trained on CoDIT-generated datasets outperform those trained on directly generated responses.

Training on CoDIT datasets yields better performance than existing instruction-tuning datasets.

CoDIT can be interpreted as distilling instruction-tuning capabilities from parameter space to text space.

Abstract

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.