Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models

TL;DR

This paper investigates the trade-off between instruction following and faithfulness in language models, proposing a Rejection Sampling method that improves alignment with less data, revealing insights into training objectives.

Contribution

It introduces ReSet, a rejection sampling-based approach that enhances language model alignment, outperforming traditional multi-task learning methods with less data.

Findings

ReSet outperforms vanilla multi-task learning.

Training with less high-quality data yields better results.

Identifies a fundamental trade-off in alignment objectives.

Abstract

Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instruction-tuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Self-instruction Tuning (ReSet), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training ReSet with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.