Noise Contrastive Alignment of Language Models with Explicit Rewards

TL;DR

This paper introduces a novel framework using Noise Contrastive Estimation for aligning language models with explicit reward data, improving over existing methods like DPO especially in complex reasoning tasks.

Contribution

The paper presents NCA and InfoNCA algorithms that directly extract language model policies from explicit reward data, extending and unifying current alignment approaches.

Findings

InfoNCA/NCA outperform preference baselines with reward data

NCA prevents likelihood decrease by optimizing absolute likelihood

NCA significantly improves complex reasoning tasks like math and coding

Abstract

User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses. In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8*7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.