A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

TL;DR

This paper introduces a gradient analysis framework for language model training that simultaneously rewards good examples and penalizes bad ones, comparing methods like MLE, ExMATE, and DPO to improve output quality.

Contribution

It provides a unified gradient analysis approach to compare and enhance LM optimization methods involving both rewards and penalties.

Findings

ExMATE is a superior surrogate for MLE.

Combining DPO with ExMATE improves performance.

Experimental results show 5-7% statistical and 18% win rate improvements.

Abstract

Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.