Language Model Decoding as Direct Metrics Optimization

TL;DR

This paper introduces a novel decoding method for language models that optimizes multiple metrics simultaneously, resulting in texts that better align with human preferences and improve upon existing decoding techniques.

Contribution

It formulates decoding as a metrics optimization problem with an analytical solution, guaranteeing improved perplexity and better alignment with human texts.

Findings

Outperforms baseline methods in metrics alignment

Achieves higher human evaluation scores

Demonstrates effectiveness across domains and model scales

Abstract

Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive texts which are often disjunctive in discourse, while search-based methods maintain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across a broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts measured by multiple metrics of desired aspects simultaneously. The resulting decoding distribution enjoys an analytical solution that scales the input language model distribution via a sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importance-Resampling technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human evaluation over strong baselines.