Language Generation with Strictly Proper Scoring Rules

TL;DR

This paper introduces a method to adapt strictly proper scoring rules for language generation, replacing the traditional log-likelihood loss, leading to improved model performance, especially in large language models.

Contribution

It proposes a novel strategy to incorporate non-local proper scoring rules into language modeling, enabling the use of Brier and Spherical scores as training objectives.

Findings

Replacing log-likelihood with proper scoring rules improves generation quality.

The approach scales effectively to large language models like LLaMA-7B and 13B.

Substituting the loss function yields significant performance gains without hyperparameter tuning.

Abstract

Language generation based on maximum likelihood estimation (MLE) has become the fundamental approach for text generation. Maximum likelihood estimation is typically performed by minimizing the log-likelihood loss, also known as the logarithmic score in statistical decision theory. The logarithmic score is strictly proper in the sense that it encourages honest forecasts, where the expected score is maximized only when the model reports true probabilities. Although many strictly proper scoring rules exist, the logarithmic score is the only local scoring rule among them that depends exclusively on the probability of the observed sample, making it capable of handling the exponentially large sample space of natural text. In this work, we propose a straightforward strategy for adapting scoring rules to language generation, allowing for language modeling with any non-local scoring rules. Leveraging this strategy, we train language generation models using two classic strictly proper scoring rules, the Brier score and the Spherical score, as alternatives to the logarithmic score. Experimental results indicate that simply substituting the loss function, without adjusting other hyperparameters, can yield substantial improvements in model's generation capabilities. Moreover, these improvements can scale up to large language models (LLMs) such as LLaMA-7B and LLaMA-13B. Source code: \url{https://github.com/shaochenze/ScoringRulesLM}.