EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling

TL;DR

This paper introduces Earth Mover Distance Optimization (EMO), a novel training method for autoregressive language models that addresses limitations of maximum likelihood estimation by better aligning model and human language distributions, leading to improved performance.

Contribution

The paper proposes EMO, a new training approach based on earth mover distance, with a feasible upper bound for efficient training, improving language modeling and downstream task performance.

Findings

EMO outperforms MLE in language modeling across various domains.

EMO enhances downstream task performance with minimal fine-tuning.

EMO serves as an effective lightweight calibration method for large-scale models.

Abstract

Neural language models are probabilistic models of human text. They are predominantly trained using maximum likelihood estimation (MLE), which is equivalent to minimizing the forward cross-entropy between the empirical data distribution and the model distribution. However, various degeneration phenomena are still widely observed when decoding from the distributions learned by such models. We establish that the forward cross-entropy is suboptimal as a distance metric for aligning human and model distribution due to its (1) recall-prioritization (2) negative diversity ignorance and (3) train-test mismatch. In this paper, we propose Earth Mover Distance Optimization (EMO) for auto-regressive language modeling. EMO capitalizes on the inherent properties of earth mover distance to address the aforementioned challenges. Due to the high complexity of direct computation, we further introduce a feasible upper bound for EMO to ease end-to-end training. Upon extensive evaluation of language models trained using EMO and MLE. We find that EMO demonstrates a consistently better language modeling performance than MLE across domains. Moreover, EMO demonstrates noteworthy enhancements in downstream performance with minimal fine-tuning on merely 25,000 sentences. This highlights the tremendous potential of EMO as a lightweight calibration method for enhancing large-scale pre-trained language models.