A Natural Bias for Language Generation Models

TL;DR

This paper proposes initializing language models with unigram frequency biases to improve learning efficiency and performance, demonstrated through neural machine translation experiments.

Contribution

It introduces a simple method of bias initialization using unigram distributions to enhance language model training.

Findings

Improved learning efficiency in neural machine translation.

Achieved better overall performance with bias initialization.

Encouraged models to focus on non-frequency language aspects.

Abstract

After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus. The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity? Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model's final linear layer with the log-unigram distribution. We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.