Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing

TL;DR

This paper proposes Syntactic Smoothing, a method to reduce frequency bias and anisotropy in language models by incorporating syntactic priors, leading to improved handling of infrequent tokens and more balanced representations.

Contribution

Introduces a novel Syntactic Smoothing technique that adjusts the training objective to mitigate frequency bias and anisotropy in language models.

Findings

Reduced frequency bias improves performance on rare tokens.

Decreased anisotropy correlates with lower frequency bias.

Syntactic Smoothing enhances representational diversity.

Abstract

Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen during training. Moreover, maximum likelihood training has been discovered to give rise to anisotropy: representations of tokens in a model tend to cluster tightly in a high-dimensional cone, rather than spreading out over their representational capacity. Our work introduces a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. We then present a method for reducing the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. Our Syntactic Smoothing method adjusts the maximum likelihood objective function to distribute the learning signal to syntactically similar tokens. This approach results in better performance on infrequent English tokens and a decrease in anisotropy. We empirically show that the degree of anisotropy in a model correlates with its frequency bias.