Semantic Smoothing for Language Models via Distribution Estimation and Embeddings

TL;DR

This paper introduces semantic smoothing, a novel method for language models that leverages embeddings to improve statistical estimation and reduce perplexity.

Contribution

It formulates semantic smoothing as a distribution estimation problem with KL proximity side information and provides theoretical risk bounds and empirical validation.

Findings

Semantic smoothing reduces test perplexity on synthetic and real data.

The method achieves optimal worst-case KL risk bounds.

Experiments show consistent improvements over traditional smoothing techniques.

Abstract

We propose semantic smoothing, a smoothing method for language models that uses embeddings to share statistical observations across semantically similar contexts. The starting point is a decomposition of log-perplexity that motivates smoothing as a collection of distribution-estimation problems under Kullback-Leibler (KL) loss. We then show that, under a Lipschitz-logit model for embedding-based language generation, proximity of context embeddings implies proximity of the corresponding next-word distributions in KL divergence. Combining these observations, we formulate semantic smoothing as distribution estimation in KL loss with KL-proximity side information. For $n$ samples on a $d$-symbol alphabet with a side-information distribution at KL distance $\Delta$, we give an interpolation estimator with worst-case KL risk $O(\min\{\Delta,d/n\})$, and prove a matching-order lower bound for uniform side information. We extend the estimator to multiple and empirically estimated synonymous distributions. Experiments on synthetic Markov data and WikiText-103 bigram models using Word2Vec, GloVe, and GPT-2 embeddings show that semantic smoothing consistently reduces test perplexity when applied to add-constant and Kneser-Ney estimates.