An Empirical Study of Smoothing Techniques for Language Modeling

TL;DR

This paper empirically compares various smoothing techniques for language modeling, examining factors like data size and corpus type, and introduces two novel methods that outperform existing ones.

Contribution

It provides the first comprehensive empirical analysis of smoothing techniques considering multiple factors and proposes two new methods that improve performance.

Findings

New smoothing techniques outperform existing methods

Performance varies with data size, corpus, and n-gram order

Introduces a variation of Jelinek-Mercer smoothing and a simple linear interpolation

Abstract

We present an extensive empirical comparison of several smoothing techniques in the domain of language modeling, including those described by Jelinek and Mercer (1980), Katz (1987), and Church and Gale (1991). We investigate for the first time how factors such as training data size, corpus (e.g., Brown versus Wall Street Journal), and n-gram order (bigram versus trigram) affect the relative performance of these methods, which we measure through the cross-entropy of test data. In addition, we introduce two novel smoothing techniques, one a variation of Jelinek-Mercer smoothing and one a very simple linear interpolation technique, both of which outperform existing methods.