$N$-gram Is Back: Residual Learning of Neural Text Generation with $n$-gram Language Model

TL;DR

This paper introduces a residual learning approach combining traditional $n$-gram models with neural language models, enhancing performance and domain adaptability across various language tasks by leveraging the strengths of both methods.

Contribution

The paper proposes a novel residual learning framework that integrates $n$-gram models with neural LMs, enabling improved performance and flexible domain adaptation without retraining the neural model.

Findings

Achieves consistent performance gains over standalone neural models.

Enables effective domain adaptation by switching $n$-gram models.

Demonstrates improvements on language modeling, translation, and summarization tasks.

Abstract

$N$-gram language models (LM) have been largely superseded by neural LMs as the latter exhibits better performance. However, we find that $n$-gram models can achieve satisfactory performance on a large proportion of testing cases, indicating they have already captured abundant knowledge of the language with relatively low computational cost. With this observation, we propose to learn a neural LM that fits the residual between an $n$-gram LM and the real-data distribution. The combination of $n$-gram and neural LMs not only allows the neural part to focus on the deeper understanding of language but also provides a flexible way to customize an LM by switching the underlying $n$-gram model without changing the neural model. Experimental results on three typical language tasks (i.e., language modeling, machine translation, and summarization) demonstrate that our approach attains additional performance gains over popular standalone neural models consistently. We also show that our approach allows for effective domain adaptation by simply switching to a domain-specific $n$-gram model, without any extra training. Our code is released at https://github.com/ghrua/NgramRes.