Dependency-based Mixture Language Models

TL;DR

This paper introduces Dependency-based Mixture Language Models that integrate syntactic dependency knowledge into neural language models, enhancing text generation across different architectures like Transformers and GPT-2.

Contribution

It proposes a novel dependency modeling objective and a mixture approach to incorporate syntactic structures into various neural language models.

Findings

Improved neural text generation performance.

Effective application across different neural architectures.

Positive human evaluation results.

Abstract

Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.