Latent Tree Language Model

TL;DR

This paper introduces the Latent Tree Language Model (LTLM), which encodes syntax and semantics as trees, improving language modeling by reducing perplexity significantly compared to traditional models.

Contribution

The paper presents a novel tree-based language model with two inference algorithms and demonstrates its effectiveness through bilingual experiments.

Findings

Up to 46% perplexity reduction for English

Up to 49% perplexity reduction for Czech

Effective combination with 4-gram models

Abstract

In this paper we introduce Latent Tree Language Model (LTLM), a novel approach to language modeling that encodes syntax and semantics of a given sentence as a tree of word roles. The learning phase iteratively updates the trees by moving nodes according to Gibbs sampling. We introduce two algorithms to infer a tree for a given sentence. The first one is based on Gibbs sampling. It is fast, but does not guarantee to find the most probable tree. The second one is based on dynamic programming. It is slower, but guarantees to find the most probable tree. We provide comparison of both algorithms. We combine LTLM with 4-gram Modified Kneser-Ney language model via linear interpolation. Our experiments with English and Czech corpora show significant perplexity reductions (up to 46% for English and 49% for Czech) compared with standalone 4-gram Modified Kneser-Ney language model.