Dynamic Nonlocal Language Modeling via Hierarchical Topic-Based Adaptation

TL;DR

This paper introduces a hierarchical, dynamic topic-based language modeling approach that improves long-distance dependency capture and significantly reduces perplexity in speech recognition tasks.

Contribution

It proposes new cluster generation, hierarchical smoothing, and adaptive topic estimation techniques for dynamic language modeling.

Findings

10.5% perplexity reduction overall

33.5% perplexity reduction on target vocabulary

Effective modeling of long-distance lexical dependencies

Abstract

This paper presents a novel method of generating and applying hierarchical, dynamic topic-based language models. It proposes and evaluates new cluster generation, hierarchical smoothing and adaptive topic-probability estimation techniques. These combined models help capture long-distance lexical dependencies. Experiments on the Broadcast News corpus show significant improvement in perplexity (10.5% overall and 33.5% on target vocabulary).