Improving Statistical Language Model Performance with Automatically Generated Word Hierarchies

TL;DR

This paper introduces an automatic hierarchical word classification system that enhances language model performance by leveraging class-based structures derived from frequency statistics, revealing linguistic hierarchies from phonemic to semantic levels.

Contribution

The paper presents a novel binary top-down clustering method using mutual information, producing hierarchical classifications that improve language models.

Findings

Hierarchical classifications reveal linguistic structure.

Class-based models outperform traditional models.

The system effectively captures phonemic to semantic levels.

Abstract

An automatic word classification system has been designed which processes word unigram and bigram frequency statistics extracted from a corpus of natural language utterances. The system implements a binary top-down form of word clustering which employs an average class mutual information metric. Resulting classifications are hierarchical, allowing variable class granularity. Words are represented as structural tags --- unique $n$-bit numbers the most significant bit-patterns of which incorporate class information. Access to a structural tag immediately provides access to all classification levels for the corresponding word. The classification system has successfully revealed some of the structure of English, from the phonemic to the semantic level. The system has been compared --- directly and indirectly --- with other recent word classification systems. Class based interpolated language models have been constructed to exploit the extra information supplied by the classifications and some experiments have shown that the new models improve model performance.