Entity-Aware Language Model as an Unsupervised Reranker

TL;DR

This paper introduces an unsupervised reranker that incorporates entity relationships and external knowledge-base features into language modeling, improving accuracy without needing manually annotated data.

Contribution

It proposes a contrastive estimation-based method for entity-aware reranking that eliminates the need for annotated n-best lists, enhancing language model performance.

Findings

Achieved 0.44% absolute WER reduction over baseline LSTM model.

Successfully integrated global features and knowledge-base information.

Demonstrated effectiveness in the music domain.

Abstract

In language modeling, it is difficult to incorporate entity relationships from a knowledge-base. One solution is to use a reranker trained with global features, in which global features are derived from n-best lists. However, training such a reranker requires manually annotated n-best lists, which is expensive to obtain. We propose a method based on the contrastive estimation method that alleviates the need for such data. Experiments in the music domain demonstrate that global features, as well as features extracted from an external knowledge-base, can be incorporated into our reranker. Our final model, a simple ensemble of a language model and reranker, achieves a 0.44\% absolute word error rate improvement over an LSTM language model on the blind test data.