Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model

TL;DR

This paper introduces a fast, accurate hierarchical RNN model for capitalization normalization, improving language modeling and real-world applications like virtual keyboards and speech recognition.

Contribution

The paper presents a novel two-level hierarchical RNN for truecasing that is both efficient and effective, enabling better language models and practical applications.

Findings

Achieves same perplexity as gold-standard models on normalized text

Reduces prediction error rates in virtual keyboard applications

Lowers character and word error rates in speech recognition

Abstract

Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text. We propose a fast, accurate and compact two-level hierarchical word-and-character-based recurrent neural network model. We use the truecaser to normalize user-generated text in a Federated Learning framework for language modeling. A case-aware language model trained on this normalized text achieves the same perplexity as a model trained on text with gold capitalization. In a real user A/B experiment, we demonstrate that the improvement translates to reduced prediction error rates in a virtual keyboard application. Similarly, in an ASR language model fusion experiment, we show reduction in uppercase character error rate and word error rate.