Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization

TL;DR

This paper introduces a hybrid text normalization method combining rule-based WFSTs with neural language models, effectively handling ambiguity and reducing errors, and achieving state-of-the-art results.

Contribution

A novel shallow fusion approach that integrates WFSTs with neural language models for improved text normalization.

Findings

Achieves comparable or better results than existing models.

Effectively resolves contextual ambiguity in text normalization.

Reduces unrecoverable errors in rule-based systems.

Abstract

Text normalization (TN) systems in production are largely rule-based using weighted finite-state transducers (WFST). However, WFST-based systems struggle with ambiguous input when the normalized form is context-dependent. On the other hand, neural text normalization systems can take context into account but they suffer from unrecoverable errors and require labeled normalization datasets, which are hard to collect. We propose a new hybrid approach that combines the benefits of rule-based and neural systems. First, a non-deterministic WFST outputs all normalization candidates, and then a neural language model picks the best one -- similar to shallow fusion for automatic speech recognition. While the WFST prevents unrecoverable errors, the language model resolves contextual ambiguity. The approach is easy to extend and we show it is effective. It achieves comparable or better results than existing state-of-the-art TN models.