Neural Named Entity Recognition from Subword Units

TL;DR

This paper explores neural NER using subword units like characters, phonemes, and bytes, reducing vocabulary size and improving performance, especially with larger datasets, across multiple languages.

Contribution

It introduces a subword-based neural NER model that reduces vocabulary size and enhances performance, especially in large-scale, multilingual settings.

Findings

Subword units approach approaches word-level performance with less vocabulary.

Combining subword units improves NER accuracy.

Subword models outperform word-only models with limited data.

Abstract

Named entity recognition (NER) is a vital task in spoken language understanding, which aims to identify mentions of named entities in text e.g., from transcribed speech. Existing neural models for NER rely mostly on dedicated word-level representations, which suffer from two main shortcomings. First, the vocabulary size is large, yielding large memory requirements and training time. Second, these models are not able to learn morphological or phonological representations. To remedy the above shortcomings, we adopt a neural solution based on bidirectional LSTMs and conditional random fields, where we rely on subword units, namely characters, phonemes, and bytes. For each word in an utterance, our model learns a representation from each of the subword units. We conducted experiments in a real-world large-scale setting for the use case of a voice-controlled device covering four languages with up to 5.5M utterances per language. Our experiments show that (1) with increasing training data, performance of models trained solely on subword units becomes closer to that of models with dedicated word-level embeddings (91.35 vs 93.92 F1 for English), while using a much smaller vocabulary size (332 vs 74K), (2) subword units enhance models with dedicated word-level embeddings, and (3) combining different subword units improves performance.