Why Attention? Analyzing and Remedying BiLSTM Deficiency in Modeling Cross-Context for NER

TL;DR

This paper identifies the limitations of BiLSTM in modeling cross-context patterns for NER and proposes simple attention-based remedies, resulting in significant performance improvements on benchmark datasets.

Contribution

It formally analyzes BiLSTM's deficiency in cross-context modeling and introduces simple attention-based structures to effectively address this issue.

Findings

Up to 8.7% improvement on multi-token mention identification

Effective remedy using self-attention and Cross-BiLSTM structures

Consistent improvements across OntoNotes 5.0 and WNUT 2017 datasets

Abstract

State-of-the-art approaches of NER have used sequence-labeling BiLSTM as a core module. This paper formally shows the limitation of BiLSTM in modeling cross-context patterns. Two types of simple cross-structures -- self-attention and Cross-BiLSTM -- are shown to effectively remedy the problem. On both OntoNotes 5.0 and WNUT 2017, clear and consistent improvements are achieved over bare-bone models, up to 8.7% on some of the multi-token mentions. In-depth analyses across several aspects of the improvements, especially the identification of multi-token mentions, are further given.