What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?
Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhijeet Awasthi,, Sunita Sarawagi

TL;DR
This paper investigates the robustness of BERT-based named entity representations to name replacements within the same entity class, revealing surprising brittleness and proposing a method to improve stability and accuracy.
Contribution
It identifies the fragility of BERT's entity representations to perturbations and introduces an ensemble-based approach to enhance robustness and performance.
Findings
State-of-the-art models are brittle to entity replacements.
Ensemble method improves robustness and accuracy.
Method effective on multiple NLP tasks.
Abstract
We evaluate named entity representations of BERT-based NLP models by investigating their robustness to replacements from the same typed class in the input. We highlight that on several tasks while such perturbations are natural, state of the art trained models are surprisingly brittle. The brittleness continues even with the recent entity-aware BERT models. We also try to discern the cause of this non-robustness, considering factors such as tokenization and frequency of occurrence. Then we provide a simple method that ensembles predictions from multiple replacements while jointly modeling the uncertainty of type annotations and label predictions. Experiments on three NLP tasks show that our method enhances robustness and increases accuracy on both natural and adversarial datasets.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Data Quality and Management
MethodsLinear Layer · Attention Dropout · Adam · Dense Connections · Residual Connection · Dropout · Linear Warmup With Linear Decay · Refunds@Expedia|||How do I get a full refund from Expedia? · Multi-Head Attention · Layer Normalization
