A Multilingual Evaluation of NER Robustness to Adversarial Inputs

TL;DR

This paper evaluates the robustness of multilingual NER models to adversarial inputs across English, German, and Hindi, revealing their vulnerabilities and exploring data augmentation and fine-tuning to enhance performance.

Contribution

It provides the first multilingual adversarial evaluation of NER robustness and compares data augmentation and fine-tuning methods for improving model resilience.

Findings

NER models are vulnerable to small input perturbations across languages.

Data augmentation and fine-tuning improve NER performance on original and adversarial data.

Re-training outperforms fine-tuning for German and Hindi NER models.

Abstract

Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in the input. Our results showed the NER models we explored across three languages (English, German and Hindi) are not very robust to such changes, as indicated by the fluctuations in the overall F1 score as well as in a more fine-grained evaluation. With that knowledge, we further explored whether it is possible to improve the existing NER models using a part of the generated adversarial data sets as augmented training data to train a new NER model or as fine-tuning data to adapt an existing NER model. Our results showed that both these approaches improve performance on the original as well as adversarial test sets. While there is no significant difference between the two approaches for English, re-training is significantly better than fine-tuning for German and Hindi.