BERT & Family Eat Word Salad: Experiments with Text Understanding

TL;DR

This paper investigates how large language models like BERT respond to incoherent and ill-formed inputs, revealing their vulnerability and proposing training methods to improve robustness against such inputs.

Contribution

The study introduces heuristics for creating incoherent inputs and demonstrates that training models to recognize invalid inputs enhances their robustness without performance loss.

Findings

Models fail to identify ill-formed inputs as invalid.

Training on invalid inputs improves robustness.

Permutation training yields comparable performance to standard models.

Abstract

In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.