Variation and generality in encoding of syntactic anomaly information in sentence embeddings

TL;DR

This paper investigates how different NLP models encode syntactic anomaly information in sentence embeddings, revealing that transformer models develop more generalized anomaly detection capabilities.

Contribution

It introduces probing tasks to analyze anomaly encoding at various hierarchical levels and examines transferability across anomaly types, highlighting differences among models.

Findings

All models encode some anomaly information.

Detection performance varies by anomaly type.

Transformer models show signs of generalized anomaly knowledge.

Abstract

While sentence anomalies have been applied periodically for testing in NLP, we have yet to establish a picture of the precise status of anomaly information in representations from NLP models. In this paper we aim to fill two primary gaps, focusing on the domain of syntactic anomalies. First, we explore fine-grained differences in anomaly encoding by designing probing tasks that vary the hierarchical level at which anomalies occur in a sentence. Second, we test not only models' ability to detect a given anomaly, but also the generality of the detected anomaly signal, by examining transfer between distinct anomaly types. Results suggest that all models encode some information supporting anomaly detection, but detection performance varies between anomalies, and only representations from more recent transformer models show signs of generalized knowledge of anomalies. Follow-up analyses support the notion that these models pick up on a legitimate, general notion of sentence oddity, while coarser-grained word position information is likely also a contributor to the observed anomaly detection.