Abstraction via exemplars? A representational case study on lexical category inference in BERT

TL;DR

This study investigates whether neural language models like BERT can develop abstract lexical categories through exemplar encoding, showing that novel tokens move towards known category regions in representation space.

Contribution

It provides empirical evidence that BERT's generalization to novel tokens involves exemplar-based encoding leading to abstraction-like behavior.

Findings

BERT generalizes to unseen tokens within lexical categories.

Representations of novel tokens move towards known category exemplars.

Exemplar encoding can produce abstraction-like generalization in neural models.

Abstract

Exemplar based accounts are often considered to be in direct opposition to pure linguistic abstraction in explaining language learners' ability to generalize to novel expressions. However, the recent success of neural network language models on linguistically sensitive tasks suggests that perhaps abstractions can arise via the encoding of exemplars. We provide empirical evidence for this claim by adapting an existing experiment that studies how an LM (BERT) generalizes the usage of novel tokens that belong to lexical categories such as Noun/Verb/Adjective/Adverb from exposure to only a single instance of their usage. We analyze the representational behavior of the novel tokens in these experiments, and find that BERT's capacity to generalize to unseen expressions involving the use of these novel tokens constitutes the movement of novel token representations towards regions of known category exemplars in two-dimensional space. Our results suggest that learners' encoding of exemplars can indeed give rise to abstraction like behavior.