Is the Computation of Abstract Sameness Relations Human-Like in Neural Language Models?

TL;DR

This paper investigates whether neural language models like BERT can compute abstract sameness relations akin to human infants, revealing that models are less capable than infants at this primitive cognitive task.

Contribution

It introduces a novel experimental framework inspired by infant studies to evaluate abstract sameness computation in pre-trained language models.

Findings

Models perform worse than infants in computing sameness relations.

The primitive cognitive mechanism is stronger in infants than in current NLP models.

Abstract

In recent years, deep neural language models have made strong progress in various NLP tasks. This work explores one facet of the question whether state-of-the-art NLP models exhibit elementary mechanisms known from human cognition. The exploration is focused on a relatively primitive mechanism for which there is a lot of evidence from various psycholinguistic experiments with infants. The computation of "abstract sameness relations" is assumed to play an important role in human language acquisition and processing, especially in learning more complex grammar rules. In order to investigate this mechanism in BERT and other pre-trained language models (PLMs), the experiment designs from studies with infants were taken as the starting point. On this basis, we designed experimental settings in which each element from the original studies was mapped to a component of language models. Even though the task in our experiments was relatively simple, the results suggest that the cognitive faculty of computing abstract sameness relations is stronger in infants than in all investigated PLMs.