Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks

TL;DR

Blackbird's language matrices (BLMs) introduce a large, generative linguistic dataset designed to evaluate neural networks' ability to learn and generalize grammatical rules, serving as a new benchmark for studying abstract linguistic generalization.

Contribution

The paper presents BLMs, a novel large-scale grammatical dataset inspired by Raven's matrices, enabling systematic investigation of linguistic generalization in neural networks.

Findings

Neural models struggle with grammatical agreement generalization.

The dataset reveals limitations in current models' linguistic abstraction.

Error analysis highlights specific challenges in rule learning.

Abstract

Current successes of machine learning architectures are based on computationally expensive algorithms and prohibitively large amounts of data. We need to develop tasks and data to train networks to reach more complex and more compositional skills. In this paper, we illustrate Blackbird's language matrices (BLMs), a novel grammatical dataset developed to test a linguistic variant of Raven's progressive matrices, an intelligence test usually based on visual stimuli. The dataset consists of 44800 sentences, generatively constructed to support investigations of current models' linguistic mastery of grammatical agreement rules and their ability to generalise them. We present the logic of the dataset, the method to automatically construct data on a large scale and the architecture to learn them. Through error analysis and several experiments on variations of the dataset, we demonstrate that this language task and the data that instantiate it provide a new challenging testbed to understand generalisation and abstraction.