Neural Network Acceptability Judgments

TL;DR

This paper evaluates neural networks' ability to judge sentence grammaticality using a new dataset, showing they outperform previous models but still fall short of human performance across various grammatical phenomena.

Contribution

Introduces the CoLA dataset for acceptability judgments and benchmarks neural network models, highlighting their strengths and limitations.

Findings

Models outperform previous unsupervised approaches on CoLA

Neural networks learn some grammatical generalizations

Models perform below human level on complex constructions

Abstract

This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.'s models and ours learn systematic generalizations like subject-verb-object order. However, all models we test perform far below human level on a wide range of grammatical constructions.