Do We Need Neural Models to Explain Human Judgments of Acceptability?

TL;DR

This study evaluates whether computational models, including neural networks and simple language features, can predict human judgments of sentence acceptability, finding that basic n-gram models perform comparably to neural models.

Contribution

The paper demonstrates that simple n-gram models with features like misspellings and word order can predict acceptability judgments as well as neural models, challenging assumptions about the necessity of neural networks.

Findings

4-gram model with misspelling count surpasses previous state-of-the-art

Neural models and simple n-grams perform similarly in predicting acceptability

Acceptability judgments are well captured by n-gram statistics and simple features

Abstract

Native speakers can judge whether a sentence is an acceptable instance of their language. Acceptability provides a means of evaluating whether computational language models are processing language in a human-like manner. We test the ability of computational language models, simple language features, and word embeddings to predict native English speakers judgments of acceptability on English-language essays written by non-native speakers. We find that much of the sentence acceptability variance can be captured by a combination of features including misspellings, word order, and word similarity (Pearson's r = 0.494). While predictive neural models fit acceptability judgments well (r = 0.527), we find that a 4-gram model with statistical smoothing is just as good (r = 0.528). Thanks to incorporating a count of misspellings, our 4-gram model surpasses both the previous unsupervised state-of-the art (Lau et al., 2015; r = 0.472), and the average non-expert native speaker (r = 0.46). Our results demonstrate that acceptability is well captured by n-gram statistics and simple language features.