Abstraction not Memory: BERT and the English Article System
Harish Tayyar Madabushi, Dagmar Divjak, Petar Milin

TL;DR
This paper compares BERT and native speakers on English article prediction, revealing BERT's superior performance and suggesting it captures human-like generalizations rather than mere memorization.
Contribution
It demonstrates that BERT outperforms humans in article prediction and provides evidence that BERT's understanding aligns more with human intuition than simple memorization.
Findings
BERT outperforms humans in article prediction tasks.
BERT is especially better at detecting zero articles.
BERT's predictions align more with annotators than with the corpus when agreement is high.
Abstract
Article prediction is a task that has long defied accurate linguistic description. As such, this task is ideally suited to evaluate models on their ability to emulate native-speaker intuition. To this end, we compare the performance of native English speakers and pre-trained models on the task of article prediction set up as a three way choice (a/an, the, zero). Our experiments with BERT show that BERT outperforms humans on this task across all articles. In particular, BERT is far superior to humans at detecting the zero article, possibly because we insert them using rules that the deep neural model can easily pick up. More interestingly, we find that BERT tends to agree more with annotators than with the corpus when inter-annotator agreement is high but switches to agreeing more with the corpus as inter-annotator agreement drops. We contend that this alignment with annotators, despite…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
MethodsMulti-Head Attention · Attention Is All You Need · Linear Layer · Linear Warmup With Linear Decay · Weight Decay · Layer Normalization · Refunds@Expedia|||How do I get a full refund from Expedia? · Softmax · Attention Dropout · WordPiece
