Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement
Bingzhi Li, Guillaume Wisniewski, and Benoit Crabb\'e

TL;DR
This paper critically examines whether neural models genuinely understand syntax by showing that simple heuristics can predict agreement, but finds Transformers do capture some grammatical structure unlike LSTMs.
Contribution
It reveals that high agreement accuracy can result from surface heuristics and demonstrates that Transformers encode more grammatical structure than LSTMs in French object-verb agreement.
Findings
Simple heuristics can predict agreement with high accuracy.
Transformers capture non-trivial grammatical structure.
LSTMs perform worse on long-range agreement tasks.
Abstract
Many recent works have demonstrated that unsupervised sentence representations of neural networks encode syntactic information by observing that neural language models are able to predict the agreement between a verb and its subject. We take a critical look at this line of research by showing that it is possible to achieve high accuracy on this agreement task with simple surface heuristics, indicating a possible flaw in our assessment of neural networks' syntactic ability. Our fine-grained analyses of results on the long-range French object-verb agreement show that contrary to LSTMs, Transformers are able to capture a non-trivial amount of grammatical structure.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
