TL;DR
This paper investigates whether large language models exhibit systematic errors in subject control sentences similar to children and examines their sensitivity to semantic role cues, revealing distinct behavioral groups and the influence of prompting.
Contribution
It categorizes language models based on their control behavior, analyzes their reliance on positional heuristics, and explores how prompting affects their outputs and semantic role understanding.
Findings
Models fall into three behavioral groups with broad differences.
Largest group models rely on positional heuristics, succeeding on subject but not on object control.
Prompting with agent-patient information significantly alters model outputs.
Abstract
Children acquiring English make systematic errors on subject control sentences even after they have reached near-adult competence (C. Chomsky, 1969), possibly due to heuristics based on semantic roles (Maratsos, 1974). Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other. We find that models can be categorized by behavior into three separate groups, with broad differences between the groups. The outputs of models in the largest group are consistent with positional heuristics that succeed on subject control but fail on object control. This result is surprising, given that object control is orders of magnitude more frequent in the text data used to train such models. We examine to what degree the models are sensitive to prompting with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
