Can Language Models Induce Grammatical Knowledge from Indirect Evidence?

TL;DR

This paper examines whether language models can efficiently learn grammatical rules from indirect evidence, revealing that current models struggle with this task and suggesting future research directions for improving their inductive capabilities.

Contribution

The paper introduces the Wug InDirect Evidence Test (WIDET) dataset to evaluate models' ability to learn grammar from indirect evidence, highlighting limitations in current models' data efficiency.

Findings

Language models do not induce grammatical knowledge from indirect evidence.

Models struggle even after repeated exposure to structured instances.

Future work should focus on models utilizing latent indirect evidence.

Abstract

What kinds of and how much data is necessary for language models to induce grammatical knowledge to judge sentence acceptability? Recent language models still have much room for improvement in their data efficiency compared to humans. This paper investigates whether language models efficiently use indirect data (indirect evidence), from which they infer sentence acceptability. In contrast, humans use indirect evidence efficiently, which is considered one of the inductive biases contributing to efficient language acquisition. To explore this question, we introduce the Wug InDirect Evidence Test (WIDET), a dataset consisting of training instances inserted into the pre-training data and evaluation instances. We inject synthetic instances with newly coined wug words into pretraining data and explore the model's behavior on evaluation data that assesses grammatical acceptability regarding those words. We prepare the injected instances by varying their levels of indirectness and quantity. Our experiments surprisingly show that language models do not induce grammatical knowledge even after repeated exposure to instances with the same structure but differing only in lexical items from evaluation instances in certain language phenomena. Our findings suggest a potential direction for future research: developing models that use latent indirect evidence to induce grammatical knowledge.