Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgments

TL;DR

This paper introduces 'grammar prompting', a method where large language models first explain grammatical rules and then use these explanations to improve sentence acceptability judgments, significantly enhancing performance across multiple languages and models.

Contribution

The paper proposes a novel explain-then-process paradigm that leverages explanations to improve grammatical acceptability judgments in language models, bridging the gap between rule knowledge and application.

Findings

Grammar prompting improves accuracy on syntactic benchmarks.

Feeding explanations back to models reduces the accuracy gap between small and large models.

The method is effective across multiple languages and model sizes.

Abstract

Large language models (LLMs) can explain grammatical rules, yet they often fail to apply those rules when judging sentence acceptability. We present "grammar prompting", an explain-then-process paradigm: a large LLM first produces a concise explanation of the relevant syntactic phenomenon, then that explanation is fed back as additional context to the target model -- either an LLM or a smaller language model (SLM) -- before deciding which sentence of a minimal pair is grammatical. On the English BLiMP, Chinese SLING, and Russian RuBLiMP benchmarks, this simple prompt design yields substantial improvements over strong baselines across many syntactic phenomena. Feeding an LLM's metalinguistic explanation back to the target model bridges the gap between knowing a rule and using it. On SLMs, grammar prompting alone trims the average LLM-SLM accuracy gap by about 20%, and when paired with chain-of-thought, by 56% (13.0 pp -> 5.8 pp), all at negligible cost. The lightweight, language-agnostic cue lets low-cost SLMs approach frontier-LLM performance in multilingual settings.