Probing Syntax in Large Language Models: Successes and Remaining Challenges

TL;DR

This paper critically examines the effectiveness of structural probes in extracting syntactic information from large language models, revealing biases, limitations, and challenges in representing complex syntactic structures.

Contribution

It introduces a controlled benchmark to systematically evaluate structural probes, highlighting their biases and limitations in capturing deep syntactic structures.

Findings

Probes are biased by word proximity in sentences.

Probes struggle with deep syntactic structures and interacting nouns.

Probes are unaffected by word predictability.

Abstract

The syntactic structures of sentences can be readily read-out from the activations of large language models (LLMs). However, the ``structural probes'' that have been developed to reveal this phenomenon are typically evaluated on an indiscriminate set of sentences. Consequently, it remains unclear whether structural and/or statistical factors systematically affect these syntactic representations. To address this issue, we conduct an in-depth analysis of structural probes on three controlled benchmarks. Our results are three-fold. First, structural probes are biased by a superficial property: the closer two words are in a sentence, the more likely structural probes will consider them as syntactically linked. Second, structural probes are challenged by linguistic properties: they poorly represent deep syntactic structures, and get interfered by interacting nouns or ungrammatical verb forms. Third, structural probes do not appear to be affected by the predictability of individual words. Overall, this work sheds light on the current challenges faced by structural probes. Providing a benchmark made of controlled stimuli to better evaluate their performance.