Syntactic Substitutability as Unsupervised Dependency Syntax

TL;DR

This paper introduces a novel unsupervised method for dependency syntax induction based on syntactic substitutability, leveraging language model attention to improve parsing accuracy without relying on annotated data.

Contribution

The authors propose a theory-agnostic approach that models syntactic dependencies through substitutability, enabling unsupervised dependency parsing using language model attention distributions.

Findings

Improves parsing accuracy with more substitutions used.

Achieves 79.5% recall on long-distance subject-verb agreement.

Generalizes well across different parsing setups.

Abstract

Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. In this work, we explore the hypothesis that syntactic dependencies can be represented in language model attention distributions and propose a new method to induce these structures theory-agnostically. Instead of modeling syntactic relations as defined by annotation schemata, we model a more general property implicit in the definition of dependency relations, syntactic substitutability. This property captures the fact that words at either end of a dependency can be substituted with words from the same category. Substitutions can be used to generate a set of syntactically invariant sentences whose representations are then used for parsing. We show that increasing the number of substitutions used improves parsing accuracy on natural data. On long-distance subject-verb agreement constructions, our method achieves 79.5% recall compared to 8.9% using a previous method. Our method also provides improvements when transferred to a different parsing setup, demonstrating that it generalizes.