Monte Carlo Syntax Marginals for Exploring and Using Dependency Parses

TL;DR

This paper introduces a Monte Carlo sampling method for dependency parsers to quantify syntactic ambiguity, aiding error analysis and improving downstream NLP tasks by propagating parse uncertainty.

Contribution

It proposes a transition sampling algorithm for dependency parsing, enabling probabilistic analysis and uncertainty propagation in syntactic structures.

Findings

Effective in error analysis and calibration of parsers

Improves downstream tasks by propagating parse uncertainty

First analysis of dependency path prediction performance

Abstract

Dependency parsing research, which has made significant gains in recent years, typically focuses on improving the accuracy of single-tree predictions. However, ambiguity is inherent to natural language syntax, and communicating such ambiguity is important for error analysis and better-informed downstream applications. In this work, we propose a transition sampling algorithm to sample from the full joint distribution of parse trees defined by a transition-based parsing model, and demonstrate the use of the samples in probabilistic dependency analysis. First, we define the new task of dependency path prediction, inferring syntactic substructures over part of a sentence, and provide the first analysis of performance on this task. Second, we demonstrate the usefulness of our Monte Carlo syntax marginal method for parser error analysis and calibration. Finally, we use this method to propagate parse uncertainty to two downstream information extraction applications: identifying persons killed by police and semantic role assignment.