Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing

TL;DR

This paper proposes using policy gradient methods as a parser-agnostic alternative to dynamic oracles for training constituency parsers, improving performance and reducing the need for custom supervision.

Contribution

It introduces a policy gradient approach that directly optimizes tree-level metrics and can replace dynamic oracles across different parser architectures.

Findings

Policy gradient outperforms static likelihood training in most settings.

It recaptures much of the performance gain of dynamic oracles when available.

The method is effective across multiple languages and parser types.

Abstract

Dynamic oracles provide strong supervision for training constituency parsers with exploration, but must be custom defined for a given parser's transition system. We explore using a policy gradient method as a parser-agnostic alternative. In addition to directly optimizing for a tree-level metric such as F1, policy gradient has the potential to reduce exposure bias by allowing exploration during training; moreover, it does not require a dynamic oracle for supervision. On four constituency parsers in three languages, the method substantially outperforms static oracle likelihood training in almost all settings. For parsers where a dynamic oracle is available (including a novel oracle which we define for the transition system of Dyer et al. 2016), policy gradient typically recaptures a substantial fraction of the performance gain afforded by the dynamic oracle.