Deep Reinforcement Learning for Mention-Ranking Coreference Models

TL;DR

This paper introduces reinforcement learning techniques to train neural mention-ranking models for coreference resolution, directly optimizing evaluation metrics and achieving state-of-the-art results on CoNLL 2012 datasets.

Contribution

It applies reinforcement learning, specifically reward-rescaled max-margin, to coreference models, improving over heuristic loss functions and setting new performance benchmarks.

Findings

Reward-rescaled max-margin outperforms REINFORCE in experiments.

Significant improvements over state-of-the-art on CoNLL 2012 datasets.

Effective direct optimization of coreference evaluation metrics.

Abstract

Coreference resolution systems are typically trained with heuristic loss functions that require careful tuning. In this paper we instead apply reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics. We experiment with two approaches: the REINFORCE policy gradient algorithm and a reward-rescaled max-margin objective. We find the latter to be more effective, resulting in significant improvements over the current state-of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task.