Evaluating and Improving the Coreference Capabilities of Machine Translation Models

TL;DR

This paper assesses how well machine translation models implicitly learn coreference resolution, develops an evaluation method for this, and explores ways to improve MT by integrating coreference information.

Contribution

It introduces a novel evaluation methodology for coreference in MT outputs and demonstrates how incorporating coreference resolution can enhance translation quality.

Findings

MT models underperform compared to dedicated coreference resolvers

Incorporating coreference information improves translation quality

Monolingual coreference resolvers outperform MT models in coreference tasks

Abstract

Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: \emph{How well do MT models learn coreference resolution from implicit signal?} To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.