Do Context-Aware Translation Models Pay the Right Attention?

TL;DR

This paper investigates whether context-aware translation models focus on the right contextual information for disambiguation, introduces a new dataset, and explores methods to align model attention with human-identified supporting context.

Contribution

The paper introduces SCAT, a new dataset of supporting context for translation, and analyzes model attention alignment with human disambiguation cues, proposing guided attention techniques.

Findings

Models often do not focus on the most relevant context for disambiguation.

Explicitly training models to attend to supporting context improves alignment.

SCAT dataset reveals key positional and lexical features of disambiguating words.

Abstract

Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolution. In this paper, we ask several questions: What contexts do human translators use to resolve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a new English-French dataset comprising supporting context words for 14K translations that professional translators found useful for pronoun disambiguation. Using SCAT, we perform an in-depth analysis of the context used to disambiguate, examining positional and lexical characteristics of the supporting words. Furthermore, we measure the degree of alignment between the model's attention scores and the supporting context from SCAT, and apply a guided attention strategy to encourage agreement between the two.