Are BLEU and Meaning Representation in Opposition?

TL;DR

This paper investigates the relationship between translation quality and the usefulness of neural machine translation-derived sentence representations, finding that higher translation quality correlates with poorer performance in classification and similarity tasks.

Contribution

The paper introduces variations of attentive NMT architectures that restore the source sentence representation point, and empirically examines the trade-off between translation quality and representation utility.

Findings

Better translation quality correlates with worse sentence representations in downstream tasks.

Restoring the source representation point affects the quality and utility of learned embeddings.

Neural translation systems may not produce optimal representations for semantic tasks.

Abstract

One of possible ways of obtaining continuous-space sentence representations is by training neural machine translation (NMT) systems. The recent attention mechanism however removes the single point in the neural network from which the source sentence representation can be extracted. We propose several variations of the attentive NMT architecture bringing this meeting point back. Empirical evaluation suggests that the better the translation quality, the worse the learned sentence representations serve in a wide range of classification and similarity tasks.