On the Evaluation of Semantic Phenomena in Neural Machine Translation Using Natural Language Inference

TL;DR

This paper introduces a method to evaluate how well neural machine translation systems encode various semantic phenomena by using their sentence representations as features for natural language inference tasks.

Contribution

It presents a novel process for assessing semantic encoding in NMT systems through NLI classifiers trained on recast semantic datasets.

Findings

NMT encoder supports syntax-semantics inferences

Limited support for world-knowledge-based inferences

Framework for evaluating semantic coverage in NMT

Abstract

We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world-knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage.