Word2Vec vs DBnary: Augmenting METEOR using Vector Representations or Lexical Resources?

TL;DR

This paper explores combining lexical resources and word vector representations to enhance the METEOR metric for machine translation evaluation, demonstrating that vector-based approaches can effectively supplement traditional lexical methods.

Contribution

It introduces a novel method of augmenting METEOR with word embeddings, showing that distributed representations can serve as a viable alternative or complement to lexical resources.

Findings

Distributed representations are effective for MT evaluation.

Vector-based METEOR versions outperform traditional lexical-only approaches.

Augmented METEOR metrics are publicly available for use.

Abstract

This paper presents an approach combining lexico-semantic resources and distributed representations of words applied to the evaluation in machine translation (MT). This study is made through the enrichment of a well-known MT evaluation metric: METEOR. This metric enables an approximate match (synonymy or morphological similarity) between an automatic and a reference translation. Our experiments are made in the framework of the Metrics task of WMT 2014. We show that distributed representations are a good alternative to lexico-semantic resources for MT evaluation and they can even bring interesting additional information. The augmented versions of METEOR, using vector representations, are made available on our Github page.