Evaluating the Morphosyntactic Well-formedness of Generated Texts

TL;DR

This paper introduces L'AMBRE, a new metric for assessing the morphosyntactic correctness of generated texts using dependency parsing and language rules, improving evaluation in multilingual NLP tasks.

Contribution

It presents a novel metric based on dependency parse analysis, a method to automatically extract morphosyntactic rules, and a robust parser training approach for noisy generated texts.

Findings

L'AMBRE effectively evaluates morphosyntactic well-formedness.

The metric performs well on machine translation tasks.

Robust parsers improve analysis of noisy outputs.

Abstract

Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L'AMBRE -- a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.