NUBIA: NeUral Based Interchangeability Assessor for Text Generation

TL;DR

NUBIA is a neural-based framework for automatic text generation evaluation that outperforms existing metrics in correlation with human judgment across multiple tasks, offering a modular and explainable approach.

Contribution

The paper introduces NUBIA, a novel neural-based methodology for automatic evaluation of text generation, emphasizing modularity, explainability, and continuous improvement.

Findings

Outperforms current metrics in machine translation and summarization evaluation

Matches or exceeds state-of-the-art correlation with human judgments

Demonstrates modularity and explainability in evaluation metrics

Abstract

We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA which outperforms metrics currently used to evaluate machine translation, summaries and slightly exceeds/matches state of the art metrics on correlation with human judgement on the WMT segment-level Direct Assessment task, sentence-level ranking and image captioning evaluation. The model implemented is modular, explainable and set to continuously improve over time.