Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

TL;DR

Data-QuestEval introduces a novel reference-less, multimodal metric for data-to-text semantic evaluation, leveraging synthetic corpora to train components and achieving state-of-the-art correlation with human judgments.

Contribution

It presents a new method to adapt QuestEval for data-to-text tasks by creating synthetic multimodal training data, enabling effective evaluation without references.

Findings

Achieves state-of-the-art correlation with human judgments on WebNLG and WikiBio.

Develops synthetic multimodal corpora for training evaluation components.

Provides open-source code and models for reproducibility.

Abstract

QuestEval is a reference-less metric used in text-to-text tasks, that compares the generated summaries directly to the source text, by automatically asking and answering questions. Its adaptation to Data-to-Text tasks is not straightforward, as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting metric is reference-less and multimodal; it obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. We make data-QuestEval's code and models available for reproducibility purpose, as part of the QuestEval project.