WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning

TL;DR

WeCheck is a weakly supervised factual consistency metric that leverages multiple resources and generative labeling to improve accuracy in evaluating generated text's factual correctness.

Contribution

The paper introduces WeCheck, a novel weakly supervised framework that effectively aggregates resources and handles noise to assess factual consistency in text generation.

Findings

Achieves 3.4% improvement on TRUE benchmark

Outperforms previous state-of-the-art methods

Demonstrates strong performance across various tasks

Abstract

A crucial issue of current text generation models is that they often uncontrollably generate factually inconsistent text with respective of their inputs. Limited by the lack of annotated data, existing works in evaluating factual consistency directly transfer the reasoning ability of models trained on other data-rich upstream tasks like question answering (QA) and natural language inference (NLI) without any further adaptation. As a result, they perform poorly on the real generated text and are biased heavily by their single-source upstream tasks. To alleviate this problem, we propose a weakly supervised framework that aggregates multiple resources to train a precise and efficient factual metric, namely WeCheck. WeCheck first utilizes a generative model to accurately label a real generated sample by aggregating its weak labels, which are inferred from multiple resources. Then, we train the target metric model with the weak supervision while taking noises into consideration. Comprehensive experiments on a variety of tasks demonstrate the strong performance of WeCheck, which achieves a 3.4\% absolute improvement over previous state-of-the-art methods on TRUE benchmark on average.