Evidence-based Factual Error Correction

TL;DR

This paper proposes a new task called factual error correction, which involves editing claims to align better with evidence, extending fact verification with a novel training approach using existing datasets and transformer models.

Contribution

It introduces a distant supervision method for training factual error correction systems from fact-checking datasets and demonstrates improved performance over previous methods.

Findings

Achieved better correction accuracy than previous models.

Produced corrections for 5 times more instances in human evaluation.

Increased SARI score by 0.125.

Abstract

This paper introduces the task of factual error correction: performing edits to a claim so that the generated rewrite is better supported by evidence. This extends the well-studied task of fact verification by providing a mechanism to correct written texts that are refuted or only partially supported by evidence. We demonstrate that it is feasible to train factual error correction systems from existing fact checking datasets which only contain labeled claims accompanied by evidence, but not the correction. We achieve this by employing a two-stage distant supervision approach that incorporates evidence into masked claims when generating corrections. Our approach, based on the T5 transformer and using retrieved evidence, achieved better results than existing work which used a pointer copy network and gold evidence, producing accurate factual error corrections for 5x more instances in human evaluation and a .125 increase in SARI score. The evaluation is conducted on a dataset of 65,000 instances based on a recent fact verification shared task and we release it to enable further work on the task.