Take It Easy: Label-Adaptive Self-Rationalization for Fact Verification and Explanation Generation

TL;DR

This paper introduces a label-adaptive self-rationalization method for fact verification that improves accuracy and explanation quality, outperforming GPT-4 and leveraging synthetic explanations to reduce annotation costs.

Contribution

It extends self-rationalization to fact verification with label-adaptive learning and demonstrates effective use of synthetic explanations for low-cost training.

Findings

Improves veracity prediction by over 10 percentage points (Macro F1)

Outperforms GPT-4 in fact verification accuracy

Synthetic explanations enable effective low-cost model fine-tuning

Abstract

Computational methods to aid journalists in the task often require adapting a model to specific domains and generating explanations. However, most automated fact-checking methods rely on three-class datasets, which do not accurately reflect real-world misinformation. Moreover, fact-checking explanations are often generated based on text summarization of evidence, failing to address the relationship between the claim and the evidence. To address these issues, we extend the self-rationalization method--typically used in natural language inference (NLI) tasks--to fact verification. We propose a label-adaptive learning approach: first, we fine-tune a model to learn veracity prediction with annotated labels (step-1 model). Then, we fine-tune the step-1 model again to learn self-rationalization, using the same data and additional annotated explanations. Our results show that our label-adaptive approach improves veracity prediction by more than ten percentage points (Macro F1) on both the PubHealth and AVeriTec datasets, outperforming the GPT-4 model. Furthermore, to address the high cost of explanation annotation, we generated 64 synthetic explanations from three large language models: GPT-4-turbo, GPT-3.5-turbo, and Llama-3-8B and few-shot fine-tune our step-1 model. The few-shot synthetic explanation fine-tuned model performed comparably to the fully fine-tuned self-rationalization model, demonstrating the potential of low-budget learning with synthetic data. Our label-adaptive self-rationalization approach presents a promising direction for future research on real-world explainable fact-checking with different labeling schemes.