Mining the Explainability and Generalization: Fact Verification Based on Self-Instruction

TL;DR

This paper introduces a self-instruction based fine-tuning method for fact verification using LLMs, balancing accuracy, explainability, and data security, and demonstrating strong results on challenging datasets.

Contribution

It pioneers the use of self-supervised learning with contrastive learning and improved DPO for fact-checking, enhancing accuracy and generalization of LLMs.

Findings

Achieves comparable or better accuracy than traditional fine-tuning methods.

Generates fluent and high-quality explanations.

Shows high generalization performance on fact-checking datasets.

Abstract

Fact-checking based on commercial LLMs has become mainstream. Although these methods offer high explainability, it falls short in accuracy compared to traditional fine-tuning approaches, and data security is also a significant concern. In this paper, we propose a self-instruction based fine-tuning approach for fact-checking that balances accuracy and explainability. Our method consists of Data Augmentation and Improved DPO fine-tuning. The former starts by instructing the model to generate both positive and negative explanations based on claim-evidence pairs and labels, then sampling the dataset according to our customized difficulty standards. The latter employs our proposed improved DPO to fine-tune the model using the generated samples. We fine-tune the smallest-scale LLaMA-7B model and evaluate it on the challenging fact-checking datasets FEVEROUS and HOVER, utilizing four fine-tuning methods and three few-shot learning methods for comparison. The experiments demonstrate that our approach not only retains accuracy comparable to, or even surpassing, traditional fine-tuning methods, but also generates fluent explanation text. Moreover, it also exhibit high generalization performance. Our method is the first to leverage self-supervised learning for fact-checking and innovatively combines contrastive learning and improved DPO in fine-tuning LLMs, as shown in the experiments.