Towards Few-Shot Fact-Checking via Perplexity

TL;DR

This paper introduces a novel few-shot fact-checking approach using perplexity scores from large language models, outperforming baselines with minimal training data and providing new datasets.

Contribution

It proposes a perplexity-based method for few-shot fact-checking, demonstrating strong performance with only two training samples and releasing new COVID-19 fact-checking datasets.

Findings

Outperforms baseline models with just two training samples

Verifies the effectiveness of perplexity scores in fact-checking

Provides publicly available COVID-19 fact-checking datasets

Abstract

Few-shot learning has drawn researchers' attention to overcome the problem of data scarcity. Recently, large pre-trained language models have shown great performance in few-shot learning for various downstream tasks, such as question answering and machine translation. Nevertheless, little exploration has been made to achieve few-shot learning for the fact-checking task. However, fact-checking is an important problem, especially when the amount of information online is growing exponentially every day. In this paper, we propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. The most notable strength of our methodology lies in its capability in few-shot learning. With only two training samples, our methodology can already outperform the Major Class baseline by more than absolute 10% on the F1-Macro metric across multiple datasets. Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models. Moreover, we construct and publicly release two new fact-checking datasets related to COVID-19.