Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection

TL;DR

This paper presents a new annotation schema and benchmark for automated claim detection, leveraging professional factcheckers' expertise, and introduces a universal sentence representation approach that outperforms existing methods.

Contribution

It develops a more consistent annotation schema and benchmark for claim detection, and proposes a novel classification approach with improved accuracy.

Findings

Achieved an F1 score of 0.83 with the new approach.

Outperformed ClaimBuster and ClaimRank by over 5%.

System received positive user feedback in production.

Abstract

In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.