Rating Facts under Coarse-to-fine Regimes

TL;DR

This paper introduces a BERT-based approach for fact rating under coarse-to-fine regimes, addressing the challenge of ordered class similarity in fake news detection with a new dataset and achieving state-of-the-art results.

Contribution

The paper presents a new dataset of 24K rated statements and a novel classification framework for ordered classes, applying BERT to improve fact rating accuracy.

Findings

BERT achieves state-of-the-art performance across regimes.

Class similarity aligns well with BERT's representations.

Additional layers offer minimal performance gains.

Abstract

The rise of manipulating fake news as a political weapon has become a global concern and highlighted the incapability of manually fact checking against rapidly produced fake news. Thus, statistical approaches are required if we are to address this problem efficiently. The shortage of publicly available datasets is one major bottleneck of automated fact checking. To remedy this, we collected 24K manually rated statements from PolitiFact. The class values exhibit a natural order with respect to truthfulness as shown in Table 1. Thus, our task represents a twist from standard classification, due to the various degrees of similarity between classes. To investigate this, we defined coarse-to-fine classification regimes, which presents new challenge for classification. To address this, we propose BERT-based models. After training, class similarity is sensible over the multi-class datasets, especially in the fine-grained one. Under all the regimes, BERT achieves state of the art, while the additional layers provide insignificant improvement.