Fact Checking with Insufficient Evidence
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle, Augenstein

TL;DR
This paper emphasizes the importance of models recognizing when evidence is insufficient for fact checking, introduces a new diagnostic dataset, and proposes a data augmentation method that enhances evidence sufficiency detection and fact checking accuracy.
Contribution
It introduces a novel task and dataset for detecting insufficient evidence in fact checking, and proposes a contrastive self-learning augmentation strategy that improves model performance.
Findings
Models struggle with detecting missing evidence in adverbial modifiers (21% accuracy).
Detection accuracy is higher for date modifiers (63%).
Data augmentation improves evidence sufficiency prediction by up to 17.8 F1 and fact checking by 2.6 F1.
Abstract
Automating the fact checking (FC) process relies on information obtained from external sources. In this work, we posit that it is crucial for FC models to make veracity predictions only when there is sufficient evidence and otherwise indicate when it is not enough. To this end, we are the first to study what information FC models consider sufficient by introducing a novel task and advancing it with three main contributions. First, we conduct an in-depth empirical analysis of the task with a new fluency-preserving method for omitting information from the evidence at the constituent and sentence level. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures and three FC datasets. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Software Engineering Research · Natural Language Processing Techniques
MethodsAttention Is All You Need · Linear Layer · Self-Learning · Dropout · Layer Normalization · Label Smoothing · Softmax · Byte Pair Encoding · Position-Wise Feed-Forward Layer · Dense Connections
