Generating Media Background Checks for Automated Source Critical Reasoning

TL;DR

This paper introduces a new NLP task of generating media background checks to assess source credibility, supported by a dataset and evaluation of models and human usefulness, addressing a gap in source critical reasoning.

Contribution

It presents a novel dataset of media background checks, defines a new task, and evaluates models and human utility, advancing source criticism in NLP.

Findings

Retrieval improves model performance significantly.

Media background checks aid human understanding.

Background checks are beneficial for retrieval-augmented models.

Abstract

Not everything on the internet is true. This unfortunate fact requires both humans and models to perform complex reasoning about credibility when working with retrieved information. In NLP, this problem has seen little attention. Indeed, retrieval-augmented models are not typically expected to distrust retrieved documents. Human experts overcome the challenge by gathering signals about the context, reliability, and tendency of source documents - that is, they perform source criticism. We propose a novel NLP task focused on finding and summarising such signals. We introduce a new dataset of 6,709 "media background checks" derived from Media Bias / Fact Check, a volunteer-run website documenting media bias. We test open-source and closed-source LLM baselines with and without retrieval on this dataset, finding that retrieval greatly improves performance. We furthermore carry out human evaluation, demonstrating that 1) media background checks are helpful for humans, and 2) media background checks are helpful for retrieval-augmented models.