Measuring Large Language Models Capacity to Annotate Journalistic Sourcing

TL;DR

This paper proposes a benchmark to evaluate large language models' ability to identify and annotate sourcing in journalistic stories, highlighting current limitations and the need for improved transparency in AI-assisted journalism.

Contribution

It introduces a novel scenario, dataset, and metrics for assessing LLMs on sourcing annotation in journalism, addressing a gap in existing benchmarks.

Findings

LLMs currently struggle to identify all sourced statements.

Matching source types remains challenging for LLMs.

Spotting source justifications is particularly difficult.

Abstract

Since the launch of ChatGPT in late 2022, the capacities of Large Language Models and their evaluation have been in constant discussion and evaluation both in academic research and in the industry. Scenarios and benchmarks have been developed in several areas such as law, medicine and math (Bommasani et al., 2023) and there is continuous evaluation of model variants. One area that has not received sufficient scenario development attention is journalism, and in particular journalistic sourcing and ethics. Journalism is a crucial truth-determination function in democracy (Vincent, 2023), and sourcing is a crucial pillar to all original journalistic output. Evaluating the capacities of LLMs to annotate stories for the different signals of sourcing and how reporters justify them is a crucial scenario that warrants a benchmark approach. It offers potential to build automated systems to contrast more transparent and ethically rigorous forms of journalism with everyday fare. In this paper we lay out a scenario to evaluate LLM performance on identifying and annotating sourcing in news stories on a five-category schema inspired from journalism studies (Gans, 2004). We offer the use case, our dataset and metrics and as the first step towards systematic benchmarking. Our accuracy findings indicate LLM-based approaches have more catching to do in identifying all the sourced statements in a story, and equally, in matching the type of sources. An even harder task is spotting source justifications.