Computational Fact-Checking of Online Discourse: Scoring scientific accuracy in climate change related news articles

TL;DR

This paper presents a semi-automated method using large language models and knowledge graphs to assess the scientific accuracy of climate change news articles, aiming to combat misinformation in online media.

Contribution

It introduces a workflow combining LLMs and knowledge graphs for semi-automatic fact-checking of climate-related media content, highlighting current limitations and future needs.

Findings

The tool provides beneficial veracity indications according to expert evaluations.

Current climate knowledge graphs are insufficient for comprehensive fact-checking.

Further development of FAIR knowledge bases is necessary for scalable verification.

Abstract

Democratic societies need reliable information. Misinformation in popular media, such as news articles or videos, threatens to impair civic discourse. Citizens are, unfortunately, not equipped to verify the flood of content consumed daily at increasing rates. This work aims to quantify the scientific accuracy of online media semi-automatically. We investigate the state of the art of climate-related ground truth knowledge representation. By semantifying media content of unknown veracity, their statements can be compared against these ground truth knowledge graphs. We implemented a workflow using LLM-based statement extraction and knowledge graph analysis. Our implementation can streamline content processing towards state-of-the-art knowledge representation and veracity quantification. Developed and evaluated with the help of 27 experts and detailed interviews with 10, the tool evidently provides a beneficial veracity indication. These findings are supported by 43 anonymous participants from a parallel user survey. This initial step, however, is unable to annotate public media at the required granularity and scale. Additionally, the identified state of climate change knowledge graphs is vastly insufficient to support this neurosymbolic fact-checking approach. Further work towards a FAIR (Findable, Accessible, Interoperable, Reusable) ground truth and complementary metrics is required to support civic discourse scientifically.