Rumour Evaluation with Very Large Language Models

TL;DR

This paper explores using large language models like GPT-3.5-turbo and GPT-4 to improve rumour veracity prediction and stance classification on social media, achieving significant performance gains and adding explainability features.

Contribution

It extends RumourEval tasks with prompting-based LLM approaches, introduces multiclass stance classification, and provides confidence scores and justifications for predictions.

Findings

Outperforms previous results in veracity prediction

Comparable performance in stance classification to finetuning methods

Provides confidence scores and explanations for predictions

Abstract

Conversational prompt-engineering-based large language models (LLMs) have enabled targeted control over the output creation, enhancing versatility, adaptability and adhoc retrieval. From another perspective, digital misinformation has reached alarming levels. The anonymity, availability and reach of social media offer fertile ground for rumours to propagate. This work proposes to leverage the advancement of prompting-dependent LLMs to combat misinformation by extending the research efforts of the RumourEval task on its Twitter dataset. To the end, we employ two prompting-based LLM variants (GPT-3.5-turbo and GPT-4) to extend the two RumourEval subtasks: (1) veracity prediction, and (2) stance classification. For veracity prediction, three classifications schemes are experimented per GPT variant. Each scheme is tested in zero-, one- and few-shot settings. Our best results outperform the precedent ones by a substantial margin. For stance classification, prompting-based-approaches show comparable performance to prior results, with no improvement over finetuning methods. Rumour stance subtask is also extended beyond the original setting to allow multiclass classification. All of the generated predictions for both subtasks are equipped with confidence scores determining their trustworthiness degree according to the LLM, and post-hoc justifications for explainability and interpretability purposes. Our primary aim is AI for social good.