Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification

TL;DR

This paper evaluates the effectiveness of large language models in detecting and classifying scientific claims related to COVID-19, aiming to combat misinformation on social media platforms.

Contribution

It provides a comparative analysis of LLMs' performance in scientific claim detection and classification, proposing a framework for their use in public health communication.

Findings

LLMs show promise as automated fact-checking tools

Performance varies across different LLM architectures

Further research is needed to optimize LLM applications in misinformation detection

Abstract

The pervasive influence of social media during the COVID-19 pandemic has been a double-edged sword, enhancing communication while simultaneously propagating misinformation. This \textit{Digital Infodemic} has highlighted the urgent need for automated tools capable of discerning and disseminating factual content. This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter. LLMs, such as OpenAI's GPT and Meta's LLaMA, offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models. We assess the performance of LLMs in detecting and classifying COVID-19-related scientific claims, thus facilitating informed decision-making. Our findings indicate that LLMs have significant potential as automated fact-checking tools, though research in this domain is nascent and further exploration is required. We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.