LLMs as information warriors? Auditing how LLM-powered chatbots tackle disinformation about Russia's war in Ukraine

TL;DR

This study examines how LLM-powered chatbots from major tech companies address disinformation about Russia's war in Ukraine, revealing varying performance improvements across languages and over time.

Contribution

It provides an empirical analysis of the effectiveness of different LLM chatbots in handling disinformation in multiple languages over time.

Findings

Perplexity chatbot shows significant performance improvement in several languages.

Gemini chatbot improves only in English, declines in low-resource languages.

Performance varies across chatbots and languages over time.

Abstract

The rise of large language models (LLMs) has a significant impact on information warfare. By facilitating the production of content related to disinformation and propaganda campaigns, LLMs can amplify different types of information operations and mislead online users. In our study, we empirically investigate how LLM-powered chatbots, developed by Google, Microsoft, and Perplexity, handle disinformation about Russia's war in Ukraine and whether the chatbots' ability to provide accurate information on the topic varies across languages and over time. Our findings indicate that while for some chatbots (Perplexity), there is a significant improvement in performance over time in several languages, for others (Gemini), the performance improves only in English but deteriorates in low-resource languages.