Just Asking Questions: Doing Our Own Research on Conspiratorial Ideation by Generative AI Chatbots

TL;DR

This study systematically reviews how leading AI chatbots respond to conspiracy theory questions, revealing significant variability in safety measures and highlighting focus areas in safeguarding against harmful content.

Contribution

It provides a comparative analysis of safety guardrails in popular AI chatbots regarding conspiracy theories, highlighting gaps and biases in current safety protocols.

Findings

Safety guardrails vary significantly across chatbots.

Chatbots focus more on preventing racist content and politically sensitive conspiracy theories.

Many chatbots lack comprehensive safety measures for emerging conspiracy theories.

Abstract

Interactive chat systems that build on artificial intelligence frameworks are increasingly ubiquitous and embedded into search engines, Web browsers, and operating systems, or are available on websites and apps. Researcher efforts have sought to understand the limitations and potential for harm of generative AI, which we contribute to here. Conducting a systematic review of six AI-powered chat systems (ChatGPT 3.5; ChatGPT 4 Mini; Microsoft Copilot in Bing; Google Search AI; Perplexity; and Grok in Twitter/X), this study examines how these leading products respond to questions related to conspiracy theories. This follows the platform policy implementation audit approach established by Glazunova et al. (2023). We select five well-known and comprehensively debunked conspiracy theories and four emerging conspiracy theories that relate to breaking news events at the time of data collection. Our findings demonstrate that the extent of safety guardrails against conspiratorial ideation in generative AI chatbots differs markedly, depending on chatbot model and conspiracy theory. Our observations indicate that safety guardrails in AI chatbots are often very selectively designed: generative AI companies appear to focus especially on ensuring that their products are not seen to be racist; they also appear to pay particular attention to conspiracy theories that address topics of substantial national trauma such as 9/11 or relate to well-established political issues. Future work should include an ongoing effort extended to further platforms, multiple languages, and a range of conspiracy theories extending well beyond the United States.