ChatGPT and Bard Responses to Polarizing Questions

TL;DR

This paper introduces a dataset of ChatGPT and Bard responses to polarizing US topics, revealing biases and differences in response patterns, which can inform policies to reduce misinformation and contentious outputs from LLMs.

Contribution

The authors created and analyzed a novel dataset of LLM responses to polarizing topics, highlighting biases and response behaviors of ChatGPT and Bard.

Findings

Both models show a left-leaning bias.

Bard is more likely to respond to polarizing topics.

Bard has fewer guardrails and more human-like responses.

Abstract

Recent developments in natural language processing have demonstrated the potential of large language models (LLMs) to improve a range of educational and learning outcomes. Of recent chatbots based on LLMs, ChatGPT and Bard have made it clear that artificial intelligence (AI) technology will have significant implications on the way we obtain and search for information. However, these tools sometimes produce text that is convincing, but often incorrect, known as hallucinations. As such, their use can distort scientific facts and spread misinformation. To counter polarizing responses on these tools, it is critical to provide an overview of such responses so stakeholders can determine which topics tend to produce more contentious responses -- key to developing targeted regulatory policy and interventions. In addition, there currently exists no annotated dataset of ChatGPT and Bard responses around possibly polarizing topics, central to the above aims. We address the indicated issues through the following contribution: Focusing on highly polarizing topics in the US, we created and described a dataset of ChatGPT and Bard responses. Broadly, our results indicated a left-leaning bias for both ChatGPT and Bard, with Bard more likely to provide responses around polarizing topics. Bard seemed to have fewer guardrails around controversial topics, and appeared more willing to provide comprehensive, and somewhat human-like responses. Bard may thus be more likely abused by malicious actors. Stakeholders may utilize our findings to mitigate misinformative and/or polarizing responses from LLMs