The moral authority of ChatGPT

TL;DR

This paper investigates the moral authority of ChatGPT, revealing its inconsistent advice and subtle influence on users' moral judgments, raising concerns about responsible AI use and proposing digital literacy training as a mitigation.

Contribution

It provides empirical evidence of ChatGPT's influence on moral judgment and discusses the limitations of transparency, suggesting digital literacy training as a new approach.

Findings

ChatGPT influences users' moral judgments even when they know it's a bot.

Users underestimate how much they are influenced by ChatGPT.

Transparency alone is ineffective in mitigating influence.

Abstract

ChatGPT is not only fun to chat with, but it also searches information, answers questions, and gives advice. With consistent moral advice, it might improve the moral judgment and decisions of users, who often hold contradictory moral beliefs. Unfortunately, ChatGPT turns out highly inconsistent as a moral advisor. Nonetheless, it influences users' moral judgment, we find in an experiment, even if they know they are advised by a chatting bot, and they underestimate how much they are influenced. Thus, ChatGPT threatens to corrupt rather than improves users' judgment. These findings raise the question of how to ensure the responsible use of ChatGPT and similar AI. Transparency is often touted but seems ineffective. We propose training to improve digital literacy.