ChatGPT's advice drives moral judgments with or without justification

TL;DR

This paper investigates why users follow moral advice from chatbots, finding that users are influenced by advice regardless of reasoning quality, which raises ethical concerns about chatbot influence and the need for digital literacy.

Contribution

It reveals that users follow chatbot moral advice irrespective of justification quality, highlighting the influence of perceived authority and the ethical implications involved.

Findings

Users follow moral advice regardless of reasoning quality.

Attributing advice to a moral advisor does not change user compliance.

Chatbots exacerbate moral dilemma avoidance by providing easily accessible advice.

Abstract

Why do users follow moral advice from chatbots? A chatbot is not an authoritative moral advisor, but it can generate seemingly plausible arguments. Users do not follow reasoned more readily than unreasoned advice, though, we find in an experiment. However, this is also true if we attribute advice to a moral advisor, not a chatbot. Hence, it seems that advice offers users a cheap way to escape from a moral dilemma. This is a concern that chatbots do not raise, but they exacerbate it as they make advice easily accessible. We conclude that it takes ethical in addition to digital literacy to harness users against moral advice from chatbots.