Smart But Not Moral? Moral Alignment In Human-AI Decision-Making

TL;DR

This paper explores the importance of moral alignment in human-AI decision-making, emphasizing how congruence between AI values and stakeholder morals impacts AI integration in sensitive areas.

Contribution

It introduces the concept of moral alignment, grounded in Moral Foundations Theory, as a key dimension beyond functional alignment for human-AI decision-making.

Findings

Moral alignment influences stakeholder trust and acceptance.

Misalignment can lead to ethical conflicts and reduced AI effectiveness.

The paper advocates for incorporating moral considerations into AI design.

Abstract

In high-stakes AI-supported decisions, considerations are not purely technical but involve moral judgments about fairness, responsibility, and harm. While prior research has focused mainly on functional or behavioral alignment, this paper argues that moral alignment may be a more fundamental dimension of human-AI decision-making. Moral alignment is defined as the perceived congruence between the values embedded in an AI system's decision logic and the moral intuitions of stakeholders. Building on Moral Foundations Theory, the paper adopts a multi-stakeholder perspective and highlights why moral (mis)alignment matters for the meaningful integration of AI in sensitive contexts.