Normative Conflicts and Shallow AI Alignment

TL;DR

This paper critiques current AI alignment methods for large language models, highlighting their inability to handle normative conflicts and arguing for the need of deeper normative reasoning to ensure safety.

Contribution

It identifies the limitations of shallow alignment strategies and emphasizes the importance of normative deliberation inspired by moral psychology for robust AI safety.

Findings

Current alignment methods reinforce superficial behavioral norms.

LLMs lack capacity for normative conflict detection and resolution.

Deeper normative reasoning is necessary for safe AI deployment.

Abstract

The progress of AI systems such as large language models (LLMs) raises increasingly pressing concerns about their safe deployment. This paper examines the value alignment problem for LLMs, arguing that current alignment strategies are fundamentally inadequate to prevent misuse. Despite ongoing efforts to instill norms such as helpfulness, honesty, and harmlessness in LLMs through fine-tuning based on human preferences, they remain vulnerable to adversarial attacks that exploit conflicts between these norms. I argue that this vulnerability reflects a fundamental limitation of existing alignment methods: they reinforce shallow behavioral dispositions rather than endowing LLMs with a genuine capacity for normative deliberation. Drawing from on research in moral psychology, I show how humans' ability to engage in deliberative reasoning enhances their resilience against similar adversarial tactics. LLMs, by contrast, lack a robust capacity to detect and rationally resolve normative conflicts, leaving them susceptible to manipulation; even recent advances in reasoning-focused LLMs have not addressed this vulnerability. This ``shallow alignment'' problem carries significant implications for AI safety and regulation, suggesting that current approaches are insufficient for mitigating potential harms posed by increasingly capable AI systems.