MoralityGym: A Benchmark for Evaluating Hierarchical Moral Alignment in Sequential Decision-Making Agents

TL;DR

MoralityGym is a new benchmark with 98 ethical dilemmas designed to evaluate how well AI agents understand and reason about hierarchical moral norms, aiding the development of ethically aligned AI.

Contribution

The paper introduces Morality Chains formalism, MoralityGym benchmark, and a Morality Metric to assess hierarchical moral reasoning in AI agents.

Findings

Baseline Safe RL methods show limitations in moral reasoning.

MoralityGym enables decoupling of task-solving from moral evaluation.

Provides a foundation for more ethically reliable AI systems.

Abstract

Evaluating moral alignment in agents navigating conflicting, hierarchically structured human norms is a critical challenge at the intersection of AI safety, moral philosophy, and cognitive science. We introduce Morality Chains, a novel formalism for representing moral norms as ordered deontic constraints, and MoralityGym, a benchmark of 98 ethical-dilemma problems presented as trolley-dilemma-style Gymnasium environments. By decoupling task-solving from moral evaluation and introducing a novel Morality Metric, MoralityGym allows the integration of insights from psychology and philosophy into the evaluation of norm-sensitive reasoning. Baseline results with Safe RL methods reveal key limitations, underscoring the need for more principled approaches to ethical decision-making. This work provides a foundation for developing AI systems that behave more reliably, transparently, and ethically in complex real-world contexts.