Beyond Ethical Alignment: Evaluating LLMs as Artificial Moral Assistants

TL;DR

This paper introduces a new framework and benchmark to evaluate large language models as Artificial Moral Assistants, emphasizing their capacity for explicit moral reasoning beyond superficial alignment, and reveals significant variability and shortcomings in current models.

Contribution

It develops a formal philosophical framework and a corresponding benchmark to assess LLMs' moral reasoning abilities as AMAs, highlighting areas for improvement.

Findings

Models show variability in moral reasoning capabilities.

Persistent shortcomings in abductive moral reasoning.

Need for dedicated strategies to improve moral reasoning in LLMs.

Abstract

The recent rise in popularity of large language models (LLMs) has prompted considerable concerns about their moral capabilities. Although considerable effort has been dedicated to aligning LLMs with human moral values, existing benchmarks and evaluations remain largely superficial, typically measuring alignment based on final ethical verdicts rather than explicit moral reasoning. In response, this paper aims to advance the investigation of LLMs' moral capabilities by examining their capacity to function as Artificial Moral Assistants (AMAs), systems envisioned in the philosophical literature to support human moral deliberation. We assert that qualifying as an AMA requires more than what state-of-the-art alignment techniques aim to achieve: not only must AMAs be able to discern ethically problematic situations, they should also be able to actively reason about them, navigating between conflicting values outside of those embedded in the alignment phase. Building on existing philosophical literature, we begin by designing a new formal framework of the specific kind of behaviour an AMA should exhibit, individuating key qualities such as deductive and abductive moral reasoning. Drawing on this theoretical framework, we develop a benchmark to test these qualities and evaluate popular open LLMs against it. Our results reveal considerable variability across models and highlight persistent shortcomings, particularly regarding abductive moral reasoning. Our work connects theoretical philosophy with practical AI evaluation while also emphasising the need for dedicated strategies to explicitly enhance moral reasoning capabilities in LLMs. Code available at https://github.com/alessioGalatolo/AMAeval