Unsupervised Elicitation of Moral Values from Language Models

TL;DR

This paper demonstrates that unsupervised methods can reveal latent moral reasoning in pretrained language models, outperforming baselines and reducing social bias, thus offering a scalable approach for AI alignment.

Contribution

The study introduces the Internal Coherence Maximization (ICM) algorithm, showing it can reliably elicit moral judgments from pretrained LMs without supervision, outperforming baselines and reducing social biases.

Findings

ICM outperforms all baselines on Norm Bank and ETHICS benchmarks.

Fine-tuning on ICM labels matches or exceeds human-labeled performance.

ICM significantly reduces social bias errors in chatbot LMs.

Abstract

As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that language models (LMs) exhibit limited inherent moral reasoning, leading to calls for explicit moral teaching. However, constructing ground truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM) algorithm across three benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pre-trained and chatbot baselines on the Norm Bank and ETHICS benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI alignment.