Learning Human-like Representations to Enable Learning Human Values

TL;DR

This paper investigates how aligning AI representations with human-like understanding can facilitate safe, efficient learning of diverse human values, improving generalization and robustness in value-based AI systems.

Contribution

It demonstrates that representational alignment supports safe exploration and generalization in learning human values across multiple domains, validated through theoretical and empirical analysis.

Findings

Aligning AI with human-like representations improves value learning safety.

Representational alignment enhances generalization across diverse human values.

Empirical results show improved exploration and value judgment accuracy.

Abstract

How can we build AI systems that can learn any set of individual human values both quickly and safely, avoiding causing harm or violating societal standards for acceptable behavior during the learning process? We explore the effects of representational alignment between humans and AI agents on learning human values. Making AI systems learn human-like representations of the world has many known benefits, including improving generalization, robustness to domain shifts, and few-shot learning performance. We demonstrate that this kind of representational alignment can also support safely learning and exploring human values in the context of personalization. We begin with a theoretical prediction, show that it applies to learning human morality judgments, then show that our results generalize to ten different aspects of human values -- including ethics, honesty, and fairness -- training AI agents on each set of values in a multi-armed bandit setting, where rewards reflect human value judgments over the chosen action. Using a set of textual action descriptions, we collect value judgments from humans, as well as similarity judgments from both humans and multiple language models, and demonstrate that representational alignment enables both safe exploration and improved generalization when learning human values.