Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning

TL;DR

This paper introduces algorithms that learn societal value systems and aligned policies in multi-agent MDPs using clustering and preference-based multi-objective reinforcement learning, enabling better understanding of diverse human preferences.

Contribution

It presents a novel approach combining clustering and PbMORL to model and learn multiple societal value systems and corresponding policies in a unified framework.

Findings

Outperforms state-of-the-art PbMORL algorithms on value alignment tasks.

Effectively models diverse user preferences through clustering.

Produces approximately Pareto-optimal policies aligned with learned value systems.

Abstract

Value-aware AI should recognise human values and adapt to the value systems (value-based preferences) of different users. This requires operationalization of values, which can be prone to misspecification. The social nature of values demands their representation to adhere to multiple users while value systems are diverse, yet exhibit patterns among groups. In sequential decision making, efforts have been made towards personalization for different goals or values from demonstrations of diverse agents. However, these approaches demand manually designed features or lack value-based interpretability and/or adaptability to diverse user preferences. We propose algorithms for learning models of value alignment and value systems for a society of agents in Markov Decision Processes (MDPs), based on clustering and preference-based multi-objective reinforcement learning (PbMORL). We jointly learn socially-derived value alignment models (groundings) and a set of value systems that concisely represent different groups of users (clusters) in a society. Each cluster consists of a value system representing the value-based preferences of its members and an approximately Pareto-optimal policy that reflects behaviours aligned with this value system. We evaluate our method against a state-of-the-art PbMORL algorithm and baselines on two MDPs with human values.