Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AI

TL;DR

This paper presents a dynamic method for aligning AI systems with diverse human preferences using Multi-Objective Reinforcement Learning, enabling adaptive and retroactive preference adjustments to better serve pluralistic values.

Contribution

It introduces a novel framework for post-learning preference adjustment in AI alignment through MORL, addressing the challenge of shifting human values.

Findings

Proposed framework effectively adjusts AI policies to changing preferences.

Discussed advantages of retroactive alignment in sociotechnical systems.

Outlined implementation details and potential benefits of the approach.

Abstract

Emerging research in Pluralistic Artificial Intelligence (AI) alignment seeks to address how intelligent systems can be designed and deployed in accordance with diverse human needs and values. We contribute to this pursuit with a dynamic approach for aligning AI with diverse and shifting user preferences through Multi Objective Reinforcement Learning (MORL), via post-learning policy selection adjustment. In this paper, we introduce the proposed framework for this approach, outline its anticipated advantages and assumptions, and discuss technical details about the implementation. We also examine the broader implications of adopting a retroactive alignment approach through the sociotechnical systems perspective.