Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

TL;DR

This paper extends reinforcement learning from AI feedback to multi-objective urban traffic control, enabling balanced policies that reflect diverse user priorities without complex reward engineering.

Contribution

It introduces a multi-objective RLAIF framework that learns balanced trade-offs among conflicting goals, advancing scalable, user-aligned policy learning in complex systems.

Findings

Multi-objective RLAIF produces balanced policies.

It reduces reliance on reward engineering.

Policies reflect diverse user priorities.

Abstract

Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.