Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration

TL;DR

This paper introduces a novel method combining inverse reinforcement learning and reinforcement learning with a contextual bandit orchestrator to enable autonomous agents to learn and follow societal constraints while maximizing rewards, with transparency and adaptability.

Contribution

It proposes a new approach that learns societal constraints from demonstrations and dynamically mixes constrained and reward-driven policies using a contextual bandit orchestrator.

Findings

The agent successfully learns to act optimally within societal constraints.

The method demonstrates the ability to blend policies for complex decision-making.

The approach is validated in a Pac-Man domain, showing effective constraint adherence and reward maximization.

Abstract

Autonomous cyber-physical agents and systems play an increasingly large role in our lives. To ensure that agents behave in ways aligned with the values of the societies in which they operate, we must develop techniques that allow these agents to not only maximize their reward in an environment, but also to learn and follow the implicit constraints of society. These constraints and norms can come from any number of sources including regulations, business process guidelines, laws, ethical principles, social norms, and moral values. We detail a novel approach that uses inverse reinforcement learning to learn a set of unspecified constraints from demonstrations of the task, and reinforcement learning to learn to maximize the environment rewards. More precisely, we assume that an agent can observe traces of behavior of members of the society but has no access to the explicit set of constraints that give rise to the observed behavior. Inverse reinforcement learning is used to learn such constraints, that are then combined with a possibly orthogonal value function through the use of a contextual bandit-based orchestrator that picks a contextually-appropriate choice between the two policies (constraint-based and environment reward-based) when taking actions. The contextual bandit orchestrator allows the agent to mix policies in novel ways, taking the best actions from either a reward maximizing or constrained policy. In addition, the orchestrator is transparent on which policy is being employed at each time step. We test our algorithms using a Pac-Man domain and show that the agent is able to learn to act optimally, act within the demonstrated constraints, and mix these two functions in complex ways.