Incorporating Behavioral Constraints in Online AI Systems

TL;DR

This paper introduces a novel online learning agent that incorporates behavioral constraints learned from observation, ensuring decision-making aligns with ethical or regulatory standards without significantly sacrificing reward optimization.

Contribution

It proposes a new algorithm called Behavior Constrained Thompson Sampling (BCTS) that extends contextual bandits to include learned behavioral constraints in online decision-making.

Findings

The agent effectively learns and adheres to behavioral constraints.

It maintains near-optimal reward performance while respecting constraints.

Theoretical regret bounds are established for the proposed algorithm.

Abstract

AI systems that learn through reward feedback about the actions they take are increasingly deployed in domains that have significant impact on our daily life. However, in many cases the online rewards should not be the only guiding criteria, as there are additional constraints and/or priorities imposed by regulations, values, preferences, or ethical principles. We detail a novel online agent that learns a set of behavioral constraints by observation and uses these learned constraints as a guide when making decisions in an online setting while still being reactive to reward feedback. To define this agent, we propose to adopt a novel extension to the classical contextual multi-armed bandit setting and we provide a new algorithm called Behavior Constrained Thompson Sampling (BCTS) that allows for online learning while obeying exogenous constraints. Our agent learns a constrained policy that implements the observed behavioral constraints demonstrated by a teacher agent, and then uses this constrained policy to guide the reward-based online exploration and exploitation. We characterize the upper bound on the expected regret of the contextual bandit algorithm that underlies our agent and provide a case study with real world data in two application domains. Our experiments show that the designed agent is able to act within the set of behavior constraints without significantly degrading its overall reward performance.