Contracting With a Reinforcement Learning Agent by Playing Trick or Treat

TL;DR

This paper addresses principal-agent problems in Markov Decision Processes, proposing algorithms for optimal contracting policies that incentivize agents to act desirably, overcoming challenges posed by hidden actions and history dependence.

Contribution

It introduces an efficient algorithm for computing optimal policies in complex principal-agent MDPs and a method to ensure incentive compatibility with minimal utility loss.

Findings

Designed an algorithm for optimal principal policies in history-dependent MDPs.

Developed a technique to make policies incentive compatible with negligible utility loss.

Extended incentive compatibility methods from classical to general MDP settings.

Abstract

We study principal-agent problems where a farsighted agent takes costly actions in an MDP. The core challenge in these settings is that agent's actions are hidden to the principal, who can only observe their outcomes, namely state transitions and their associated rewards. Thus, the principal's goal is to devise a policy that incentives the agent to take actions leading to desirable outcomes. This is accomplished by committing to a payment scheme (a.k.a. contract) at each step, specifying a monetary transfer from the principal to the agent for every possible outcome. Interestingly, we show that Markovian policies are unfit in these settings, as they do not allow to achieve the optimal principal's utility and are constitutionally intractable. Thus, accounting for history in unavoidable, and this begets considerable additional challenges compared to standard MDPs. Nevertheless, we design an efficient algorithm to compute an optimal policy, leveraging a compact way of representing histories for this purpose. Unfortunately, the policy produced by such an algorithm cannot be readily implemented, as it is only approximately incentive compatible, meaning that the agent is incentivized to take the desired actions only approximately. To fix this, we design an efficient method to make such a policy incentive compatible, by only introducing a negligible loss in principal's utility. This method can be generally applied to any approximately-incentive-compatible policy, and it generalized a related approach that has already been discovered for classical principal-agent problems to more general settings in MDPs.