Interactive Inverse Reinforcement Learning for Cooperative Games

TL;DR

This paper introduces an interactive inverse reinforcement learning approach for cooperative games where an autonomous agent learns to cooperate with a suboptimal partner without access to the joint reward, focusing on efficient reward learning and near-optimal policy development.

Contribution

It proposes a novel framework for interactive inverse reinforcement learning in two-agent cooperative settings, analyzing how to efficiently learn the reward function through agent interactions.

Findings

Reward function can be learned efficiently when policies significantly influence transitions.

The first agent can optimize its actions to rapidly infer the joint reward.

The approach enables near-optimal joint policies without direct access to the joint reward.

Abstract

We study the problem of designing autonomous agents that can learn to cooperate effectively with a potentially suboptimal partner while having no access to the joint reward function. This problem is modeled as a cooperative episodic two-agent Markov decision process. We assume control over only the first of the two agents in a Stackelberg formulation of the game, where the second agent is acting so as to maximise expected utility given the first agent's policy. How should the first agent act in order to learn the joint reward function as quickly as possible and so that the joint policy is as close to optimal as possible? We analyse how knowledge about the reward function can be gained in this interactive two-agent scenario. We show that when the learning agent's policies have a significant effect on the transition function, the reward function can be learned efficiently.