Partner Approximating Learners (PAL): Simulation-Accelerated Learning with Explicit Partner Modeling in Multi-Agent Domains

TL;DR

This paper introduces Partner Approximating Learners (PAL), a framework that accelerates reinforcement learning in multi-agent settings by modeling partners explicitly and using simulation to reduce real-world data needs.

Contribution

The paper presents a novel framework combining online partner modeling with simulation-accelerated reinforcement learning for improved multi-agent collaboration.

Findings

Fast learning due to simulation environment

Effective adaptation to changing partners

Reduced real-world data requirements

Abstract

Mixed cooperative-competitive control scenarios such as human-machine interaction with individual goals of the interacting partners are very challenging for reinforcement learning agents. In order to contribute towards intuitive human-machine collaboration, we focus on problems in the continuous state and control domain where no explicit communication is considered and the agents do not know the others' goals or control laws but only sense their control inputs retrospectively. Our proposed framework combines a learned partner model based on online data with a reinforcement learning agent that is trained in a simulated environment including the partner model. Thus, we overcome drawbacks of independent learners and, in addition, benefit from a reduced amount of real world data required for reinforcement learning which is vital in the human-machine context. We finally analyze an example that demonstrates the merits of our proposed framework which learns fast due to the simulated environment and adapts to the continuously changing partner due to the partner approximation.