A Hierarchical Approach to Population Training for Human-AI Collaboration

TL;DR

This paper introduces a hierarchical reinforcement learning approach that enables AI agents to adapt dynamically to novel human partners with different play styles and skill levels in collaborative tasks.

Contribution

It proposes a hierarchical method for population training that allows AI to learn multiple response policies and switch between them based on partner behavior, improving collaboration robustness.

Findings

The method adapts to diverse partner styles and skills in Overcooked.

It outperforms non-hierarchical approaches in dynamic partner scenarios.

Human studies confirm improved collaboration with real users.

Abstract

A major challenge for deep reinforcement learning (DRL) agents is to collaborate with novel partners that were not encountered by them during the training phase. This is specifically worsened by an increased variance in action responses when the DRL agents collaborate with human partners due to the lack of consistency in human behaviors. Recent work have shown that training a single agent as the best response to a diverse population of training partners significantly increases an agent's robustness to novel partners. We further enhance the population-based training approach by introducing a Hierarchical Reinforcement Learning (HRL) based method for Human-AI Collaboration. Our agent is able to learn multiple best-response policies as its low-level policy while at the same time, it learns a high-level policy that acts as a manager which allows the agent to dynamically switch between the low-level best-response policies based on its current partner. We demonstrate that our method is able to dynamically adapt to novel partners of different play styles and skill levels in the 2-player collaborative Overcooked game environment. We also conducted a human study in the same environment to test the effectiveness of our method when partnering with real human subjects.