Learning When to Cooperate Under Heterogeneous Goals

TL;DR

This paper introduces a hierarchical learning approach for agents to decide when to cooperate in environments with heterogeneous goals, improving collaboration effectiveness.

Contribution

It extends ad hoc teamwork to heterogeneous goals and proposes a novel hierarchical imitation-reinforcement learning method for better cooperation.

Findings

Outperforms baseline methods in cooperative environments

Hierarchical approach effectively learns when to cooperate

Auxiliary teammate modeling component's impact varies with observable information

Abstract

A significant element of human cooperative intelligence lies in our ability to identify opportunities for fruitful collaboration; and conversely to recognise when the task at hand is better pursued alone. Research on flexible cooperation in machines has left this meta-level problem largely unexplored, despite its importance for successful collaboration in heterogeneous open-ended environments. Here, we extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap. We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning, and show that it outperforms baseline methods across extended versions of two cooperative environments. We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.