BCR-DRL: Behavior- and Context-aware Reward for Deep Reinforcement Learning in Human-AI Coordination

TL;DR

This paper introduces BCR-DRL, a novel reward mechanism for deep reinforcement learning in human-AI coordination, which leverages behavior and context to improve exploration, exploitation, and overall performance in complex tasks.

Contribution

It proposes a behavior- and context-aware reward scheme that enhances exploration and exploitation in DRL for human-AI coordination, addressing sparse rewards and unpredictable human behaviors.

Findings

Increases cumulative sparse rewards by ~20%.

Improves sample efficiency by ~38%.

Demonstrates effectiveness in Overcooked environment.

Abstract

Deep reinforcement Learning (DRL) offers a powerful framework for training AI agents to coordinate with human partners. However, DRL faces two critical challenges in human-AI coordination (HAIC): sparse rewards and unpredictable human behaviors. These challenges significantly limit DRL to identify effective coordination policies, due to its impaired capability of optimizing exploration and exploitation. To address these limitations, we propose an innovative behavior- and context-aware reward (BCR) for DRL, which optimizes exploration and exploitation by leveraging human behaviors and contextual information in HAIC. Our BCR consists of two components: (i) A novel dual intrinsic rewarding scheme to enhance exploration. This scheme composes an AI self-motivated intrinsic reward and a human-motivated intrinsic reward, which are designed to increase the capture of sparse rewards by a logarithmic-based strategy; and (ii) A new context-aware weighting mechanism for the designed rewards to improve exploitation. This mechanism helps the AI agent prioritize actions that better coordinate with the human partner by utilizing contextual information that can reflect the evolution of learning. Extensive simulations in the Overcooked environment demonstrate that our approach can increase the cumulative sparse rewards by approximately 20%, and improve the sample efficiency by around 38% compared to state-of-the-art baselines.