Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents

TL;DR

This paper introduces an influence-based intrinsic motivation framework for multi-agent reinforcement learning, enhancing exploration and coordination in complex cooperative tasks with sparse rewards.

Contribution

It proposes a novel influence-based intrinsic reward mechanism and a framework for agents to simulate counterfactuals, improving coordination in multi-agent systems.

Findings

Significant performance improvements over baselines in StarCraft Multi-Agent Challenge.

Effective exploration in sparse reward, partially observable environments.

Enhanced coordination protocols learned through influence assessment.

Abstract

Discovering successful coordinated behaviors is a central challenge in Multi-Agent Reinforcement Learning (MARL) since it requires exploring a joint action space that grows exponentially with the number of agents. In this paper, we propose a mechanism for achieving sufficient exploration and coordination in a team of agents. Specifically, agents are rewarded for contributing to a more diversified team behavior by employing proper intrinsic motivation functions. To learn meaningful coordination protocols, we structure agents' interactions by introducing a novel framework, where at each timestep, an agent simulates counterfactual rollouts of its policy and, through a sequence of computations, assesses the gap between other agents' current behaviors and their targets. Actions that minimize the gap are considered highly influential and are rewarded. We evaluate our approach on a set of challenging tasks with sparse rewards and partial observability that require learning complex cooperative strategies under a proper exploration scheme, such as the StarCraft Multi-Agent Challenge. Our methods show significantly improved performances over different baselines across all tasks.