Counterfactual Conditional Likelihood Rewards for Multiagent Exploration

TL;DR

This paper introduces Counterfactual Conditional Likelihood rewards to improve multiagent exploration by rewarding agents based on their unique contributions to team discovery, especially in sparse reward environments.

Contribution

The paper proposes a novel CCL reward mechanism that isolates individual agent contributions to team exploration, enhancing coordination and learning efficiency.

Findings

CCL rewards accelerate learning in sparse reward domains.

CCL improves coordination in tasks requiring tight agent cooperation.

Experiments demonstrate effectiveness over prior exploration methods.

Abstract

Efficient exploration is critical for multiagent systems to discover coordinated strategies, particularly in open-ended domains such as search and rescue or planetary surveying. However, when exploration is encouraged only at the individual agent level, it often leads to redundancy, as agents act without awareness of how their teammates are exploring. In this work, we introduce Counterfactual Conditional Likelihood (CCL) rewards, which score each agent's exploration by isolating its unique contribution to team exploration. Unlike prior methods that reward agents solely for the novelty of their individual observations, CCL emphasizes observations that are informative with respect to the joint exploration of the team. Experiments in continuous multiagent domains show that CCL rewards accelerate learning for domains with sparse team rewards, where most joint actions yield zero rewards, and are particularly effective in tasks that require tight coordination among agents.