Hybrid Information-driven Multi-agent Reinforcement Learning

TL;DR

This paper proposes a hybrid approach combining information theory and reinforcement learning to improve the efficiency of multi-agent systems in large, sparse state spaces, showing significant exploration improvements.

Contribution

It introduces a novel hybrid MARL framework that uses information theoretic heuristics to guide exploration and learning in resource-constrained multi-agent environments.

Findings

Agents are approximately 1000 times more efficient at exploring sparse state spaces.

The approach effectively integrates information heuristics with RL for better exploration.

Preliminary results indicate promising directions for scalable multi-agent control.

Abstract

Information theoretic sensor management approaches are an ideal solution to state estimation problems when considering the optimal control of multi-agent systems, however they are too computationally intensive for large state spaces, especially when considering the limited computational resources typical of large-scale distributed multi-agent systems. Reinforcement learning (RL) is a promising alternative which can find approximate solutions to distributed optimal control problems that take into account the resource constraints inherent in many systems of distributed agents. However, the RL training can be prohibitively inefficient, especially in low-information environments where agents receive little to no feedback in large portions of the state space. We propose a hybrid information-driven multi-agent reinforcement learning (MARL) approach that utilizes information theoretic models as heuristics to help the agents navigate large sparse state spaces, coupled with information based rewards in an RL framework to learn higher-level policies. This paper presents our ongoing work towards this objective. Our preliminary findings show that such an approach can result in a system of agents that are approximately three orders of magnitude more efficient at exploring a sparse state space than naive baseline metrics. While the work is still in its early stages, it provides a promising direction for future research.