Multi-agent Markov Entanglement

TL;DR

This paper reveals that the effectiveness of value decomposition in multi-agent RL hinges on a property called Markov entanglement, which parallels quantum entanglement, and introduces measures to quantify and bound the decomposition error.

Contribution

It establishes a mathematical link between value decomposition and Markov entanglement, providing a new theoretical framework and practical tools for multi-agent RL.

Findings

Value decomposition is effective when Markov entanglement is low.

Index policies are shown to be weakly entangled with sublinear decomposition error.

A practical method to estimate Markov entanglement in multi-agent systems.

Abstract

Value decomposition has long been a fundamental technique in multi-agent dynamic programming and reinforcement learning (RL). Specifically, the value function of a global state $(s_1,s_2,\ldots,s_N)$ is often approximated as the sum of local functions: $V(s_1,s_2,\ldots,s_N)\approx\sum_{i=1}^N V_i(s_i)$. This approach traces back to the index policy in restless multi-armed bandit problems and has found various applications in modern RL systems. However, the theoretical justification for why this decomposition works so effectively remains underexplored. In this paper, we uncover the underlying mathematical structure that enables value decomposition. We demonstrate that a multi-agent Markov decision process (MDP) permits value decomposition if and only if its transition matrix is not "entangled" -- a concept analogous to quantum entanglement in quantum physics. Drawing inspiration from how physicists measure quantum entanglement, we introduce how to measure the "Markov entanglement" for multi-agent MDPs and show that this measure can be used to bound the decomposition error in general multi-agent MDPs. Using the concept of Markov entanglement, we proved that a widely-used class of index policies is weakly entangled and enjoys a sublinear $\mathcal O(\sqrt{N})$ scale of decomposition error for $N$-agent systems. Finally, we show how Markov entanglement can be efficiently estimated in practice, providing practitioners with an empirical proxy for the quality of value decomposition.