Variational Policy Propagation for Multi-agent Reinforcement Learning

TL;DR

This paper introduces Variational Policy Propagation (VPP), a collaborative multi-agent reinforcement learning method that models joint policies as Markov Random Fields, enabling efficient sampling and improved performance on large-scale tasks.

Contribution

The paper presents a novel VPP algorithm that leverages variational inference to learn joint policies as Markov Random Fields, reducing policy complexity and enhancing scalability.

Findings

VPP effectively models joint policies as Markov Random Fields.

VPP outperforms previous methods on large-scale tasks.

The approach enables efficient sampling and differentiability of policies.

Abstract

We propose a \emph{collaborative} multi-agent reinforcement learning algorithm named variational policy propagation (VPP) to learn a \emph{joint} policy through the interactions over agents. We prove that the joint policy is a Markov Random Field under some mild conditions, which in turn reduces the policy space effectively. We integrate the variational inference as special differentiable layers in policy such that the actions can be efficiently sampled from the Markov Random Field and the overall policy is differentiable. We evaluate our algorithm on several large scale challenging tasks and demonstrate that it outperforms previous state-of-the-arts.