Distributed Online Planning for Min-Max Problems in Networked Markov Games

TL;DR

This paper introduces a distributed online planning algorithm for multi-agent networked Markov games that approximates min-max solutions, improving worst-agent performance through local sampling and distributed optimization.

Contribution

It presents a novel modular, distributed, online planning approach for min-max problems in networked Markov games with neighborhood-dependent dynamics.

Findings

Effective in formation control simulations

Converges to optimal actions via distributed optimization

Handles neighborhood-dependent transition and reward functions

Abstract

Min-max problems are important in multi-agent sequential decision-making because they improve the performance of the worst-performing agent in the network. However, solving the multi-agent min-max problem is challenging. We propose a modular, distributed, online planning-based algorithm that is able to approximate the solution of the min-max objective in networked Markov games, assuming that the agents communicate within a network topology and the transition and reward functions are neighborhood-dependent. This set-up is encountered in the multi-robot setting. Our method consists of two phases at every planning step. In the first phase, each agent obtains sample returns based on its local reward function, by performing online planning. Using the samples from online planning, each agent constructs a concave approximation of its underlying local return as a function of only the action of its neighborhood at the next planning step. In the second phase, the agents deploy a distributed optimization framework that converges to the optimal immediate next action for each agent, based on the function approximations of the first phase. We demonstrate our algorithm's performance through formation control simulations.