Multi-agent Reinforcement Learning with Deep Networks for Diverse Q-Vectors

TL;DR

This paper introduces a deep Q-network algorithm for multi-agent reinforcement learning that learns diverse Q-vectors using different strategic approaches, demonstrated through a collaborative robotic task.

Contribution

It extends existing MARL methods by enabling deep networks to learn multiple Q-vector strategies like Max, Nash, and Maximin.

Findings

Effective learning of diverse Q-vectors demonstrated

Improved coordination in robotic arm collaboration

Versatile strategy learning in multi-agent environments

Abstract

Multi-agent reinforcement learning (MARL) has become a significant research topic due to its ability to facilitate learning in complex environments. In multi-agent tasks, the state-action value, commonly referred to as the Q-value, can vary among agents because of their individual rewards, resulting in a Q-vector. Determining an optimal policy is challenging, as it involves more than just maximizing a single Q-value. Various optimal policies, such as a Nash equilibrium, have been studied in this context. Algorithms like Nash Q-learning and Nash Actor-Critic have shown effectiveness in these scenarios. This paper extends this research by proposing a deep Q-networks (DQN) algorithm capable of learning various Q-vectors using Max, Nash, and Maximin strategies. The effectiveness of this approach is demonstrated in an environment where dual robotic arms collaborate to lift a pot.