Deep Decentralized Reinforcement Learning for Cooperative Control

TL;DR

This paper introduces a novel deep decentralized multi-agent reinforcement learning framework designed to improve cooperative control among agents with unknown goals, addressing challenges like non-stationarity and credit assignment.

Contribution

The paper presents a modular deep decentralized reinforcement learning approach that incorporates dynamic sample prioritization, system dynamics modeling, and artificial experience generation for cooperative control.

Findings

Effective in simulated nonlinear cooperative control tasks

Handles non-stationarity and multi-agent credit assignment

Improves coordination among agents

Abstract

In order to collaborate efficiently with unknown partners in cooperative control settings, adaptation of the partners based on online experience is required. The rather general and widely applicable control setting, where each cooperation partner might strive for individual goals while the control laws and objectives of the partners are unknown, entails various challenges such as the non-stationarity of the environment, the multi-agent credit assignment problem, the alter-exploration problem and the coordination problem. We propose new, modular deep decentralized Multi-Agent Reinforcement Learning mechanisms to account for these challenges. Therefore, our method uses a time-dependent prioritization of samples, incorporates a model of the system dynamics and utilizes variable, accountability-driven learning rates and simulated, artificial experiences in order to guide the learning process. The effectiveness of our method is demonstrated by means of a simulated, nonlinear cooperative control task.