Event-Based Communication in Distributed Q-Learning

TL;DR

This paper introduces an event-based communication method for distributed Q-learning that significantly reduces data transmission while maintaining convergence, applicable to multi-agent systems.

Contribution

It proposes the EBd-Q system, combining event-triggered communication with convergence guarantees in distributed Q-learning.

Findings

Substantial reduction in communication data rates.

Convergence guarantees similar to standard Q-learning.

Potential for application in complex multi-agent systems.

Abstract

We present an approach to reduce the communication of information needed on a Distributed Q-Learning system inspired by Event Triggered Control (ETC) techniques. We consider a baseline scenario of a distributed Q-learning problem on a Markov Decision Process (MDP). Following an event-based approach, N agents explore the MDP and communicate experiences to a central learner only when necessary, which performs updates of the actor Q functions. We design an Event Based distributed Q learning system (EBd-Q), and derive convergence guarantees with respect to a vanilla Q-learning algorithm. We present experimental results showing that event-based communication results in a substantial reduction of data transmission rates in such distributed systems. Additionally, we discuss what effects (desired and undesired) these event-based approaches have on the learning processes studied, and how they can be applied to more complex multi-agent systems.