Learning to Communicate with Reinforcement Learning for an Adaptive Traffic Control System

TL;DR

This paper compares independent Q-learning and differentiable inter-agent learning in an adaptive traffic control system, demonstrating that learned communication improves performance and training efficiency in multi-agent reinforcement learning scenarios.

Contribution

It introduces the application of DIAL with learned communication to adaptive traffic control, showing its advantages over independent Q-learning in a realistic simulation.

Findings

DIAL outperforms IQL in training time and reward.

Communication enables better coordination among agents.

Simulation results validate the effectiveness of learned communication.

Abstract

Recent work in multi-agent reinforcement learning has investigated inter agent communication which is learned simultaneously with the action policy in order to improve the team reward. In this paper, we investigate independent Q-learning (IQL) without communication and differentiable inter-agent learning (DIAL) with learned communication on an adaptive traffic control system (ATCS). In real world ATCS, it is impossible to present the full state of the environment to every agent so in our simulation, the individual agents will only have a limited observation of the full state of the environment. The ATCS will be simulated using the Simulation of Urban MObility (SUMO) traffic simulator in which two connected intersections are simulated. Every intersection is controlled by an agent which has the ability to change the direction of the traffic flow. Our results show that a DIAL agent outperforms an independent Q-learner on both training time and on maximum achieved reward as it is able to share relevant information with the other agents.