Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real

TL;DR

This paper presents a method for transferring multi-agent reinforcement learning policies from simulation to real-world autonomous driving, demonstrating improved performance over rule-based methods and analyzing the impact of domain randomization.

Contribution

It introduces a novel approach combining MAPPO and domain randomization for effective Sim-to-Real transfer in multi-agent autonomous driving.

Findings

Transferred policies outperform rule-based methods by 1.85 times in rewards.

Different levels of parameter randomization significantly affect the Sim-to-Real gap.

The method successfully transfers multi-agent policies to the Duckietown testbed.

Abstract

Autonomous Driving requires high levels of coordination and collaboration between agents. Achieving effective coordination in multi-agent systems is a difficult task that remains largely unresolved. Multi-Agent Reinforcement Learning has arisen as a powerful method to accomplish this task because it considers the interaction between agents and also allows for decentralized training -- which makes it highly scalable. However, transferring policies from simulation to the real world is a big challenge, even for single-agent applications. Multi-agent systems add additional complexities to the Sim-to-Real gap due to agent collaboration and environment synchronization. In this paper, we propose a method to transfer multi-agent autonomous driving policies to the real world. For this, we create a multi-agent environment that imitates the dynamics of the Duckietown multi-robot testbed, and train multi-agent policies using the MAPPO algorithm with different levels of domain randomization. We then transfer the trained policies to the Duckietown testbed and compare the use of the MAPPO algorithm against a traditional rule-based method. We show that the rewards of the transferred policies with MAPPO and domain randomization are, on average, 1.85 times superior to the rule-based method. Moreover, we show that different levels of parameter randomization have a substantial impact on the Sim-to-Real gap.