Learning from Simulation, Racing in Reality

TL;DR

This paper demonstrates that a reinforcement learning policy trained in simulation with model randomization and novel regularization can be effectively transferred to a real miniature race car, achieving competitive lap times and safety improvements.

Contribution

It introduces a novel policy regularization and action space design that enables sim-to-real transfer for autonomous racing, outperforming baseline RL methods and approaching MPC performance.

Findings

RL policy outperforms SAC baseline in simulation and real-world.

Refinement with 3 hours of real data achieves MPC-like lap times.

Reinforcement learning reduces track violations by 50%.

Abstract

We present a reinforcement learning-based solution to autonomously race on a miniature race car platform. We show that a policy that is trained purely in simulation using a relatively simple vehicle model, including model randomization, can be successfully transferred to the real robotic setup. We achieve this by using novel policy output regularization approach and a lifted action space which enables smooth actions but still aggressive race car driving. We show that this regularized policy does outperform the Soft Actor Critic (SAC) baseline method, both in simulation and on the real car, but it is still outperformed by a Model Predictive Controller (MPC) state of the art method. The refinement of the policy with three hours of real-world interaction data allows the reinforcement learning policy to achieve lap times similar to the MPC controller while reducing track constraint violations by 50%.