Drive Fast, Learn Faster: On-Board RL for High Performance Autonomous Racing

TL;DR

This paper presents a novel on-board reinforcement learning framework for autonomous racing that eliminates the need for simulation pre-training, enabling real-time adaptation and outperforming existing controllers with minimal training.

Contribution

It introduces a residual Soft Actor-Critic algorithm with multi-step TD learning and heuristic reward adjustment for real-time autonomous racing without simulation pre-training.

Findings

Achieved up to 11.5% reduction in lap times over state-of-the-art methods.

Demonstrated real-time learning and adaptation on the F1TENTH platform.

Surpassed previous best results with an end-to-end RL controller trained on track.

Abstract

Autonomous racing presents unique challenges due to its non-linear dynamics, the high speed involved, and the critical need for real-time decision-making under dynamic and unpredictable conditions. Most traditional Reinforcement Learning (RL) approaches rely on extensive simulation-based pre-training, which faces crucial challenges in transfer effectively to real-world environments. This paper introduces a robust on-board RL framework for autonomous racing, designed to eliminate the dependency on simulation-based pre-training enabling direct real-world adaptation. The proposed system introduces a refined Soft Actor-Critic (SAC) algorithm, leveraging a residual RL structure to enhance classical controllers in real-time by integrating multi-step Temporal-Difference (TD) learning, an asynchronous training pipeline, and Heuristic Delayed Reward Adjustment (HDRA) to improve sample efficiency and training stability. The framework is validated through extensive experiments on the F1TENTH racing platform, where the residual RL controller consistently outperforms the baseline controllers and achieves up to an 11.5 % reduction in lap times compared to the State-of-the-Art (SotA) with only 20 min of training. Additionally, an End-to-End (E2E) RL controller trained without a baseline controller surpasses the previous best results with sustained on-track learning. These findings position the framework as a robust solution for high-performance autonomous racing and a promising direction for other real-time, dynamic autonomous systems.