Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimization

TL;DR

This paper introduces $ ext{ extalpha}$-RPO, an extension of residual policy learning that produces standalone policies for autonomous racing, reducing system complexity and improving real-world performance.

Contribution

The paper proposes $ ext{ extalpha}$-RPO, a novel method that attenuates the base policy to create a standalone neural policy, enabling privileged learning and better real-world transfer in autonomous racing.

Findings

$ ext{ extalpha}$-RPO reduces system complexity in autonomous racing.

It improves driving performance in simulation and real-world tests.

The method enables zero-shot transfer to real robots.

Abstract

Residual policy learning (RPL), in which a learned policy refines a static base policy using deep reinforcement learning (DRL), has shown strong performance across various robotic applications. Its effectiveness is particularly evident in autonomous racing, a domain that serves as a challenging benchmark for real-world DRL. However, deploying RPL-based controllers introduces system complexity and increases inference latency. We address this by introducing an extension of RPL named attenuated residual policy optimization ($\alpha$-RPO). Unlike standard RPL, $\alpha$-RPO yields a standalone neural policy by progressively attenuating the base policy, which initially serves to bootstrap learning. Furthermore, this mechanism enables a form of privileged learning, where the base policy is permitted to use sensor modalities not required for final deployment. We design $\alpha$-RPO to integrate seamlessly with PPO, ensuring that the attenuated influence of the base controller is dynamically compensated during policy optimization. We evaluate $\alpha$-RPO by building a framework for 1:10-scaled autonomous racing around it. In both simulation and zero-shot real-world transfer to Roboracer cars, $\alpha$-RPO not only reduces system complexity but also improves driving performance compared to baselines - demonstrating its practicality for robotic deployment. Our code is available at: https://github.com/raphajaner/arpo_racing.