F1tenth Autonomous Racing With Offline Reinforcement Learning Methods

TL;DR

This paper explores offline reinforcement learning for autonomous F1tenth racing, using expert demonstrations and various policy architectures to improve agent performance and transferability in dynamic racing environments.

Contribution

It introduces an offline RL approach with a return-conditioned decision tree policy for autonomous racing, comparing it with neural network and Transformer-based methods.

Findings

Offline RL improves lap completion rates.

Decision tree policies show competitive transferability.

Method comparison offers insights into policy selection for autonomous driving.

Abstract

Autonomous racing serves as a critical platform for evaluating automated driving systems and enhancing vehicle mobility intelligence. This work investigates offline reinforcement learning methods to train agents within the dynamic F1tenth racing environment. The study begins by exploring the challenges of online training in the Austria race track environment, where agents consistently fail to complete the laps. Consequently, this research pivots towards an offline strategy, leveraging `expert' demonstration dataset to facilitate agent training. A waypoint-based suboptimal controller is developed to gather data with successful lap episodes. This data is then employed to train offline learning-based algorithms, with a subsequent analysis of the agents' cross-track performance, evaluating their zero-shot transferability from seen to unseen scenarios and their capacity to adapt to changes in environment dynamics. Beyond mere algorithm benchmarking in autonomous racing scenarios, this study also introduces and describes the machinery of our return-conditioned decision tree-based policy, comparing its performance with methods that employ fully connected neural networks, Transformers, and Diffusion Policies and highlighting some insights into method selection for training autonomous agents in driving interactions.