Reinforcement Learning with Human Feedback for Realistic Traffic Simulation

TL;DR

This paper introduces TrafficRLHF, a reinforcement learning framework that uses human feedback to improve the realism of traffic simulations for autonomous vehicle testing, addressing the challenge of aligning models with human preferences.

Contribution

It presents the first dataset for traffic realism alignment and demonstrates a novel RLHF-based approach to generate human-aligned realistic traffic scenarios.

Findings

TrafficRLHF produces traffic scenarios aligned with human preferences.

The framework outperforms baseline models in realism evaluation.

The new dataset supports future research in traffic realism modeling.

Abstract

In light of the challenges and costs of real-world testing, autonomous vehicle developers often rely on testing in simulation for the creation of reliable systems. A key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge, an aspect that has proven challenging due to the need to balance realism and diversity. This works aims to address this by developing a framework that employs reinforcement learning with human preference (RLHF) to enhance the realism of existing traffic models. This study also identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models. To tackle these issues, we propose using human feedback for alignment and employ RLHF due to its sample efficiency. We also introduce the first dataset for realism alignment in traffic modeling to support such research. Our framework, named TrafficRLHF, demonstrates its proficiency in generating realistic traffic scenarios that are well-aligned with human preferences, as corroborated by comprehensive evaluations on the nuScenes dataset.