RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation

TL;DR

This paper introduces RIFT, a novel group-relative reinforcement learning fine-tuning method for traffic simulation that improves realism and controllability by combining data-driven imitation learning with physics-based fine-tuning.

Contribution

The paper proposes a dual-stage simulation framework with RIFT, a new RL fine-tuning strategy that enhances style-level controllability and reduces covariate shift in traffic simulation.

Findings

RIFT improves realism in traffic simulation.

RIFT enhances controllability and reduces covariate shift.

The approach exposes limitations of current AV systems in closed-loop scenarios.

Abstract

Achieving both realism and controllability in closed-loop traffic simulation remains a key challenge in autonomous driving. Dataset-based methods reproduce realistic trajectories but suffer from covariate shift in closed-loop deployment, compounded by simplified dynamics models that further reduce reliability. Conversely, physics-based simulation methods enhance reliable and controllable closed-loop interactions but often lack expert demonstrations, compromising realism. To address these challenges, we introduce a dual-stage AV-centric simulation framework that conducts imitation learning pre-training in a data-driven simulator to capture trajectory-level realism and route-level controllability, followed by reinforcement learning fine-tuning in a physics-based simulator to enhance style-level controllability and mitigate covariate shift. In the fine-tuning stage, we propose RIFT, a novel group-relative RL fine-tuning strategy that evaluates all candidate modalities through group-relative formulation and employs a surrogate objective for stable optimization, enhancing style-level controllability and mitigating covariate shift while preserving the trajectory-level realism and route-level controllability inherited from IL pre-training. Extensive experiments demonstrate that RIFT improves realism and controllability in traffic simulation while simultaneously exposing the limitations of modern AV systems in closed-loop evaluation. Project Page: https://currychen77.github.io/RIFT/