GSDrive: Reinforcing Driving Policies by Multi-mode Future Trajectory Probing with 3D Gaussian Splatting Environment

TL;DR

GSDrive introduces a novel framework combining imitation learning and reinforcement learning with a 3D Gaussian Splatting environment to improve end-to-end autonomous driving policies through future trajectory probing and reward shaping.

Contribution

It presents a multi-mode trajectory probing method using a differentiable 3D environment, enhancing policy learning with dense rewards and iterative refinement.

Findings

Outperforms other simulation-based RL methods on nuScenes dataset.

Uses a cyclic IL-RL training loop for iterative policy improvement.

Demonstrates effective future-aware trajectory evaluation in 3D environment.

Abstract

End-to-end (E2E) autonomous driving aims to directly map sensory observations to driving actions, but its real-world deployment is hindered by evolving data distributions and the high cost of continual annotation. While combining imitation learning (IL) and reinforcement learning (RL) is a common strategy for policy improvement, conventional RL training relies on delayed, event-based rewards, where policies learn only from catastrophic outcomes such as collisions, leading to premature convergence to suboptimal behaviors. To address these limitations, we propose GSDrive, a framework that uses a differentiable 3D Gaussian Splatting (3DGS) environment for future-aware trajectory probing and reward shaping in E2E driving. GSDrive first learns a multi-mode trajectory probe via IL and then uses RL to evaluate multiple candidate futures in the 3DGS environment, converting their simulated returns into dense shaping rewards for policy optimization. This yields a cyclic hybrid IL-RL training loop, where IL supplies structured future priors and RL provides interactive feedback for iterative refinement. Evaluated on the reconstructed nuScenes dataset, our method outperforms other simulation-based RL approaches in closed-loop experiments. Code is available at https://github.com/ZionGo6/GSDrive.