Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies

TL;DR

This paper introduces a two-stage distillation framework where a dense-reward-trained world model guides a sparse-reward policy for autonomous driving, resulting in better generalization and safety on unseen routes.

Contribution

The authors propose reward-privileged world model distillation, enabling the transfer of dense reward knowledge into sparse-reward policies without action imitation, improving generalization in autonomous driving.

Findings

Sparse-reward students outperform dense-reward teachers in CARLA benchmarks.

Distillation improves success rates by 23% on unseen routes.

Students achieve up to 27x success improvement in overtaking on new routes.

Abstract

We study how to exploit dense simulator-defined rewards in vision-based autonomous driving without inheriting their misalignment with deployment metrics. In realistic simulators such as CARLA, privileged state (e.g., lane geometry, infractions, time-to-collision) can be converted into dense rewards that stabilize and accelerate model-based reinforcement learning, but policies trained directly on these signals often overfit and fail to generalize when evaluated on sparse objectives such as route completion and collision-free overtaking. We propose reward-privileged world model distillation, a two-stage framework in which a teacher DreamerV3-style agent is first trained with a dense privileged reward, and only its latent dynamics are distilled into a student trained solely on sparse task rewards. Teacher and student share the same observation space (semantic bird's-eye-view images); privileged information enters only through the teacher's reward, and the student does not imitate the teacher's actions or value estimates. Instead, the student's world model is regularized to match the teacher's latent dynamics while its policy is learned from scratch on sparse success/failure signals. In CARLA lane-following and overtaking benchmarks, sparse-reward students outperform both dense-reward teachers and sparse-from-scratch baselines. On unseen lane-following routes, reward-privileged distillation improves success by about 23 percent relative to the dense teacher while maintaining comparable or better safety. On overtaking, students retain near-perfect performance on training routes and achieve up to a 27x improvement in success on unseen routes, with improved lane keeping. These results show that dense rewards can be leveraged to learn richer dynamics models while keeping the deployed policy optimized strictly for sparse, deployment-aligned objectives.