Dynamics Distillation for Efficient and Transferable Control Learning

TL;DR

This paper presents a framework called Sim2Sim2Sim that distills high-fidelity vehicle simulation dynamics into a scalable learned model, enabling efficient policy training and reliable transfer for autonomous driving.

Contribution

The introduction of Sim2Sim2Sim, a novel approach that combines high-fidelity simulation with scalable reinforcement learning through dynamics distillation.

Findings

Distilled environment improves policy optimization efficiency.

Policies trained in the distilled environment transfer reliably to high-fidelity simulators.

Predictive accuracy alone does not determine the quality of the learned dynamics model.

Abstract

Robust control policy learning for autonomous driving requires training environments to be both physically realistic and computationally scalable, properties that existing simulators provide only in isolation. We introduce Sim2Sim2Sim, a framework that bridges high-fidelity vehicle simulation and scalable reinforcement learning by distilling simulator dynamics into a highly parallelizable learned dynamics model. By training control policies purely within this distilled environment and deploying them back into the high-fidelity source simulator, we demonstrate more efficient policy optimization and reliable transfer under challenging dynamics. We further show that predictive accuracy alone does not fully characterize a learned dynamics model's suitability as a reinforcement learning training environment, which should also be assessed by the quality of the policies it enables.