Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA

TL;DR

This paper introduces a novel semantic rollout and adversarial regularization approach to improve zero-shot transfer of fixed-route driving agents in CARLA, demonstrating enhanced success rates in unseen towns under controlled conditions.

Contribution

It proposes a new training method combining semantic prediction and town-adversarial supervision to enhance zero-shot generalization in autonomous driving within simulated environments.

Findings

The proposed model achieves the highest success rate among Dreamer-family methods in unseen towns.

Semantic rollout supervision improves route completion in zero-shot transfer.

Town-adversarial regularization contributes to better generalization in the evaluated setting.

Abstract

Learned driving agents often degrade when deployed in unseen environments. This paper studies a deliberately bounded instance of that problem in the CARLA simulator: zero-shot transfer of a closed-loop fixed-route driving agent from Town05 and Town06 to unseen Town03 and Town04. The study isolates structural town shift by keeping weather fixed to ClearNoon and removing traffic and pedestrians. We build on a Dreamer-style latent world-model agent and add two training-only auxiliary losses: multi-horizon prediction of future visual-semantic embeddings along imagined rollouts and town-adversarial supervision on a semantic projection of the recurrent latent state. A causal context feature conditions the semantic rollout predictor, while the actor and critic retain the standard control feature. The policy receives no navigation command, route polyline, goal pose, or map input; the reference route is used only by the environment for reward, progress, success, and termination. Across the evaluated held-out towns, the proposed model achieves the highest mean success rate among the included Dreamer-family methods. Secondary safety and lane-keeping metrics are mixed across towns. These results support a bounded conclusion: in this controlled fixed-weather CARLA setting, semantic rollout supervision combined with town-adversarial regularization improves mean held-out-town route completion.