Beyond Specialization: Robust Reinforcement Learning Navigation via Procedural Map Generators

TL;DR

This paper investigates how procedural map generators affect the generalization of reinforcement learning navigation policies, demonstrating that combined generators and subgoal inputs significantly improve robustness and transfer to real-world scenarios.

Contribution

It systematically compares different procedural map generators and identifies key factors like subgoal inputs that enhance policy robustness and transferability.

Findings

Combined generators achieve 91.5% success rate, outperforming individual ones.

Subgoal inputs from A* path-planner boost success to 98.9%.

Recurrent policies outperform feedforward and classical controllers at higher speeds.

Abstract

Deep reinforcement learning (DRL) navigation policies often overfit to the structure of their training environments, as environmental diversity is typically constrained by the manual effort required to design diverse scenarios. While procedural map generation offers scalable diversity, no prior work systematically compares how different generator types affect policy generalization. We integrate four generators (sparse, maze, graph, and Wave Function Collapse) with guaranteed navigability into MuRoSim, a 2D simulator focusing on training efficiency for LiDAR-based navigation. We cross-evaluate five navigation policies on 1000 seeded maps per generator across three training seeds. Results show a strongly asymmetric cross-generator transfer: a specialist trained on sparse layouts falls to 3.3% success on mazes, whereas a policy trained on the combined generator set achieves 91.5 +/- 1.1% mean success. We further demonstrate that A* path-planner subgoal inputs are the dominant factor for robustness, raising success from the 90.2 +/- 1.4% feedforward baseline to 98.9 +/- 0.4% and outperforming GRU recurrence, which only improves the reactive baseline. The DRL policies outperform a classical Carrot+A* controller, which matches their success only at low speeds (1.0 m/s) but collapses to 24.9% at 2.0 m/s. This highlights learned speed adaptation as the decisive advantage of the learned approach. Real-world experiments on a RoboMaster confirm sim-to-real transfer in a cluttered arena, while a maze-like layout exposes remaining failure modes that recurrence helps mitigate.