Combining Optimal Control and Learning for Visual Navigation in Novel Environments

TL;DR

This paper presents a hybrid approach combining model-based control with learning-based perception to enable reliable and efficient robot navigation in unknown, cluttered environments without relying on detailed 3D maps.

Contribution

It introduces a method that couples perception-driven waypoint generation with model-based planning, improving navigation in novel environments over existing geometric or learning-only methods.

Findings

Outperforms purely geometric mapping-based methods in reliability and efficiency.

Works effectively with low frame rates and in real-world cluttered environments.

Generalizes well from simulation to real-world scenarios.

Abstract

Model-based control is a popular paradigm for robot navigation because it can leverage a known dynamics model to efficiently plan robust robot trajectories. However, it is challenging to use model-based methods in settings where the environment is a priori unknown and can only be observed partially through on-board sensors on the robot. In this work, we address this short-coming by coupling model-based control with learning-based perception. The learning-based perception module produces a series of waypoints that guide the robot to the goal via a collision-free path. These waypoints are used by a model-based planner to generate a smooth and dynamically feasible trajectory that is executed on the physical system using feedback control. Our experiments in simulated real-world cluttered environments and on an actual ground vehicle demonstrate that the proposed approach can reach goal locations more reliably and efficiently in novel environments as compared to purely geometric mapping-based or end-to-end learning-based alternatives. Our approach does not rely on detailed explicit 3D maps of the environment, works well with low frame rates, and generalizes well from simulation to the real world. Videos describing our approach and experiments are available on the project website.