Benchmarking Classic and Learned Navigation in Complex 3D Environments

TL;DR

This paper compares classic and learning-based navigation methods in complex 3D environments, evaluating their performance across various settings and sensory inputs, and benchmarking against human performance.

Contribution

It provides a comprehensive evaluation framework for comparing traditional and learned navigation systems in diverse indoor environments.

Findings

Classic navigation performs well when properly tuned in cluttered spaces.

Learned systems are more robust with limited sensors.

Both approaches lag behind human navigation performance.

Abstract

Navigation research is attracting renewed interest with the advent of learning-based methods. However, this new line of work is largely disconnected from well-established classic navigation approaches. In this paper, we take a step towards coordinating these two directions of research. We set up classic and learning-based navigation systems in common simulated environments and thoroughly evaluate them in indoor spaces of varying complexity, with access to different sensory modalities. Additionally, we measure human performance in the same environments. We find that a classic pipeline, when properly tuned, can perform very well in complex cluttered environments. On the other hand, learned systems can operate more robustly with a limited sensor suite. Overall, both approaches are still far from human-level performance.