An in-depth experimental study of sensor usage and visual reasoning of robots navigating in real environments

TL;DR

This paper conducts an in-depth experimental analysis of visual navigation in real environments, comparing traditional and deep learning approaches, and demonstrating that simulation-trained agents can perform competitively in real-world robot navigation without explicit sim2real transfer.

Contribution

It provides a comprehensive evaluation of sensor usage, reasoning, and generalization of deep learning-based navigation agents deployed on real robots, highlighting their capabilities and limitations.

Findings

Simulation-trained agents can perform well in real environments without explicit transfer.

Sensor signals and visual reasoning are crucial for navigation performance.

Pre-training on diverse tasks enhances real-world navigation capabilities.

Abstract

Visual navigation by mobile robots is classically tackled through SLAM plus optimal planning, and more recently through end-to-end training of policies implemented as deep networks. While the former are often limited to waypoint planning, but have proven their efficiency even on real physical environments, the latter solutions are most frequently employed in simulation, but have been shown to be able learn more complex visual reasoning, involving complex semantical regularities. Navigation by real robots in physical environments is still an open problem. End-to-end training approaches have been thoroughly tested in simulation only, with experiments involving real robots being restricted to rare performance evaluations in simplified laboratory conditions. In this work we present an in-depth study of the performance and reasoning capacities of real physical agents, trained in simulation and deployed to two different physical environments. Beyond benchmarking, we provide insights into the generalization capabilities of different agents training in different conditions. We visualize sensor usage and the importance of the different types of signals. We show, that for the PointGoal task, an agent pre-trained on wide variety of tasks and fine-tuned on a simulated version of the target environment can reach competitive performance without modelling any sim2real transfer, i.e. by deploying the trained agent directly from simulation to a real physical robot.