Robot Active Neural Sensing and Planning in Unknown Cluttered Environments

TL;DR

This paper introduces a neural active sensing method enabling robots to efficiently explore and reconstruct unknown cluttered environments using minimal observations, with successful transfer from simulation to real-world scenarios.

Contribution

The approach combines active viewpoint planning, scene aggregation, and object inference trained on synthetic data, demonstrating effective sim-to-real transfer in complex environments.

Findings

High reconstruction accuracy in real-world cluttered environments

Reduced number of viewpoints needed for scene coverage

Faster planning and scene reconstruction compared to baselines

Abstract

Active sensing and planning in unknown, cluttered environments is an open challenge for robots intending to provide home service, search and rescue, narrow-passage inspection, and medical assistance. Although many active sensing methods exist, they often consider open spaces, assume known settings, or mostly do not generalize to real-world scenarios. We present the active neural sensing approach that generates the kinematically feasible viewpoint sequences for the robot manipulator with an in-hand camera to gather the minimum number of observations needed to reconstruct the underlying environment. Our framework actively collects the visual RGBD observations, aggregates them into scene representation, and performs object shape inference to avoid unnecessary robot interactions with the environment. We train our approach on synthetic data with domain randomization and demonstrate its successful execution via sim-to-real transfer in reconstructing narrow, covered, real-world cabinet environments cluttered with unknown objects. The natural cabinet scenarios impose significant challenges for robot motion and scene reconstruction due to surrounding obstacles and low ambient lighting conditions. However, despite unfavorable settings, our method exhibits high performance compared to its baselines in terms of various environment reconstruction metrics, including planning speed, the number of viewpoints, and overall scene coverage.