RoboPanoptes: The All-seeing Robot with Whole-body Dexterity

TL;DR

RoboPanoptes introduces a robot system with whole-body vision and dexterity, enabling complex manipulation tasks through a distributed camera system and visuomotor learning from demonstrations.

Contribution

It presents a novel robot architecture combining whole-body vision and dexterity, with a visuomotor policy learned from demonstrations for complex manipulation.

Findings

Successfully unboxes in narrow spaces

Performs multi-object sweeping and oversized object handling

Outperforms baselines in adaptability and efficiency

Abstract

We present RoboPanoptes, a capable yet practical robot system that achieves whole-body dexterity through whole-body vision. Its whole-body dexterity allows the robot to utilize its entire body surface for manipulation, such as leveraging multiple contact points or navigating constrained spaces. Meanwhile, whole-body vision uses a camera system distributed over the robot's surface to provide comprehensive, multi-perspective visual feedback of its own and the environment's state. At its core, RoboPanoptes uses a whole-body visuomotor policy that learns complex manipulation skills directly from human demonstrations, efficiently aggregating information from the distributed cameras while maintaining resilience to sensor failures. Together, these design aspects unlock new capabilities and tasks, allowing RoboPanoptes to unbox in narrow spaces, sweep multiple or oversized objects, and succeed in multi-step stowing in cluttered environments, outperforming baselines in adaptability and efficiency. Results are best viewed on https://robopanoptes.github.io.