SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty

TL;DR

SPIRIT introduces a perceptive shared autonomy framework that dynamically switches between autonomous and teleoperated control based on perception confidence, enhancing robustness and safety in robotic manipulation under deep learning uncertainty.

Contribution

The paper presents a novel uncertainty-aware perception system using Neural Tangent Kernels and integrates it into a shared autonomy framework for improved robotic manipulation.

Findings

Enhanced manipulation reliability in challenging scenarios

Successful user study with 15 participants demonstrating robustness

SPIRIT system outperforms traditional approaches in safety-critical tasks

Abstract

Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.