Physical Human-Robot Interaction for Grasping in Augmented Reality via Rigid-Soft Robot Synergy

TL;DR

This paper introduces an AR-based teleoperation framework for hybrid rigid-soft robots, enabling intuitive grasping in unstructured environments through virtual simulation and real-to-simulation parameter identification.

Contribution

It presents a novel AR interface combined with a parameter identification pipeline for accurate modeling and control of hybrid rigid-soft robots.

Findings

Successful virtual-to-physical task execution demonstrated

Enhanced modeling accuracy through parameter identification

Improved teleoperation intuitiveness in unstructured environments

Abstract

Hybrid rigid-soft robots combine the precision of rigid manipulators with the compliance and adaptability of soft arms, offering a promising approach for versatile grasping in unstructured environments. However, coordinating hybrid robots remains challenging, due to difficulties in modeling, perception, and cross-domain kinematics. In this work, we present a novel augmented reality (AR)-based physical human-robot interaction framework that enables direct teleoperation of a hybrid rigid-soft robot for simple reaching and grasping tasks. Using an AR headset, users can interact with a simulated model of the robotic system integrated into a general-purpose physics engine, which is superimposed on the real system, allowing simulated execution prior to real-world deployment. To ensure consistent behavior between the virtual and physical robots, we introduce a real-to-simulation parameter identification pipeline that leverages the inherent geometric properties of the soft robot, enabling accurate modeling of its static and dynamic behavior as well as the control system's response.