A Vision-based Sensing Approach for a Spherical Soft Robotic Arm

TL;DR

This paper introduces a vision-based proprioceptive sensing method for a soft robotic arm using embedded cameras and neural networks, enabling accurate, real-time orientation estimation without mechanical sensors.

Contribution

It presents a novel, non-mechanical, camera-based sensing approach integrated into soft actuators, with a neural network predicting orientation from visual data.

Findings

Achieves about one degree accuracy in orientation estimation.

Enables real-time control of the soft robotic arm using visual proprioception.

Demonstrates effective closed-loop control based on the sensory feedback.

Abstract

Sensory feedback is essential for the control of soft robotic systems and to enable deployment in a variety of different tasks. Proprioception refers to sensing the robot's own state and is of crucial importance in order to deploy soft robotic systems outside of laboratory environments, i.e. where no external sensing, such as motion capture systems, is available. A vision-based sensing approach for a soft robotic arm made from fabric is presented, leveraging the high-resolution sensory feedback provided by cameras. No mechanical interaction between the sensor and the soft structure is required and consequently, the compliance of the soft system is preserved. The integration of a camera into an inflatable, fabric-based bellow actuator is discussed. Three actuators, each featuring an integrated camera, are used to control the spherical robotic arm and simultaneously provide sensory feedback of the two rotational degrees of freedom. A convolutional neural network architecture predicts the two angles describing the robot's orientation from the camera images. Ground truth data is provided by a motion capture system during the training phase of the supervised learning approach and its evaluation thereafter. The camera-based sensing approach is able to provide estimates of the orientation in real-time with an accuracy of about one degree. The reliability of the sensing approach is demonstrated by using the sensory feedback to control the orientation of the robotic arm in closed-loop.