Force/Torque Sensing for Soft Grippers using an External Camera

TL;DR

This paper introduces Visual Force/Torque Sensing (VFTS), a novel external camera-based method that visually estimates force/torque data for soft robotic grippers, reducing costs and improving robustness over traditional sensors.

Contribution

The paper presents a deep learning-based external visual sensing approach for force/torque estimation in soft robotics, demonstrating its effectiveness across various tasks and environments.

Findings

VFTS outperforms motor current-based estimates.

It generalizes well to new environments.

It enables autonomous healthcare manipulation tasks.

Abstract

Robotic manipulation can benefit from wrist-mounted force/torque (F/T) sensors, but conventional F/T sensors can be expensive, difficult to install, and damaged by high loads. We present Visual Force/Torque Sensing (VFTS), a method that visually estimates the 6-axis F/T measurement that would be reported by a conventional F/T sensor. In contrast to approaches that sense loads using internal cameras placed behind soft exterior surfaces, our approach uses an external camera with a fisheye lens that observes a soft gripper. VFTS includes a deep learning model that takes a single RGB image as input and outputs a 6-axis F/T estimate. We trained the model with sensor data collected while teleoperating a robot (Stretch RE1 from Hello Robot Inc.) to perform manipulation tasks. VFTS outperformed F/T estimates based on motor currents, generalized to a novel home environment, and supported three autonomous tasks relevant to healthcare: grasping a blanket, pulling a blanket over a manikin, and cleaning a manikin's limbs. VFTS also performed well with a manually operated pneumatic gripper. Overall, our results suggest that an external camera observing a soft gripper can perform useful visual force/torque sensing for a variety of manipulation tasks.