PneuGelSight: Soft Robotic Vision-Based Proprioception and Tactile Sensing

TL;DR

This paper introduces PneuGelSight, a vision-based tactile and proprioceptive sensing system for soft pneumatic robots, utilizing a simulation pipeline for zero-shot transfer from virtual models to real-world applications.

Contribution

The work presents a novel embedded camera-based sensing approach and a comprehensive simulation pipeline for soft robots, enabling high-resolution tactile and proprioceptive feedback with zero-shot sim-to-real transfer.

Findings

Successful implementation of high-resolution tactile sensing

Effective zero-shot transfer from simulation to real robot

Enhanced sensory capabilities for soft robotic manipulators

Abstract

Soft pneumatic robot manipulators are popular in industrial and human-interactive applications due to their compliance and flexibility. However, deploying them in real-world scenarios requires advanced sensing for tactile feedback and proprioception. Our work presents a novel vision-based approach for sensorizing soft robots. We demonstrate our approach on PneuGelSight, a pioneering pneumatic manipulator featuring high-resolution proprioception and tactile sensing via an embedded camera. To optimize the sensor's performance, we introduce a comprehensive pipeline that accurately simulates its optical and dynamic properties, facilitating a zero-shot knowledge transition from simulation to real-world applications. PneuGelSight and our sim-to-real pipeline provide a novel, easily implementable, and robust sensing methodology for soft robots, paving the way for the development of more advanced soft robots with enhanced sensory capabilities.