# Correspondence-Free, Function-Based Sim-to-Real Learning for Deformable Surface Control

## TL;DR

This paper introduces a novel correspondence-free, function-based sim-to-real learning approach for deformable surface control, enabling effective transfer from simulation to real-world deformable objects without requiring point correspondences.

## Contribution

It proposes a neural network-based deformation function and confidence map that facilitate sim-to-real transfer without marker correspondences, adaptable to various sensing modalities.

## Key findings

- Effective transfer to real deformable surfaces without correspondences
- Versatile application across vision devices and soft robots
- Seamless integration into inverse kinematics pipelines

## Abstract

This paper presents a correspondence-free, function-based sim-to-real learning method for controlling deformable freeform surfaces. Unlike traditional sim-to-real transfer methods that strongly rely on marker points with full correspondences, our approach simultaneously learns a deformation function space and a confidence map -- both parameterized by a neural network -- to map simulated shapes to their real-world counterparts. As a result, the sim-to-real learning can be conducted by input from either a 3D scanner as point clouds (without correspondences) or a motion capture system as marker points (tolerating missed markers). The resultant sim-to-real transfer can be seamlessly integrated into a neural network-based computational pipeline for inverse kinematics and shape control. We demonstrate the versatility and adaptability of our method on both vision devices and across four pneumatically actuated soft robots: a deformable membrane, a robotic mannequin, and two soft manipulators.