Correspondence-Free, Function-Based Sim-to-Real Learning for Deformable Surface Control

TL;DR

This paper introduces a novel correspondence-free, function-based sim-to-real learning approach for deformable surface control, enabling effective transfer from simulation to real-world deformable objects without requiring point correspondences.

Contribution

It proposes a neural network-based deformation function and confidence map that facilitate sim-to-real transfer without marker correspondences, adaptable to various sensing modalities.

Findings

Effective transfer to real deformable surfaces without correspondences

Versatile application across vision devices and soft robots

Seamless integration into inverse kinematics pipelines

Abstract

This paper presents a correspondence-free, function-based sim-to-real learning method for controlling deformable freeform surfaces. Unlike traditional sim-to-real transfer methods that strongly rely on marker points with full correspondences, our approach simultaneously learns a deformation function space and a confidence map -- both parameterized by a neural network -- to map simulated shapes to their real-world counterparts. As a result, the sim-to-real learning can be conducted by input from either a 3D scanner as point clouds (without correspondences) or a motion capture system as marker points (tolerating missed markers). The resultant sim-to-real transfer can be seamlessly integrated into a neural network-based computational pipeline for inverse kinematics and shape control. We demonstrate the versatility and adaptability of our method on both vision devices and across four pneumatically actuated soft robots: a deformable membrane, a robotic mannequin, and two soft manipulators.

Tables

Table 1. TABLE I : List of Notations

Symbol	Description	Symbol	Description
$𝐚$	Actuation parameters (i.e., chamber pressures)	${𝒮}^t$	The target body shape to be achieved by IK
$𝒮$	The simulated shape of the deformable surface	${𝒮}^{\ast }$	The corrected shape obtained from our sim-to-real method
${𝒮}^c$	The compact shape descriptor (i.e., control points) of $𝒮$	${𝒮}^p$	The 3D shape acquired on the physical setup
${𝒩}_{fk}$	Forward kinematics network that predicts ${𝒮}^c$ from $𝐚$	${𝒩}_{rbf}$	Function prediction network that predicts $𝜸$ by ${𝒮}^c$
$𝜸=[{𝜶}_0,\dots {𝜷}_i]$	Variables to determine the function space	${𝒩}_{s2r}$	Combination of ${𝒩}_{rbf}$ and $𝚽(\cdot )$
$𝐩$, ${𝐩}^{\ast }$, ${𝐩}^p\in {ℝ}^3$	The surface point on $𝒮$, ${𝒮}^{\ast }$ and ${𝒮}^p$ respectively	$\left\{{𝐪}_i\right\}$	Centers of Gaussian kernels in $𝚽\left(𝐩\right)$
$𝚽(\cdot )$	The space warping function used to deform $𝒮$ to ${𝒮}^{\ast }$	${𝐜}^{\ast }$	Closet point of ${𝐩}^{\ast }$ on ${𝒮}^t$
$(𝐑,𝐭)$	Rotation matrix and translation vector to re-pose ${𝒮}^t$	$𝐁(\cdot )$	B-Spline function as decoder mapping $(u_p,v_p)$ to $𝐩$
$(u_p,v_p)$	A point in the parametric domain of B-spline surface	$M_u,M_v$	# of control points along $u$ and $v$ directions respectively
$w_c(\cdot )$	Real-value confidence map on the simulated shape	${𝒩}_{conf}$	Confidence estimation network that predicts $w_c(\cdot )$

Table 2. TABLE II : Training Details for Different Hardware Setups

Network	Detail Parameters	Deformable Membrane	Soft Manipulator	Robotic Mannequin		Compressible Manipulator
		(3D Scanner)	(3D Scanner)	(3D Scanner)	(MoCap)	(MoCap)
${𝒩}_{fk}$	Actuation DoFs	9	6	9		9
	# of Simulated Shapes for Training	1,000	1,000	1,000		4,000
	Avg. Time (sec.) for Simulating Each Pose	110.0	900.0	150.0		5.0
	# of Hidden Layers	2	2	2		3
	# of Neurons per Hidden Layer	128	128	128		128
	# of Control Points	30 $\times$ 30	30 $\times$ 30	30 $\times$ 30		30 $\times$ 30
	# of Kernels†	100	100	100	34	100
	# of Hidden Layer	2	2	2	2	3
	# of Neurons per Hidden Layer	24	24	24	24	64
${𝒩}_{s2r}$	# of Frames Captured for Training	40	30	40	40	500
	# of Sample Points Used for Each Frame	200	400	1000
	Total # of Valid Sample Points	7,661	9,891	31,588
	% of Missing Sample Points	4.2%	17.6%	21.0%
	# of Frames with Missing Sample Points	5	16	25
	# of Makers Used for Each Frame				34	400
	Total # of Valid Markers				1,330	180,362
	% of Missing Markers				2.2%	9.8%
	# of Frames with Missing Markers				18	161
${𝒩}_{conf}$	# of Hidden Layer	3	3	3	3	3
	# of Neurons per Hidden Layer	128	128	128	128	128

Table 3. TABLE III : Statistics of IK computing time for each iteration

Target	Average Time of Each Iteration (Unit: ms)				Total Iter. #
	NN-based FK Pipeline		ICP	Total Time†
	Forward	Gradient
M1	65.8	137.7	171.0	374.5	10
M2	75.7	139.6	174.3	389.6	22
M3	83.5	118.8	175.2	377.5	19
M4	69.9	120.6	213.8	404.3	15
S1	65.1	131.1	210.7	406.9	13
S2	74.8	133.8	180.3	388.9	25
S3	75.1	125.3	193.4	393.8	12
S4	80.1	125.8	186.7	392.6	17
S5	68.3	139.3	179.2	386.8	14
S6	73.5	122.4	178.5	374.4	19

Equations

$$\mathbf{p}=\mathbf{B}(u,v,{\mathcal{S}}^c)=\sum _{i=1}^{M_u}\sum _{j=1}^{M_v}{N}_{i,k}(u)N_{j,l}(v){\mathcal{S}}_{ij}^c.$$

$${\mathbf{p}}^{\ast }=\mathbf{\Phi }(\mathbf{p})={\mathbf{\alpha }}_0+A\mathbf{p}+\sum _{i=1}^N{\mathbf{\beta }}_i{e}^{-c\Vert \mathbf{p}-{\mathbf{q}}_i{\Vert }^2}$$

$$\mathbf{\gamma }={\mathcal{N}}_{rbf}({\mathcal{S}}^c)$$

$${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathbf{p}_{i}^{*}=\Phi_{\bm{\gamma}}(\mathbf{p})=\Phi_{\mathcal{N}_{rbf}(\mathcal{S}_{j}^{c})}\left(\mathbf{B}(u_{i},v_{i},\mathcal{S}_{j}^{c})\right).}}$$

$${\mathbf{p}}^{\ast }={\mathcal{N}}_{s2r}(\mathbf{B}(u_p,v_p,{\mathcal{S}}^c(\mathbf{a})),{\mathcal{S}}^c(\mathbf{a}),\{\mathbf{B}(u_q,v_q,{\mathcal{S}}^c(\mathbf{a})\}).$$

$$\tilde{D}({\mathcal{S}}_j^{\ast },{\mathcal{S}}_j^p)$$

$$+\frac{1}{|{\mathcal{S}}_j^p|}\sum _{\mathbf{x}\in {\mathcal{S}}_j^p}\underset{{\mathbf{p}}^{\ast }\in {\mathcal{S}}_j^{\ast }}{\min }\Vert \mathbf{x}-{\mathbf{p}}^{\ast }{\Vert }_2^2,$$

$${\mathcal{L}}_{cd}=\sum _j\tilde{D}({\mathcal{S}}_j^{\ast },{\mathcal{S}}_j^p).$$

$${\mathcal{L}}_{cr}:=\sum _j\sum _{\mathbf{p}\in {\mathcal{S}}_j}(1-w_c(\mathbf{p}){)}^2.$$

$${\mathcal{L}}_{nc}:=\sum _j\sum _{\mathbf{x}\in {\mathcal{S}}_j^p}(1-\mathbf{n}(\mathbf{x})\cdot \mathbf{n}(\mathbf{c}(\mathbf{x}))),$$

$$\sum _{i=1}^N{\mathbf{\beta }}_i=\sum _{i=1}^N{\mathbf{\beta }}_i{\left({\mathbf{q}}_i\right)}_x$$

$${\mathcal{L}}_{gr}$$

$$+\sum _j\Vert \sum _{i=1}^N{\mathbf{\beta }}_i({\mathbf{q}}_i{)}_y{\Vert }^2+\sum _j\Vert \sum _{i=1}^N{\mathbf{\beta }}_i({\mathbf{q}}_i{)}_z{\Vert }^2.$$

$${\mathcal{L}}_{total}:={\mathcal{L}}_{cd}+{\omega }_{cr}{\mathcal{L}}_{cr}+{\omega }_{nc}{\mathcal{L}}_{nc}+{\omega }_{gr}{\mathcal{L}}_{gr}$$

$$\tilde{D}({\mathcal{S}}_j^{\ast },{\mathcal{S}}_j^p)=\frac{1}{|{\mathcal{S}}_j|}\sum _{{\mathbf{p}}_i\in {\mathcal{S}}_j}{w}_c({\mathbf{p}}_i)\Vert {\mathbf{p}}_i^{\ast }-{\mathbf{x}}_i{\Vert }_2^2;$$

$${\mathcal{S}}^c={\mathcal{N}}_{fk}(\mathbf{a}).$$

$$\arg \underset{\mathbf{a},\mathbf{R},\mathbf{t}}{\min }D({\mathcal{S}}^{\ast }(\mathbf{a}),{\mathcal{S}}^t)=\sum _{{\mathbf{p}}^{\ast }\in {\mathcal{S}}^{\ast }(\mathbf{a})}\Vert {\mathbf{p}}^{\ast }-\left(\mathbf{R}{\mathbf{c}}^{\ast }+\mathbf{t}\right){\Vert }^2,$$

$$\frac{\partial D}{\partial \mathbf{a}}$$

$$\frac{\partial {\mathbf{p}}^{\ast }}{\partial \mathbf{a}}$$

$$=\left(\frac{\partial {\mathcal{N}}_{s2r}}{\partial \mathbf{B}}\frac{\partial \mathbf{B}}{\partial {\mathcal{N}}_{fk}}+\frac{\partial {\mathcal{N}}_{s2r}}{\partial {\mathcal{N}}_{fk}}+\frac{\partial {\mathcal{N}}_{s2r}}{\partial \mathbf{Q}}\frac{\partial \mathbf{Q}}{\partial {\mathcal{N}}_{fk}}\right)\frac{\partial {\mathcal{N}}_{fk}}{\partial \mathbf{a}}.$$

$$D_{\text{Haus}}({\mathcal{S}}_j^{\ast },{\mathcal{S}}_j^p)$$

$$\underset{\mathbf{x}\in {\mathcal{S}}_j^p}{\max }\underset{{\mathbf{p}}^{\ast }\in {\mathcal{S}}_j^{\ast }}{\min }\Vert {\mathbf{p}}^{\ast }-\mathbf{x}{\Vert }_2),$$

$$D_{\text{Cham}}({\mathcal{S}}_j^{\ast },{\mathcal{S}}_j^p)$$

$$+\frac{1}{|{\mathcal{S}}_j^p|}\sum _{\mathbf{x}\in {\mathcal{S}}_j^p}\underset{{\mathbf{p}}^{\ast }\in {\mathcal{S}}_j^{\ast }}{\min }\Vert {\mathbf{p}}^{\ast }-\mathbf{x}{\Vert }_2^2$$

$${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathcal{L}_{direct}=D_{\text{Haus}}+5.0D_{\text{Cham}}.}}$$

$$\eta ({\mathbf{p}}_j)=1-\mathbf{n}({\mathbf{p}}_j)\cdot \mathbf{n}({\mathbf{c}}^p({\mathbf{p}}_j))$$

$$\frac{\partial {\mathcal{N}}_{s2r}}{\partial \mathbf{Q}}$$

$$\frac{\partial {\mathbf{p}}^{\ast }}{\partial {\mathbf{q}}_i}$$

$$\frac{\partial e^{-c\Vert \mathbf{p}-{\mathbf{q}}_i{\Vert }^2}}{\partial {\mathbf{q}}_i}=$$

$$-\frac{\partial e^{-c\Vert \mathbf{p}-{\mathbf{q}}_i{\Vert }^2}}{\partial |\mathbf{p}-{\mathbf{q}}_i|}\cdot ((\mathbf{p}-{\mathbf{q}}_i{)}^T(\mathbf{p}-{\mathbf{q}}_i){)}^{-\frac{1}{2}}(\mathbf{p}-{\mathbf{q}}_i).$$

$$\frac{\partial {\mathcal{N}}_{s2r}}{\partial \mathbf{B}}=\frac{\partial {\mathbf{p}}^{\ast }}{\partial \mathbf{p}}$$

$$\frac{\partial e^{-c\Vert \mathbf{p}-{\mathbf{q}}_i{\Vert }^2}}{\partial \mathbf{p}}=-\frac{\partial e^{-c\Vert \mathbf{p}-{\mathbf{q}}_i{\Vert }^2}}{\partial {\mathbf{q}}_i}.$$

$$\frac{\partial {\mathbf{p}}^{\ast }}{\partial \mathbf{\gamma }}$$

Full text

I Introduction

Soft robotic systems have demonstrated the ability to dynamically adapt their shapes in response to programmed actuation or environmental stimuli. Existing prototypes were developed for haptic interfaces (ref. [15, 32, 28, 23, 21]) and soft manipulators (ref. [7, 33, 30]). Many of these soft robotic systems have free-form surfaces with large deformation to be controlled – e.g., a soft robotic mannequin developed in [34], which is driven by pneumatic actuators to mimic the front shapes of different human bodies.

Shape control of such freeform surfaces is challenging due to the infinite degrees-of-freeform (DoFs) deformation embedded in the elastic bodies of soft robots.

I-A Kinematics for Shape Control

An essential shape control problem of soft robotics is to compute the actuation (denoted by a vector $\mathbf{a}$) that can drive the free-form surface $\mathcal{S}$ to approach a target shape $\mathcal{S}^{t}$. It presents much higher DoFs compared to the articulated robots, which leads to high complexity in both the forward shape estimation (i.e., predicting the resultant shape $\mathcal{S}$) and the inverse kinematics (IK) computation to determine the values of actuation parameters $\mathbf{a}$. Earlier works have been developed to conduct an iteration-based IK solver with Jacobian physically evaluated on hardware setup (e.g., [34, 45]). However, this approach is time-consuming in both computation and data acquisition. Additionally, its performance is unstable as the Jacobian evaluation is highly dependent on the accuracy of shapes captured on the physical setup, which is hard to be guaranteed in practice.

In the existing research works of soft robotics, high DoFs in the configuration space are typically computed by either reduced analytical models or discrete numerical models (see the comprehensive review in [1, 2]). Reduced analytical models are often adopted for soft manipulators with simple bending deformation (e.g., ref. [24, 33]), where the surface deformation is trivial in general. Differently, numerical simulation is usually employed to establish the mapping $\mathcal{S}(\mathbf{a})$ between the actuation parameters $\mathbf{a}$ and the deformed freeform shapes $\mathcal{S}$ (ref. [10, 17, 8, 20]). The differentiation of this forward kinematic mapping can then be used to solve the IK problem for determining the actuation parameters $\mathbf{a}$ according to the target shape $\mathcal{S}^{t}$. This was formulated as an optimization problem to iteratively run a simulator to determine the optimal actuation $\mathbf{a}_{opt}$ that minimizes the difference between $\mathcal{S}(\mathbf{a})$ and $\mathcal{S}^{t}$ [11]. However, numerical simulation in general cannot be computed efficiently to realize a fast IK solver. A widely adopted strategy to mitigate the efficiency issue is to train a neural network (NN) by the simulation dataset as a surrogate model for the forward shape estimation (e.g., ref. [16, 4, 13]). This paper also adopts this strategy for IK computation.

I-B Challenges of Sim-to-Real Transfer

Freeform surfaces predicted by numerical simulation often diverge from physically deformed shapes due to hardware uncertainties and simplifications in modeling. These discrepancies become significant during large deformations of a soft robot’s surface (see Fig. 1(a) for an example). To address this issue, sim-to-real transfer networks have been trained using marker-based MoCap systems in prior works [46, 9, 13], where marker positions collected from physical experiments are used to align simulation outputs with real-world shapes. However, marker-based methods face three major limitations.

• •

First, they offer limited spatial coverage, as only a sparse set of markers can be tracked. For example, only the position of the end effector is corrected in [13], while the shape of the entire manipulator remains unaccounted errors.

• •

Second, these methods require full correspondence between captured and simulated points, making them incompatible with 3D scanners that output unstructured point clouds (see Fig.1(b)).

• •

Lastly, marker-based capture is not robust to occlusion. In cases of significant deformation, some markers may be hidden from views – such as the missing belly markers in Fig.1(c), making the affected frames unusable.

Consequently, a correspondence-free method for sim-to-real learning is needed to correct shape errors under large deformations.

I-C Our Method

This paper focuses on a learning-based sim-to-real transfer method that corrects shape errors in NN-based forward prediction trained on simulation data. The method establishes a function-based mapping between simulated and real deformation responses, enabling efficient and accurate NN-based IK computation for soft robot control. It builds on a learning-based theoretical foundation rather than a new control strategy.

Using a compact B-spline based representation of freeform surfaces, we propose a function-based pipeline that enables robust sim-to-real learning even when the input lacks of point correspondences (e.g., 3D scanned point clouds) or contains missing markers (e.g., MoCap data) / regions (e.g., 3D scan). First of all, it is based on a new architecture that learns the function space of shape deformation represented by radial basis functions (RBF). Specifically, each simulated free-form surface is approximated by a B-spline surface, the control points of which are employed as the shape descriptor in a lower dimension. The input of our sim-to-real network is the control points of a simulated body shape, and the output is the coefficients of RBFs that define a spatial warping function $\bm{\Phi}(\mathbf{p})$ by using the positions of kernel centers (Sec. II). The sim-to-real network $\mathcal{N}{s2r}$ consists of two components enclosed by the black dashed lines as shown in Fig. 2. Rather than relying on explicit point correspondences, it is trained jointly with a network $\mathcal{N}{conf}$ that predicts a confidence map. This map serves as a weighting function for evaluating the Chamfer distance between simulated and captured shapes. The training objective of sim-to-real transfer minimizes this weighted distance across numerous pairs of simulated and captured shapes, as illustrated in the differentiable alignment training module highlighted by the red dashed lines in Fig. 2. Details of this training process will be presented in Sec. III.

After learning, the space warping function $\bm{\Phi}(\mathbf{p})$ is obtained in a closed-form and gives a continuous mapping from the shape of a simulated model to the shape of a corrected model that matches the reality. Working together with the forward kinematics prediction that is trained by simulation, this forms an end-to-end NN-based pipeline that can accurately predict the physical shapes of robotic deformable surfaces, which is crucial for efficient IK computation. This computation pipeline based on NN is analytically differentiable so that the gradient-based iteration for IK can be computed efficiently (details given in Sec. IV-A). The quality of IK solutions will be verified by scanning the physical shape $\mathcal{S}^{p}$ to compare with the input target shape $\mathcal{S}^{t}$. All notations used in this paper are summarized in Table I.

The technique contributions of our work are summarized as follows:

• •

A resilient sim-to-real learning pipeline for deformable free-form surfaces that bridges the reality gap through a space of RBF-based spatial warping functions;

• •

A differentiable alignment based joint training method that eliminates the need for correspondences between simulated and captured freeform surfaces;

• •

A neural network-based method that can efficiently solve the inverse kinematics problem for shape control in soft deformable robots.

The effectiveness of our sim-to-real learning and the resultant fast IK solver has been verified on four pneumatically actuated soft robots, including a deformable membrane, a deformable robotic mannequin, and two distinct soft manipulators.

This article significantly extends our preliminary work presented in [35]. First, we newly introduce an advanced differentiable alignment-based training method (Sec. III) that enables correspondence-free sim-to-real learning using unorganized point clouds with partial occlusions instead of relying on fully corresponding markers captured by a MoCap system. In addition, we have conducted extensive validation and experiments on three new types of soft robots, including a deformable membrane and two distinct soft manipulators to demonstrate the generality and robustness of our approach (Sec. V).

II Deformation Function Space for Sim-to-Real

This section introduces our function based sim-to-real learning for shape prediction. Given a set of points $\{\mathbf{p}\in\mathbb{R}^{3}\}$ on the free-form surface of a simulated model $\mathcal{S}$, we propose a method to learn a network $\mathcal{N}_{s2r}$ that can predict a deformation function for mapping these points onto a corrected model $\mathcal{S}^{*}$ (i.e., the corrected shape that has eliminated the sim-to-real gap). The shape difference between $\mathcal{S}^{*}$ and the physical shape $\mathcal{S}^{p}$ should be minimized by the sim-to-real transfer. $\mathcal{N}_{s2r}$ is expected to be a general network that can handle different shapes on the same hardware111Different networks still need be trained for different soft robotic setups..

II-A Compact Representation by B-Spline Surface

Using all sample points on the surface of $\mathcal{S}$ may introduce redundant information and therefore requires a complex network for prediction. To overcome this issue, we first convert the freeform surface $\mathcal{S}$ of a soft deformable robot into a B-spline surface representation through 1) the parametrization taken at its rest shape (see Fig.3(a-c) for an example) and 2) the surface fitting conducted at every deformed shapes to determine control points of B-spline surface (denoted by $\mathcal{S}^{c}$ and see Fig.3(d)). The control points and B-Spline surfaces for four different soft robots are given in Fig.3(e). We then employ the control points of B-spline surface $\mathcal{S}^{c}$ as the shape descriptor of $\mathcal{S}$, with which any point in the parametric domain with the parameter $(u_{p},v_{p})$ can be mapped to a point $\mathbf{p}\in\mathbb{R}^{3}$ as $\mathbf{p}=\mathbf{B}(u_{p},v_{p},\mathcal{S}^{c})$. According to the definition of a B-spline surface [26], we have

[TABLE]

The sample point $\mathbf{p}$ on the B-spline surface is the sum of the linear combination of the B-spline basis function (i.e., $N_{ik}(u)$ and $N_{jl}(v)$) and the control point $\mathcal{S}^{c}_{ij}$. $k$ and $l$ denote the order of B-spline basis function in $u$ and $v$ direction, where cubic B-spline is chosen in the paper (i.e., $k=l=4$). $M_{u}$ and $M_{v}$ are the number of control points along $u$ and $v$ direction respectively. This function $\mathbf{B}(\cdot)$ is named as the B-spline based shape decoder in our computational pipeline (illustrated as the light green block at the top of Fig.2).

The dimension of $\mathcal{S}^{c}$ using B-spline representation as $M_{u}\times M_{v}$ is much lower than mesh representation. For instance, we use $30\times 30$ control points for all robots as shown in Fig.3(e). Note that, since the surfaces obtained via simulation retain an unchanged mesh topology (i.e., preserving correspondence), the computation of B-spline surface fitting is based on the parameterization determined by the rest shape of $\mathcal{S}$.

II-B RBF-based Spatial Warping

Considering the gap between the simulated shape $\mathcal{S}$ and the physical shape $\mathcal{S}^{p}$, we propose to model it as continuous spatial warping by the radial basis functions (RBF) for any 3D point $\mathbf{p}\in\mathbb{R}^{3}$ as follows.

[TABLE]

with $N$ being the number of kernels employed for the space warping and $A=[\bm{\alpha}_{1},\bm{\alpha}_{2},\bm{\alpha}_{3}]$. The centers of Gaussian kernels, $\{\mathbf{q}_{i}=\mathbf{B}(u_{i},v_{i},\mathcal{S}^{c})\}$, are chosen by sparsely sampling across the UV domain to achieve comprehensive coverage of the simulated model’s surface with the control points $\mathcal{S}^{c}$. The number of kernels for different robots will be provided in Table II. The value of coefficient $c$ controlling the width of the Gaussian kernel is chosen as $3.0\times 10^{-5}$ by experiments. We remark that $\bm{\gamma}=[\bm{\alpha}_{0},\bm{\alpha}_{1},...\bm{\beta}_{i}]$ with $\bm{\alpha}_{j},\bm{\beta}_{i}\in\mathbb{R}^{3}$ are variables of the function space to be determined via the learning process. Note that the same kernel on different simulated shapes will have the same UV parameters but different center-positions in $\mathbb{R}^{3}$.

With a determined warping function $\bm{\Phi}(\mathbf{p})$, we are able to map any point on the simulated surface $\mathcal{S}$ denoted as $\mathbf{p}\in\mathcal{S}$ to its corrected position as $\mathbf{p}^{*}\in\mathcal{S}^{*}$ – see Fig.4 for the illustration. However, different warping functions – by changing the values of $\gamma$ – need to be employed for different simulated shapes. Specifically, we introduce a network $\mathcal{N}_{rbf}$ to predict the function coefficients $\bm{\gamma}$ according to the simulated shape $\mathcal{S}$ as illustrated in Fig.2.

II-C NN-based Deformation Function Space

Our function-prediction based sim-to-real pipeline is formed by three building blocks: 1) the network $\mathcal{N}_{rbf}$, 2) the B-spline based shape decoder $\mathbf{B}(\cdot)$ and 3) the RBF-based space warping $\Phi(\cdot)$ (i.e., Eq.(2)). The diagram has been shown in the top of Fig.2. The purpose of network $\mathcal{N}_{rbf}$ is to generate shape-dependent RBF functions by determining their weights as

[TABLE]

with its input as $M_{u}\times M_{v}$ control points of a simulated shape (denoted as $\mathcal{S}^{c}$) and the output $\bm{\gamma}=[\bm{\alpha}_{0},\bm{\alpha}_{1},...\bm{\beta}_{i}]$ as the collection of $N+4$ coefficient vectors in $\mathbb{R}^{3}$.

When a deformable robot is actuated into various shapes $\{\mathcal{S}_{j}\}$ by corresponding actuation $\{\mathbf{a}_{j}\}$, the function-prediction network $\mathcal{N}_{rbf}$ can be trained on pairs of shape differences between the shape $\mathcal{S}_{j}^{*}$ predicted by the sim-to-real network $\mathcal{N}_{s2r}$ and the physically captured shape $\mathcal{S}_{j}^{p}$. More specifically, for the $j$-th deformation, the surface of the simulated shape $\mathcal{S}_{j}$ is sampled as a set of points $\{(u_{i},v_{i})\}$ in its parametric domain. After applying sim-to-real correction, the resultant surface $\mathcal{S}_{j}^{*}$ is represented as a point set $\mathcal{S}_{j}^{*}=\{\mathbf{p}^{*}_{i}\}$, where each point can be computed by

[TABLE]

The warping function $\bm{\Phi}_{j}$ is parameterized by $\bm{\gamma}$, which is predicted by the network as $\bm{\gamma}=\mathcal{N}_{rbf}(\mathcal{S}^{c}_{j})$. The corresponding physically acquired shape $\mathcal{S}_{j}^{p}$ is also represented as a point set $\{\mathbf{p}^{p}_{k}\}$. For training the network $\mathcal{N}_{rbf}$, the shape differences between all pairs of point clouds as $S_{j}^{*}=\{\mathbf{p}_{k}^{*}\}$ and $S_{j}^{p}=\{\mathbf{p}^{p}_{k}\}$ are employed to define loss functions to be minimized. Note that our loss functions are defined in a way that does not require pre-defined correspondences between two point clouds and also tolerate missed regions. Details will be presented in Sec. III.

Given a well trained network $\mathcal{N}_{rbf}(\cdot)$, the sim-to-real transfer can be evaluated by the pipeline at the top of Fig.2 in the inference phase. For any point $(u_{p},v_{p})$ sampled in the UV domain on a simulated shape compactly encoded as a set of control points $\mathcal{S}^{c}$, we can obtain its predicted position by Eq.(4). By modeling $\mathcal{S}^{c}$ as a neural network–based function of the actuation vector $\mathbf{a}$, the above sim-to-real transfer process can be interpreted as a network $\mathcal{N}_{s2r}(\cdot)$ that gives

[TABLE]

All operators in this sim-to-real pipeline is differentiable w.r.t. the actuation parameter $\mathbf{a}$. Detail analysis of differentiability can be found in Appendix A-A. In Sec. IV, we will present the detail of NN-based forward kinematics and the IK computing for shape control, which is based on minimizing the distance between $\mathbf{p}^{*}$ and its closest point $\mathbf{c}^{*}$ on the target shape $\mathcal{S}^{t}$.

III Differentiable Alignment for Training

To enable correspondence-free training of the function-prediction network $\mathcal{N}_{rbf}$, we introduce a confidence-map weighted Chamfer distance as a robust metric for evaluating discrepancies between the predicted shape $\mathcal{S}_{j}^{*}$ and the captured shape $\mathcal{S}_{j}^{p}$. This weighted Chamfer distance serves as the primary loss for aligning $\mathcal{S}_{j}^{*}$ with $\mathcal{S}_{j}^{p}$ during training. The process jointly optimizes $\mathcal{N}_{rbf}$ and a newly introduced confidence-prediction network $\mathcal{N}_{conf}$, aided by additional regularization terms – namely, confidence regularization, normal compatibility, and geometric regularization. Finally, we describe how this framework can be adapted to scenarios involving partially captured markers with known correspondences.

III-A Confidence Weighted Chamfer Distance

Chamfer distance has been widely used in previous works for rigid registration (e.g., [44]). In contrast, our task involves non-rigid registration on models with large deformations, which presents additional challenges. Moreover, the captured shapes of free-form surfaces undergoing large deformations often contain missing regions. To address this, we introduce a real-valued confidence map $w_{c}(\cdot)\in[0,1]$ on the surface of each simulated shape $\mathcal{S}_{j}$, representing the likelihood of finding reliable correspondences on the scanned shape $\mathcal{S}_{j}^{p}$.

Without loss of generality, this confidence map is parameterized by a neural network $\mathcal{N}_{conf}$, whose parameters are determined jointly during the sim-to-real training process. The input to $\mathcal{N}_{conf}$ includes a surface point $\mathbf{p}\in\mathcal{S}$ and the set of control points $\mathcal{S}^{c}(\mathbf{a})$ that encode the simulated shape, allowing the network to adapt to varying shape configurations. The network outputs a scalar in the range $[0,1]$, representing the confidence that the point $\mathbf{p}$ has a reliable correspondence on the scanned shape $\mathcal{S}_{j}^{p}$ – i.e., $w_{c}(\mathbf{p})=\mathcal{N}_{conf}(\mathbf{p},\mathcal{S}^{c}(\mathbf{a}))$.

The Chamfer distance between the predicted shape $\mathcal{S}_{j}^{*}$ and the captured shape $\mathcal{S}_{j}^{p}$ can then be weighted by the confidence map $w_{c}(\cdot)$ as follows:

[TABLE]

where $\mathbf{p}^{*}\in\mathbb{R}^{3}$ is the sim-to-real transferred surface point corresponding to $\mathbf{p}=\mathbf{B}(u_{p},v_{p},\mathcal{S}^{c}(\mathbf{a}))$, as defined in Eq. (4). Here, $|\cdot|$ denotes the number of sample points on surface.

As only the scanned shape $\mathcal{S}^{p}_{j}$ may have missed regions, the confidence map is not applied to the closest point search from the physically sampled geometry $\mathcal{S}^{p}_{j}$ to the predicted shape $\mathcal{S}^{*}_{j}$ that is assumed to be complete.

III-B Loss Functions for Training

The loss function for shape alignment based training is defined by the confidence weighted Chamfer distances as

[TABLE]

Besides of this primary loss, other three loss terms are defined to serve for purpose of regularization.

III-B1 Confidence Regularization

To prevent trivial solutions where the confidence map $w_{c}(\cdot)$ is zero across the entire set of surfaces, we introduce a confidence regularization loss:

[TABLE]

III-B2 Normal Compatibility

To ensure the normals of a predicted surface $\mathcal{S}_{j}^{*}$ compatible with those of the captured surface $\mathcal{S}_{j}^{p}$, we introduce a normal compatibility loss:

[TABLE]

where $\mathbf{c}(\mathbf{x})$ denotes $\mathbf{x}$’s closest point on $\mathcal{S}_{j}^{*}$, and $\mathbf{n}(\cdot)$ represents the unit surface normal. Notably, we perform a one-way search from the captured shape $\mathcal{S}_{j}^{p}$ to the predicted shape $\mathcal{S}_{j}^{*}$, which is assumed to be complete. This asymmetry improves robustness in cases where the captured surface contains large holes or missing regions.

III-B3 Geometric Regularization

The shape produced by the RBF-based warping function may appear weird when the number of kernels is limited and the discrepancy between the simulated and captured shapes is large. In such cases, the learned high-dimensional function space may fail to generate smooth deformations without appropriate regulation, as illustrated in Fig.5(a–c). Similar challenges have been observed in previous RBF-based deformation studies (ref. [40, 38]). To address these issues and improve control over the resulting deformed 3D shapes, prior research imposed the following compatibility conditions.

[TABLE]

To impose similar constraint on the deformation function predicted by the network $\mathcal{N}_{rbf}$, we formulate the compatibility conditions of Eq.(10) as a geometric regularization loss:

[TABLE]

The effectiveness of this geometric regularization term has been demonstrated in Fig.5(c, d).

In summary, the total loss for training is defined as

[TABLE]

which combines the aforementioned loss terms with corresponding weights. The function-prediction network $\mathcal{N}_{rbf}$ and the confidence-prediction network $\mathcal{N}_{conf}$ are jointly trained by minimizing $\mathcal{L}_{total}$ (see also Fig.2 for the illustration of training pipeline). The weight coefficients $\omega_{cr}=12.0$, $\omega_{nc}=0.5$, and $\omega_{gr}=\text{1.5e-4}$ are empirically selected based on experiments and remain fixed across all examples.

III-C Training with Known Correspondences

In scenarios where hardware such as a MoCap system is used for sim-to-real learning, the captured surface $\mathcal{S}_{j}^{p}$ is represented by a set of markers ${\mathbf{x}_{i}}$ with known correspondences. This allows the training process to be simplified. Specifically, for each marker, its corresponding surface sample point $(u_{i},v_{i})$ in the parametric domain can be obtained through registration with the deformable surface in its rest shape. Consequently, the set of simulated surface points becomes $\mathcal{S}_{j}=\{\mathbf{p}_{i}=\mathbf{B}(u_{i},v_{i},\mathcal{S}^{c}_{j})\}$.

Training proceeds similarly to the correspondence-free case, with the following modifications:

• •

Replace the nearest-neighbor search in Eq.(III-A) with the known correspondence term $\|\mathbf{p}_{i}^{*}-\mathbf{x}_{i}\|_{2}^{2}$, which gives

[TABLE]

• •

Set $w_{c}(\mathbf{p}_{i})=1$ if marker $\mathbf{x}_{i}$ is captured in the $j$-th frame, or $w_{c}(\mathbf{p}_{i})=0$ when it is missing;

• •

Assign $\omega_{nc}=\omega_{cr}=0.0$, as normal compatibility and confidence regularization are no longer necessary with known correspondences.

The effectiveness of this simplified sim-to-real training process using a MoCap system will be demonstrated in Sec.V (see the right of Fig.13(b)).

IV Neural Networks Based Kinematic Computing

This section presents the NN-based computational pipeline for kinematics. With the help of a sim-to-real network trained above, accurate shape control of deformable free-form surfaces can be realized by our fast IK solution.

IV-A NN-based Forward Kinematics

While the recently developed fast simulator provides an efficient solution to predict the deformed shape of soft robots [12], applying it to estimate gradients of the IK computation by numerical differentiation remains time-consuming [11]. To enable fast IK computation, we propose to employ an NN-based computational pipeline for forward kinematics as shown in Fig.6. The input of our shape prediction network $\mathcal{N}_{fk}$ is the actuation parameter $\mathbf{a}$, and the output is the control points $\{\mathcal{S}^{c}\}$ as a compact representation of freeform surfaces. This gives

[TABLE]

The network $\mathcal{N}_{fk}$ actually serves as a surrogate model for the numerical simulation of soft robot’s deformation.

The network can be trained via a supervised learning process. Given a set of randomly generated actuation parameters as $\{\mathbf{a}_{k}\}$, we can run the numerical simulators to obtain the deformed shapes and generate their corresponding control points as $\{\mathcal{S}^{c}_{k}\}$. A training dataset with $M$ such pairs of results $\{\mathbf{a}_{k}:\mathcal{S}^{c}_{k}\}_{k=1,\ldots,M}$ can be obtained by a simulator (e.g., [12]). It is important to generate example shapes that span the entire space of shape variation. The dataset is then employed to train the network $\mathcal{N}_{fk}$.

The computation of forward kinematics is conducted with the help of $\mathcal{N}_{fk}$ and $\mathcal{N}_{s2r}$. Given an actuation $\mathbf{a}$, the control points of its simulated surface $\mathcal{S}^{c}$ can be obtained by Eq.(14). The corresponding coefficients $\bm{\gamma}$ of the RBF function for sim-to-real transfer can then be obtained by Eq.(3).

As a result, for any point $(u_{p},v_{p})$ sampled in the $u,v$-parametric domain, its position on the physical model can be predicted by Eq.(4). This NN-based computational pipeline for forward kinematics was illustrated in Fig.6 – see also the pseudo-code given in Algorithm 1. The predicted shape $\mathcal{S}^{*}(\mathbf{a})$ is represented by a set of points $\{\mathbf{p}^{*}\}$, which is a differentiable function in terms of the actuation $\mathbf{a}$.

IV-B Gradient Iteration Based Inverse Kinematics

Following the strategy proposed in [11], the IK computation is formulated as an optimization problem that minimizes the shape difference between the predicted shape $\mathcal{S}^{*}$ and the target shape $\mathcal{S}^{t}$ while allowing the pose change. That gives

[TABLE]

where $\mathbf{c}^{*}$ is the closet point of $\mathbf{p}^{*}$ on the target shape $\mathcal{S}^{t}$. The rotation matrix $\mathbf{R}$ and the translation vector $\mathbf{t}$ are applied to the target model to eliminate the influence of pose change. The objective function can be minimized using the gradient descent method with linear search. After each iteration of updating $\mathbf{a}$, we apply an ICP-based rigid registration [5] to determine a new pair of $(\mathbf{R},\mathbf{t})$. In other words, the optimization process alternates between updating the actuation parameter $\mathbf{a}$ and adjusting the pose of target model $(\mathbf{R},\mathbf{t})$222For cases where the target shape’s pose is fixed (e.g., the soft manipulator examples), the registration step is omitted by setting $(\mathbf{R}=\mathbf{I},\mathbf{t}=\mathbf{0})$.. The diagram of our fast IK solver is as illustrated in Fig.7, and the pseudo-code of our IK solver can be found in Algorithm 2.

Thanks to the formulation of NN-based forward kinematics including sim-to-real, the gradients of $D(\cdot)$ in Eq.(15) can be computed analytically. That is

[TABLE]

where

[TABLE]

$\mathbf{Q}$ denotes those terms contributed by the set of kernels for RBF-base warping while $\mathbf{B}$ is for the surface point $\mathbf{p}$. Detail formulas can be found in Appendix A-A.

V Implementation Details and Results

We have implemented the proposed approach in C++ and Python. The network training phase is implemented on the PyTouch platform with the learning rate at $0.001$ and the maximum number of epochs as $150$. For the inference phase, we transferred the trained networks to integrate with our IK solver in C++ running on CPU. The analytical gradients of the networks are employed in our IK solver. Our source code will be released on GitHub upon the acceptance of this paper at: https://yinggwan.github.io/CFS2R.github.io. All the training and computational tests are conducted on a PC with Intel i7-12700H CPU, RTX $3060$ GPU and $32$ GB RAM.

V-A Hardware for Verification

To evaluate the generality of our function-based sim-to-real pipeline, we tested it on four different soft robots as shown in Fig.8), where an open source pneumatic platform – OpenPneu [36] – was used to actuate these robots with well-controlled pressures. In addition, two types of vision systems, a Vicon motion capture system [39] with 8 cameras and an Artec Eva 3D Scanner [3], were used for data acquisition. Note that the scanner is also used for capturing ground-truth shapes for validation.

V-A1 Deformable Membrane

The first soft robot as shown in Fig.8(a) consists of nine chambers that can be inflated independently by pneumatic actuation. Each chamber is covered by a silicone membrane with thickness as 2mm. The dimension of the membrane is 32.0 cm (L) x 32.0 cm (W). By adjusting combination of air pressures across the nine chambers, the deformable membrane can be shaped into a variety of free-form surfaces [31].

V-A2 Soft Manipulator

The second soft robot for this research is a two-segment soft manipulator [24], with each segment containing three chambers. The total length of the manipulator is 26.0 cm. Rotational deformation in its global structure produces significant shape changes across the robot’s surface, as illustrated in Fig.8(b). In existing methods (e.g., [41]), the deformed states of this robot are simplified by only modeling the global structure shape, disregarding the substantial deformation of its outer curved surface. This inaccuracy will lead to the difficulty of realizing the planned collision-free motion on a physical setup, which can be effectively solved by using our sim-to-real method.

V-A3 Deformable Mannequin

The third robot, shown in Fig. 8(c), contains a soft robotic mannequin representing both the front and back sides of a human body at a 1:2 scale. The total height of the mannequin is 31.5 cm. It can be deformed by pressurized air and features a design with three layers: an inner soft chamber layer attached to the base solid core, a middle soft chamber layer attached on top of the inner layer, and a top membrane layer (passively morphed) covering the base layers to form the overall smooth shape. Detailed design can be found in [34, 35]. By pumping air at varying pressures into the chambers, the soft mannequin can be deformed into different shapes.

V-A4 Compressible Soft Manipulator

The fourth robot is an advanced soft manipulator composed of three segments, each containing three soft-bellow actuators as shown in Fig. 8(d). By adjusting the internal pressure of each actuator, the soft robot can elongate or contract through chamber inflation (positive pressure) or compression (negative pressure) while simultaneously performing deformations in bending. For example, the robot can change its length from a natural state of 92.0 cm down to 65.0 cm. The sim-to-real approach proposed in this paper can accurately model the robot’s entire external surface, which is essential for enabling collision-free motion planning.

The four hardware setups were selected to span representative deformation modes and structural complexities in pneumatically actuated soft robots. The deformable membrane provides inflation-induced deformations with regular chamber geometry, while the robotic mannequin involves inflation through freeform chamber boundaries with overlaps, resulting in more complex and spatially coupled surface deformations. The two soft manipulators highlight structural bending, with one driven purely by bending and the other combining inflation / compression and bending, thereby introducing hybrid deformation behaviors. Together, these systems cover the spectrum from simple to complex chamber inflation and from pure bending to coupled bending, inflation and compression, demonstrating that our sim-to-real learning framework generalizes across diverse actuation principles and deformation characteristics rather than being tailored to a single morphology.

V-B Details of Networks and Training

V-B1 Forward kinematics network $\mathcal{N}_{fk}$

For training $\mathcal{N}_{fk}$, a dataset with $m\geq 1000$ different pairs of actuation and shapes is collected from the simulation. This dataset contains shapes with the minimal and the maximal pressures applied to every chamber – in total $2^{k}$ shapes were generated with $k$ being the DoFs of actuation (see Table II shown parameters for different robots). We also randomly sampled other $(m-2^{k})$ actuation parameters using the Halton sequence [18]. All these $m$ samples of actuation are applied in the simulation system to obtain their simulated shapes to explore the whole deformation space. Considering the large non-linearity between actuation and shape, we non-uniformly resample the actuation space by the amount of chamber inflation to enhance the training accuracy. The average time used for completing each simulation has been reported in Table II. This dataset based on simulation is separated in the ratio of $7:3$ for training and testing.

We carefully select the network structure, which includes the number of hidden layers (h), and the number of neurons in each hidden layer (l). The experimental results of the maximal shape prediction error according to different network structures for $\mathcal{N}_{fk}$ are studied and illustrated in Fig. 9(a), and we choose two hidden layers with each layer contains 128 neurons as the final network structure to balance the quality of training result and the computational cost. In our implementation, ReLU is selected as the activation function and batch normalization is applied to improve stability in training. Note that our experiment finds that prediction with better quality can be achieved when learning the translation vectors applied to the positions of control points on an average model of all shapes.

V-B2 Function prediction network $\mathcal{N}_{rbf}$

For learning the sim-to-real transfer, we employ a network architecture with 2 or 3 hidden layers as $\mathcal{N}_{rbf}$, where each layer has a certain number of neurons using the ReLU activation function. The input layer contains the positions of $30\times 30$ control points, and the output layer is the coefficients for the RBF-based warping function as $\bm{\gamma}\in\mathbb{R}^{3(N+4)}$ with $N$ being the number of kernels. The training of $\mathcal{N}_{rbf}$ is taken on shapes captured by either 3D scanner or MoCap while randomly varying the pneumatic actuation within the working range of the chambers. Among these captured shapes, a certain number of frames have missing regions (or markers). However, all shapes are employed to train $\mathcal{N}_{rbf}$. The total number of valid samples (or markers) for different cases have been reported in Table II. Using the dataset of deformable mannequin generated from MoCap as an eaxmple, it can be observed from the study as shown in Fig. 9(b) that the prediction error starts to increase when the total number of neurons exceeds 50 – i.e., overfitting happens. Based on this analysis, we select 24 neurons for each layer in $\mathcal{N}_{rbf}$.

V-B3 Confidence map network $\mathcal{N}_{conf}$

The input of the confidence map network $\mathcal{N}_{conf}$ is a vector comprising the simulated position $\mathbf{p}\in\mathbb{R}^{3}$ of a point on the surface, along with the control points $\mathcal{S}^{c}$ that define the global shape features. This combined input is processed through three fully connected hidden layers, each with 128 neurons and ReLU activation, followed by a fully connected layer with a single neuron and sigmoid activation. The output is a score in the range of $[0,1]$.

In our implementation, we first train $\mathcal{N}_{fk}$, and then jointly train $\mathcal{N}_{rbf}$ and $\mathcal{N}_{conf}$ while keeping $\mathcal{N}_{fk}$ unchanged. The RBF-based space warping function $\mathbf{\Phi}$ and the B-spline shape decoder $\mathbf{B}$ are both implemented as differentiable layers, enabling seamless integration into the training process of $\mathcal{N}_{rbf}$ by backpropagation.

V-C Results of Sim-to-Real Learning by 3D Scanner

We now study the performance of our sim-to-real learning approach by a structure-light-based 3D scanner, obtaining 3D shapes as point clouds without correspondences. The tests have been conducted on the deformable membrane (Fig.8(a)) and the soft manipulator with two segments (Fig.8(b)).

V-C1 Sim-to-real vs. direct learning

In the first study, we apply 10 random unseen actuation parameters to the deformable membrane, where the corresponding shapes are predicted by our $\mathcal{N}_{fk}$ and $\mathcal{N}_{s2r}$ networks and compared with the scanned shapes. The results of the comparison are shown in Fig.10, where the geometric errors are visualized on surfaces by colormaps. The statistical error distributions are given as histograms. For the selected two configurations – S1 and S2, the average errors are reduced by 75.5% and 63.3% when using our sim-to-real approach, and the maximal errors are reduced by 41.8% and 58.6% accordingly.

An experiment is taken now to study if we can directly learn to predict the deformed shape from the actuation parameters using neural networks. We applied 40 random sets of actuation parameters to deform the membrane and then scanned the corresponding physically deformed shapes as the training dataset. The direct learning network consists of two hidden MLP layers with 128 neurons in each layer. The input of the network is actuation parameters (i.e., 9 values for each shape) and the output is 200 points sampled on the deformed surface. To reduce the geometry discrepancy between the predicted surfaces $\mathcal{S}_{j}^{*}$ and the scanned surfaces $\mathcal{S}_{j}^{p}$, the Hausdorff distance [19] as

[TABLE]

and the chamfer distance [29] as

[TABLE]

are jointly used to penalize the maximum and the average deviations between two point clouds by a loss as

[TABLE]

The resultant surfaces of prediction are then generated by RBF-based surface deformation using these 200 predicted sample points. The result comparison between this direct learning method and our method has been given in Fig.10. It can be observed that direct learning leads to results with significantly larger errors.

We further conducted this comparison by using more sample shapes for both the direct learning and our sim-to-real based learning – 500 shapes are captured by using 80% for training and 20% for testing. As shown in Fig.11, our method also delivers substantially better performance. When using 40 frames in our sim-to-real based training, the error can already be reduced to a very low level similar to the direct training using 400 frames.

V-C2 Effectiveness of joint learning

We now conduct a study to demonstrate the effectiveness of jointly learning the sim-to-real transfer network and the confidence map network with the help of the differentiable alignment module. The experiments are performed on the soft manipulator with two segments. Due to large deformations, the scanned 3D shapes often contain missing regions (see Fig.12). For comparison, we also generate sim-to-real training results using unchanged correspondences that are pre-determined by a state-of-the-art non-rigid registration method [43]. This actually treats the sample points as ‘virtual’ markers by using $L^{2}$-norm distances. As shown in Fig.12(a.1), using static correspondences can lead the training to become stuck in poor local minima. Furthermore, we also conduct a test by using a fixed confidence weight $w_{c}(\mathbf{p})=1.0$ but using Chamfer distance. The result is as shown in Fig.12(a.2), where the surface of transferred model shows unwanted distortion caused by the points falling in the missed regions. In contrast, our joint learning method consistently produces results with significantly smaller geometric errors. Examples of our method applied to different shapes are shown in Fig. 12(b)–(d).

V-C3 Number of samples and kernels

When performing sim-to-real learning with dense point cloud input, we usually downsample the scanned 3D point clouds to a fixed number of points to define $\mathcal{S}^{p}$. To determine an appropriate sampling density, we created datasets with varying numbers of points and trained the sim-to-real network $\mathcal{N}_{s2r}$ on each dataset. After training, these networks were evaluated on unseen cases to assess shape prediction accuracy. As shown in Fig.13(a), the approximation error decreases as the number of sample points increases, and begins to plateau around 1,000 points. Furthermore, when comparing networks trained on point clouds without correspondence, those incorporating MoCap-provided correspondences consistently achieve higher accuracy under comparable sampling densities (see Fig.13(b)). This is likely because motion capture datasets provide precise correspondences, whereas 3D scan datasets are subject to alignment noise and inaccuracies.

The number of Gaussian kernels, denoted as $N$ in Eq.(2), is also a factor influencing the accuracy and efficiency of the sim-to-real method. With too few kernels, the space warping function lacks the capacity to capture complex variations, leading to notable errors. Conversely, an excessive number of kernels increases computational demands, heavily impacting the IK computation as it must repeatedly apply the RBF-based space warping. To determine the number of kernels, we maintain 1000 sample points and train different sim-to-real networks on the same dataset with 40 shapes, varying only the number of kernels. Shape approximation errors were then evaluated on four unseen cases, with results shown in Fig.14. As a result, more kernels generally lead to higher prediction accuracy but come with the cost of longer computation times. To balance between the accuracy and the efficiency, we usually select 100 kernels for our implementation and tests.

V-C4 Ablation study – geometric regularization

An ablation study was conducted on the soft manipulator with two segments to illustrate the importance of geometric regularization loss in achieving accurate shapes and reducing the gap between simulation and reality. This setup is selected due to its complex deformation, where both the global structure shape and the local surface shape are significantly changed. To further validate the effectiveness and the generality of our sim-to-real pipeline with compatibility loss, we have tested it by using two different simulators – one is based on FEA [42] and the other is based on the geometry-based simulator (ref. [11, 12]). The results can be found in Fig.15. The normal error at any point $\mathbf{p}_{j}$ on a simulated (or corrected) surface is defined as:

[TABLE]

with $\mathbf{c}^{p}(\cdot)$ giving the closest point of $\mathbf{p}_{j}$ on the scanned point cloud $\mathcal{S}^{p}$. The distribution of the distance errors and the normal difference errors are plotted as the histograms shown in Fig.15(b). Note that the histograms of normal errors are capped at 0.06 – i.e., an angular difference of less than $20^{\circ}$ between the two normals. This allows to focus on a range of interest for better comparison.

The experimental tests shown in Fig.15 demonstrate that the geometric regularization loss, derived from the compatibility condition of the RBF warping function (i.e., Eq.(10)), is essential for achieving better shape accuracy on the corrected models. These tests also confirm the generality of our network when working on different types of numerical simulators.

V-C5 Ablation study – surrogate vs. direct simulation

To further evaluate the generalizability of our sim-to-real approach, we tested the performance of the sim-to-real network $\mathcal{N}_{s2r}$ – trained using a surrogate simulator – on input shapes generated by direct physical simulation. The comparison results are shown in Fig. 16. We observe that the sim-to-real network can still effectively reduce the shape approximation errors. However, the errors are slightly higher than those using input shapes predicted by the surrogate simulator $\mathcal{N}_{fk}$. This is likely because that the sim-to-real network is trained on the data from the surrogate simulation, which introduces a small domain gap when being applied to the physically simulated data.

V-D Results of Sim-to-Real Learning by MoCap

The performance of our approach has also been tested by the data captured on a Mocap system.

V-D1 Deformable mannequin

Experimental tests have been conducted to verify the performance of our sim-to-real method with sparsely and partially acquired marker sets – obtained on the deformable mannequin setup. First of all, we randomly change all the nine actuation parameters and use our forward kinematic network to predict free-form surface shapes. Four instances are selected to apply the actuation onto the physical setup. The resultant shapes on the soft robotic mannequin are scanned and compared with the simulated models (i.e., without sim-to-real transfer) and the corrected models (i.e., by applying the sim-to-real transfer). The shape approximate errors are evaluated by first correcting the pose of scanned model using the ICP-based registration and then compute the distances between every surface sample points to their closest points on the scanned model. As can be found from the results shown in Fig.17, the shape approximation errors on the corrected models were significantly reduced.

V-D2 Compressible manipulator

To validate the effectiveness of our method in handling deformation under compression, we tested our sim-to-real transfer approach on the advanced soft manipulator setup with a $3\times 3$ chamber configuration (Fig.8(d)). The simulation dataset was generated using MuJoCo [37] (see Fig.18(a)), while the corresponding physical dataset was collected using a MoCap system. A total of 700 sample shapes were recorded during the experiments, with 500 samples used for training and 200 for testing. As shown in Fig.18(b), our proposed sim-to-real method achieves significantly improved accuracy on the test set. Additionally, Fig. 18(c) presents 6 representative frames from the test set – capturing different states of deformations – where the simulated model, corrected model, and ground truth are visualized together to illustrate the differences.

V-E Results of Inverse Kinematics

Our sim-to-real approach can improve the IK algorithm introduced in Sec. IV-B, enabling more accurate realization of the target shape. To validate the effectiveness of this approach in IK computation, we conducted physical experiments on soft robots to demonstrate its performance.

V-E1 Shape Approximation on Deformable Membrane

The deformation of a soft membrane can be programmed into different shapes, enabling its function as a reconfigurable mold for 3D printing. The hardware setup of our 3D printing experiment is shown in Fig.19(a), where the task of 3D printing is conducted by a UR10e robot arm equipped with a PulsarTM pellet extruder. The material deposition printing process is conducted by following a toolpath computed on a surface. After accurately deforming the membrane into a target shape of this surface, the robotic hardware can print Polylactic Acid (PLA) directly on this curved surface.

This application requires accurate shape control of the mold. For example, as shown in the top row of Fig.19(b) and (d) circled by red dashed lines, large gaps are formed between the printer head and the mold in the regions with large shape approximation errors (indicated by red arrows). Apparently, this leads to inaccurate material deposition – see the top row of Fig.19(c) and (e) around the regions indicated by red arrows where misalignment between two layers can be found due to the shape error of mold. The errors can be significantly reduced after incorporating our sim-to-real method in the loop of IK computation. The results with sim-to-real are given in the bottom row of Fig.19(b-e), where precise material deposition can be achieved in 3D printing – i.e., less misalignment is observed in these turning regions highlighted by blue arrows.

V-E2 Motion Planning of Soft Manipulator

The accurate shape prediction in kinematic computation is important for the motion planning of a soft robot. We conducted the physical experiment on a soft manipulator with 6 DoFs as shown in Fig.20. Specifically, a sampling-based motion planning algorithm [22] is employed to compute a sequence of samples as actuation parameters that progressively deform the soft manipulator to touch the target ball while presenting an obstacle stick around the manipulator – see Fig.20(a) for the experimental setup. Without the sim-to-real correction, the shapes predicted by the kinematic computation are significantly different from the reality. This leads to the risk of collision on a planned collision-free trajectory (see Fig.20(c) for the planned collision-free swept volume and the scanned physical execution colliding with the obstacle stick). The problem can be well solved after incorporating our sim-to-real method into the kinematic computation – see the results in Fig. 20(d).

V-E3 Shape Approximation on Deformable Mannequin

We have tested the performance of our NN-based fast IK solver on a variety of individual shapes presented in the CAESAR dataset [6]. Fig.21 shows the progressive results when applying our IK solver to realize the target shape on the physical setup of pneumatic driven deformable mannequin. It can be observed that our method converges very fast – i.e., around 15 iterations where each iteration takes less than 0.5 second. The distribution of shape approximation errors are generated by scanning the soft mannequin that has been deformed using the resultant actuation determined by our IK solver. Results on three different models are given in Fig.22. Our fast IK solver has also been tested on other models randomly selected from the CAESAR dataset as shown in Appendix A-B.

VI Conclusion and Discussion

In this paper, we addressed the challenge of bridging the gap between simulated and physically deformed free-form surfaces. We proposed a novel sim-to-real framework that jointly learns a deformation function space and a confidence map to transfer simulated geometries to their real-world counterparts using diverse and potentially incomplete input data. Unlike prior methods, our approach does not rely on pre-established correspondences and is robust to partial observations. Integrated into a neural network-based computational pipeline, the method effectively solves the inverse kinematics problem across various pneumatically actuated deformable robots, including a deformable membrane, two soft manipulators, and a deformable mannequin. Our approach advances the state of the art in shape control of deformable free-form surfaces, paving the way for more accurate and practical deployment in soft robotic applications.

Our approach imposes an implicit assumption on the configuration-independence of $\Phi(\cdot)$, which is a potential theoretical limitation. However, we actually did not find difficulty in practice mainly based on two reasons:

• •

Firstly, the simulated and the real shapes usually share similar patterns so that they can be effectively addressed through sim-to-real transfer;

• •

Secondly, the training dataset has been well designed to cover most of the possible configurations to relieve the challenge.

Although the experimental tests conducted on four soft robots is promising, our method shares the limitation of all learning-based approaches – the results heavily rely on the quality of training datasets. Moreover, our method cannot perform well on surfaces with sharp features, which is due to the fact that RBFs are inherently smooth and continuous. Therefore, we only apply it to soft robots with freeform deformable surfaces. In our future work, we will use the soft deformable membrane and mannequin as molds to fabricate products with customized shape. Specifically, as demonstrated in the proof-of-concept experiments in collaboration with the Dutch startup NEFFA [25], our soft robotic mannequin is first deformed to the body shape of a customer and then serves as a reusable 3D mold for making a garment using biodegradable materials (see Fig.23 for the fabrication process). Reliable shape control of deformable soft robots under different temperatures needs to be investigated in our future research. Furthermore, we plan to test the soft manipulators in pick-and-place tasks.

Acknowledgement

The project is partially supported by the chair professorship fund of the University of Manchester and the research fund of UK Engineering and Physical Sciences Research Council (EPSRC) (Ref.#: EP/W024985/1). The sustainable fabrication of garments shown in Fig.23 is conducted with the help of Aniela Hoitink and Iman Hadzhivalcheva at NEFFA by using their material and fabrication process.

Appendix A Differentiation of NN-based Forward Kinematics

A-A *Differentiation of NN-based Forward

Kinematics*

The gradient computation of NN for forward kinematics (including sim-to-real transfer) is crucial for our fast IK solver since it provides first-order information for optimization.

First of all, the gradient of a sample point $\mathbf{p}^{*}$ on a corrected model $\mathcal{S}^{*}$ with respect to the actuation parameters $\mathbf{a}$ has been presented in Eq.(IV-B), and all terms in this equation will be explained below.

The term $\frac{\partial\mathcal{N}_{s2r}}{\partial\mathbf{Q}}$ can be calculated through the RBF-based space warping function described in Eq.(2). Note that $\mathbf{A}=[\bm{\alpha}_{1},\bm{\alpha}_{2},\bm{\alpha}_{3}]$ and $\bm{\gamma}=[\bm{\alpha}_{0},\bm{\alpha}_{1},...\bm{\beta}_{N}]$. All vectors are column vectors. Therefore, the gradient can be calculated as:

[TABLE]

For each term in Eq. (23), we can have

[TABLE]

where the last term of Eq. (24) can be further expanded as

[TABLE]

Similarly, the term $\frac{\partial\mathcal{N}_{s2r}}{\partial\mathbf{B}}$ can be calculated by

[TABLE]

Note that the last term of Eq. (26) is similar to Eq. (A-A), and we only need to change the sign as

[TABLE]

The gradient of a sample point $\mathbf{p}^{*}$ on the corrected model with respect to the function variable $\bm{\gamma}$ can be obtained as

[TABLE]

Each component of $\frac{\partial\mathbf{p}^{*}}{\partial\bm{\gamma}}$ as shown in Eq.(29) can be calculated by following equations:

[TABLE]

[TABLE]

In Eq. (30), $(\mathbf{p})_{x}$, $(\mathbf{p})_{y}$ and $(\mathbf{p})_{z}$ denote the x, y, and z components (scalar value) of the sample point $\mathbf{p}$ on the simulation surface.

The terms $\frac{\partial\mathbf{Q}}{\partial\mathcal{N}_{fk}}$ and $\frac{\partial\mathbf{B}}{\partial\mathcal{N}_{fk}}$ are the gradients of the point(s) on the simulation surface with respect to control points $\mathcal{S}^{c}$ (stored as a flattened column vector). According to Eq.(1), the gradient can then be calculated through:

[TABLE]

The gradient of B-spline control points to B-spline control points offsets are identity matrix $\mathbf{I}$.

The terms $\frac{\partial\bm{\gamma}}{\partial\mathcal{N}_{fk}}$ and $\frac{\partial\mathcal{N}_{fk}}{\partial\mathbf{a}}$ can be easily acquired through the back-propagation of $\mathcal{N}_{rbf}$ and $\mathcal{N}_{fk}$.

A-B Additional IK Tests with More Target Shapes

We further test the performance of our fast IK solver by six additional target models from the CAESAR dataset with large variations in body dimensions, labeled S1 to S6. We verify the effectiveness of our IK solver by applying the determined actuation parameters to physically deform the mannequin. The resultant free-form shape is then scanned and compared with the target shape. The comparisons have been shown in Fig. 24.

Furthermore, the statistics of computational time have been given in Table III to demonstrate the efficiency of our algorithm. It can be found that the most time-consuming step is the ICP-based pose estimation. Each iteration of the IK computation can be completed in 300-400 ms.