ScanMove: Motion Prediction and Transfer for Unregistered Body Meshes

TL;DR

This paper introduces a rig-free, data-driven framework for predicting and transferring motion on unregistered 3D body meshes, overcoming challenges posed by noise and lack of correspondences.

Contribution

It presents a novel combination of a motion embedding network and a learned per-vertex feature field for deformation prediction on raw, unregistered meshes.

Findings

Effective on challenging unregistered meshes

Outperforms existing methods in benchmarks

Versatile across different motion types

Abstract

Unregistered surface meshes, especially raw 3D scans, present significant challenges for automatic computation of plausible deformations due to the lack of established point-wise correspondences and the presence of noise in the data. In this paper, we propose a new, rig-free, data-driven framework for motion prediction and transfer on such body meshes. Our method couples a robust motion embedding network with a learned per-vertex feature field to generate a spatio-temporal deformation field, which drives the mesh deformation. Extensive evaluations, including quantitative benchmarks and qualitative visuals on tasks such as walking and running, demonstrate the effectiveness and versatility of our approach on challenging unregistered meshes.

Tables

Table 1. Table 1: Train / test splits for each dataset.

	DFAUST	WALK
Type	In place	Long-range
# vertices	6890	6890
# faces	13776	13776
# meshes (training)	325	615
# sequences (training)	35	15
# meshes (test)	61	246
# sequences (test)	7	6

Table 2. Table 2: Metrics for shape interpolation on DFAUST. We report the mean squared error between predicted vertex positions and ground truth positions along with mean cosine similarity between predicted normals and ground truth normals.

	MSE $\downarrow$	Cosim $(\times {10}^{-1})\downarrow$
NJF [3]	9.50	1.30
Meshconv [64]	6.62	1.42
ARAPReg [25]	5.49	1.56
SMS [13]	4.42	1.13
Ours	4.23	1.20

Table 3. Table 3: Metrics for motion generation on WALK. We report the mean squared error between predicted vertex positions and ground truth positions along with mean cosine similarity between predicted normals and ground truth normals.

	MSE $\downarrow$	Cosim $(\times {10}^{-2})\downarrow$
3DCODED [20]	12.45	9.03
NJF [3]	6.26	4.20
Ours	4.24	5.10

Table 4. Table 4: Ablation studies evaluated on the WALK dataset. We report MSE and cosine similarity when replacing the model of the modules (1st column) with other candidates (2nd column).

		MSE $\downarrow$	Cosim $\downarrow$
Motion embed.	/wo GRU	6.35	7.12
Loss	${ℒ}^{MSE}$	4.89	5.98
	${ℒ}^{MSE}$ + ${ℒ}^N$	4.37	5.14
	${ℒ}^{CH}$	8.35	8.14
Ours	(GRU + $ℒ$)	4.24	5.10

Table 5. Table 5: Inference time (in seconds) to generate a whole sequence of 200 frames. We report the mean over 3 inferences. For [ 49 ] , we only reported the time for deformation transfer but the method also require a dense correspondence estimation step.

# $V_{source}$	low (2000)	mid (15000)	high (50000)
[49]	32	243	768
SMS [13]	16	28	99
Ours	7	18	64

Table 6. Table 6: Metrics deviation from the training mesh topology for different types of remeshing of the test set. We report the relative difference (in %)

	MSE deviation	Cosim deviation
Original	0%	0%
DS(2)	3.9%	1.2%
US(2)	4.1%	3.1%
VD	12.1%	8.3%

Table 7. Table 7: Tests reports on the WALK dataset. We report MSE and cosine similarity when replacing the model of the modules (1st column) with other candidates (2nd column).

		MSE $\downarrow$	Cosim $(\times {10}^{-2})\downarrow$
Motion emb.	SA [54]	9.59	10.35
Feat. ext.	MLP	16.36	14.9
Decoder	Prism [6]	7.21	8.54
	NJF [3]	5.07	6.06
Ours		4.24	5.10

Equations

$$DN(S)=(f_i{)}_{1\le i\le N_S}\in {\mathbb{R}}^{N_S\times F}$$

$${\tilde{z}}_t=GRU(PointNet(M_t))0\le t\le T$$

$${\tilde{f}}_{i,t}=(f_i,z_t,{\hat{v}}_{i,t-1})\in {\mathbb{R}}^{N_S\times (f+d+3)}$$

$$v_{i,t}={\Delta }_{i,t}+v_{i,t-1}=D({\tilde{f}}_{i,t})+v_{i,t-1}$$

$$\mathcal{L}={\mathcal{L}}^{MSE}+{\lambda }_N{\mathcal{L}}^N+{\lambda }_I{\mathcal{L}}^{AIAP}$$

$${\mathcal{L}}^{MSE}({\hat{M}}_t,M_t)=\frac{1}{T.N_S}\sum _{t=0}^T\sum _{i=1}^{N_S}\Vert v_{i,t}-{\hat{v}}_{i,t}{\Vert }_2^2$$

$${\mathcal{L}}^N=\frac{1}{T.N_S}\sum _{t=0}^T\sum _{i=1}^{N_S}\left(1-{\vec{n}}_i^{M_t}.{\vec{n}}_i^{{\hat{M}}_t}\right)$$

$${\mathcal{L}}^I=\frac{1}{T|\mathcal{E}|}\sum _{t=0}^T\sum _{(i,j)\in \mathcal{E}}{\left(\frac{\Vert {\hat{v}}_{i,t}-{\hat{v}}_{j,t}\Vert -\Vert v_{i,0}-v_{j,0}\Vert }{\Vert v_{i,0}-v_{j,0}\Vert }\right)}^2$$

$$\begin{gathered} d_{CD}({\hat{M}}_t,M_t)=\frac{1}{N_S}\sum _{i=1}^{N_S}\underset{j}{\min }\Vert {\hat{v}}_{i,t}-v_{j,t}{\Vert }_2^2+ \\ \frac{1}{N_t}\sum _{j=1}^{N_t}\underset{i}{\min }\Vert v_{j,t}-{\hat{v}}_{i,t}{\Vert }_2^2 \end{gathered}$$

$${\mathcal{L}}^{CD}=\frac{1}{T}\sum _{t=0}^T{d}_{CD}({\hat{M}}_t,M_t)$$

$${\Delta }_{rel}=\frac{|Met(Remesh)-Met(Orig)|}{Met(Orig)}$$

Full text

1 Introduction

In 3D computer vision, deformation modeling has seen tremendous improvements regarding human body movement generation and behavior prediction. Several models and frameworks are now available to tackle complex human body motions such as walking, running, or even dancing. However, most of these methods are built upon a parametric model such as the widely appreciated SMPL body model [36, 39]. The latter offers a mapping from a 3D rigid skeleton, the rig, characterized by a specific set of joints, to a triangular, fixed topology surface mesh: this mapping from the rig to the surface mesh is called a skinning map. The limitation comes when unregistered data such as scans or challenging synthetic surface meshes are presented (see some examples on the left in Fig. 1). Recent advances in 3D scanning technologies have enabled the capture of highly detailed and complex body meshes in uncontrolled environments. However, these ”in-the-wild” scans often present noise, irregular sampling, and lack of registration, posing significant challenges for traditional rig-based animation techniques. Our work aims to address this issue by introducing a robust, rig-free framework capable of predicting realistic and temporally coherent deformations directly from raw mesh data. This capability not only enhances the flexibility of motion synthesis but also opens new avenues for applications in augmented reality, virtual character animation, and robotics. A straightforward approach can be to register this new data by optimizing the pose and identity parameters. However, without careful initialization and proper data acquisition, this approach can be both costly and may struggle to capture fine-grained details. Moreover, parametric models are limited to a category of shapes (e.g. human bodies for SMPL) and require additional work to adapt to new categories. Our goal is to propose a generic method for any category of non-rigid surface mesh data. Additionally, the proposed framework is not limited to human motions as shown later in Fig. 6.

Most existing methods necessitate additional explicit motion priors such as joint position, especially during training. In this work, we present a recursive model that takes as input any source mesh and unregistered motion sequence and generates a complex motion as a time-dependent and vertex-wise deformation field, overcoming several previous limitations. Our method does not require any skeletal information (for training and inference) and to be versatile enough for complex motions such as walking and running and robust enough to be used in raw scans, as shown in Fig. 1. In comparison, SMPL based methods are highly efficient but require rigging, and skinning and cannot be used to deform unregistered meshes. Deformation models such as [20, 3] are robust and versatile but cannot directly capture complex time-dependencies. Recent approaches are breaking free of the need for a consistent skinning mapping [37, 35] but heavily rely on a good rigging. In this work, we suppress the need for any skeletal rig and encode the motion with a simple yet effective robust embedding network.

To summarize, the contributions presented in this paper consist in (i) a new end-to-end deep learning framework for deformation prediction to predict complex motions; (ii) we detail and explore the learned space-time feature space to predict vertex-level deformations; (iii) comprehensive comparisons and visualization of applications such as motion prediction and transfer on challenging unregistered meshes like raw scans.

2 Related work

2.1 Triangular mesh deformation

Computing surface mesh deformations is a long standing challenge in computer graphics. Most methods can be either defined on the surface itself or acting on the ambient space (see [9], chap. 9 for more). The former includes intrinsic energy minimizing optimization (shell [51, 14], elastic [24, 23], ARAP [4, 47], and others [10]), multi-scale methods [11], vector displacements [17, 22] and differential coordinates editing (gradient based [60, 61] or laplacian based [34, 46]). However, because these methods are essentially linear, they tend to fail for large deformations such as complex human motions. Also, they often involves solving complex optimization processes and post processing (local correction, smoothing, …) which can be time consuming for high resolution meshes such as scans. In contrast space deformations are fully extrinsic and include in particular lattice-based deformations [43] and cage-based deformations [53, 28, 27, 48]. However, these methods are also challenging to use for large non-linear deformations and generalized harmonics or Green functions can induce undesirable non-local changes. Recent learning-based models aims at improving the generalizability and flexibility of such methods [59] but still suffer from similar drawbacks. In contrast, we propose to leverage recent deep neural deformation model arranged in a new recursive framework designed to tackle dynamic motions and non linear deformations by learning the latter from data.

2.2 Mesh-Invariant Learning of Deformations

Existing methods for mesh deformations that are said to be ”robust” against remeshing fall into two categories: 1) methods that animate a fixed template mesh to match an unregistered target shape. 2) methods that predict per-point (or per-vertex/face) deformations directly on an unregistered mesh. The framework presented in this paper belongs to the second category, which can produce ”in-the-wild” deformations. To achieve this, recent geometric deep learning research works proposed acting on a vertex level [20, 44, 6, 13, 3] to avoid the connectivity constraint. Some recent works that bear similarities with our approach include methods that combine spectral and spatial representations to obtain mesh-invariant features [13] as well as frameworks relying on Riemannian shape analysis [23]. Building on this, Temporal Residual Jacobians (TRJ) [37] couples spatial deformation predictors with neural ordinary differential equations (ODEs) to ensure temporal coherence over long motion sequences. Previous works have addressed localized deformations in specific contexts, such as facial expression synthesis [41] and speech-driven lip motion synthesis [38]) but relatively few have successfully tackled full-body motion without any rigging [37], especially when considering long-range dynamics like walking or running.

Our proposed method builds on these recent ideas by further decoupling the mesh deformation process from any fixed skeletal structure. By directly predicting per-point deformations on unregistered meshes—and by leveraging both spatial and temporal cues, our framework achieves robust motion synthesis even in the presence of topological inconsistencies and scanning noise.

2.3 Human Motion Synthesis and prediction

Human motion synthesis can be naturally formulated as a sequential deformation prediction problem. Because the space of plausible motions is high dimensional and highly non-linear, early approaches relied on skeletal rigging and skinning methods [5, 36, 39], in which a fixed skeleton is mapped onto a template mesh with pre-defined connectivity. Especially, the motion is entirely conditioned on this predefined skeleton which allows one to generate complex motions [21, 40, 58, 18, 42] from a set of rigid joint rotations. This strong control allows one to condition the generated motion on multi-modal inputs such as text description [52, 62, 63], audio files [26] and another avatar behavior [19]. Although these rig-based approaches offer efficiency and precise control, they require all meshes to conform to a fixed topology and depend on a manually designed skeleton, limiting the flexibility in applications where scans are unregistered or exhibit variable connectivity. In parallel, unrigged, static deformation transfer has been tackled [45, 57, 56, 33, 49] but has not been extended to dynamic settings yet and may show jittering artifacts because of the lack of time consistency optimization as shown in Footnote 2.

Recent advances in geometric deep learning have begun to relax these constraints. Instead of imposing a rigid prior on the underlying motion, several methods now directly learn deformation fields on the mesh using graph convolutional neural network (GCN) [16, 64, 55]. These enable us to build efficient neural 3D morphable models [12] with applications to motion predictions [31] but the latter are limited to registered data.

3 Method

Given a triangulated source mesh $S=(\mathcal{V},\mathcal{T})$ such that $\mathcal{V}=(v_{i})_{0\leq i\leq N_{S}}$ is its (arbitrary) set of vertices and $\mathcal{T}$ is its face connectivity, often referred as the topology of the mesh. Our aim is to animate $S$ given a motion sequence $(M_{t})_{1\leq t\leq T}$ of arbitrary length $T>0$. Effectively, we want to predict the vertex trajectories $(v_{i,t})_{\begin{subarray}{c}0\leq i\leq N_{S}\\ 0\leq t\leq T\end{subarray}}$. In our setting, $S$ can have an arbitrary arrangement of its triangles and an arbitrary resolution $N_{S}$, referring here to the number of vertices. Additionally, the meshes in $(M_{t})_{1\leq t\leq T}$ are also not aligned with any fixed topology and no skeletal rig is available a priori.

3.1 Overview

First, ScanMove predicts a vertex-wise feature field on $S$, with a robust mesh feature extractor. Simultaneously, we embed the motion sequence as a vector time series using a robust motion embedding network. Then, for each time step $t$, the corresponding motion vector is concatenated to each feature on the feature field, along with the vertex positions at time $t-1$ to form a new augmented feature field. This new feature field is used as input signal to a deformation generator predicting a deformation field that is added to the source mesh $S$. An overview of the proposed model is presented in Fig. 3.

3.2 Robust mesh feature extractor.

To predict a vertex-wise feature field, we feed vertex positions $(v_{i})_{1\leq i\leq N_{S}}$ and normals $(\vec{n}_{i})_{1\leq i\leq N_{S}}$ of the source mesh $S$, to 4 DiffusionNet blocks [44]. This deep learning network relies on heat diffusion over surfaces (through a learned diffusion time) to derive a robust convolution operation to extract discretization-agnostic features based on the local geometry of the mesh. In our framework, the feature extractor is denoted as $DN$ such that

[TABLE]

where $F$ is the feature dimension.

This feature extractor is robust to a change of mesh topology so it focus on learning local geometric features. It is robust to changes in resolution, face connectivity and can also be adapted on point clouds [44] . Hence, the source mesh can have any resolution $N_{S}$ and can generalize to unregistered scans as shown experimentally in [41, 50]. The limitation comes when the mesh presents disconnected components, in which case the intrinsic heat diffusion process cannot reach the other components.

3.3 Robust motion embedding network.

In order to guide the deformation of the source mesh in a time consistent way, we encode a time-dependent descriptor for the target motion with a motion embedding network detailed in Fig. 4. For each frame $M_{t}$ of the motion sequence, we map the mesh to a latent vector $z_{t}$ of size $d$ using a PointNet [15] encoder for which the code is taken from the official implementation of [1]. Doing this for each frame yields a collection of latent vectors $(z_{t})_{1\leq t\leq T}$. Then, this collection of vectors is fed to three layers of bi-directional Gated Recurrent Units (GRU) to improve time coherence. Hence, we obtain a vector time series $(\tilde{z}_{t})_{1\leq t\leq T}$ that describe the motion.

[TABLE]

Note that this motion embedding network can be used to learn a ”dictionary” of complex movements, similar to what audio encoders do with speeches. With more motions in the training data, more movements are accessible to the deformation decoder. More details are given in Sec. 4.4.

3.4 Time-dependent deformation generator.

At a given (discrete) time step $t$, we duplicate and concatenate the corresponding motion vector $\tilde{z}_{t}$ along with the predicted vertex position of the previous time step $\hat{v}_{i,t-1}$ to each features $f_{i}$ to obtain a new feature field $\tilde{f}_{i,t}$ such that

[TABLE]

Effectively, the space-time feature tensor $(f_{i,t})_{\begin{subarray}{c}0\leq i\leq N_{S}\\ 0\leq t\leq T\end{subarray}}$ is a 3D tensor. This augmented feature field now has the information of how to deform each point of the source mesh at time $t$ and so, for each time.

Finally, at time $t>0$, the deformation generator $D$, which is a shared MLP, predicts the displacement field $(\Delta_{i,t})_{1\leq i\leq N_{S}}$ from the the predicted vertex position at time $t-1$ so that

[TABLE]

To wrap up, ScanMove learns a spatio-temporal deformation field $(\Delta_{i,t})_{\begin{subarray}{c}0\leq i\leq N_{S}\\ 0\leq t\leq T\end{subarray}}$ independently of $T$ and $N_{S}$.

3.5 Training strategy.

Registered setting. In this setting, the framework is trained end-to-end and the optimization can be supervised with ground truth motions. Hence, we use the Mean Squared Error (MSE) loss $\mathcal{L}^{MSE}$ coupled with a cosine similarity loss $\mathcal{L}^{N}$ between predicted normals and ground truth normals for the reconstruction loss so that $\mathcal{L}^{rec}=\mathcal{L}^{MSE}+\lambda_{N}\mathcal{L}^{N}$. Additionally, to improve the smoothness of predicted sequences, we add an As-Isometric-As-Possible (AIAP) regularization loss $\mathcal{L}^{I}$ based on [29] and used in [20] to avoid small non-isometric local deformations.

[TABLE]

Then, if we note $\hat{M}_{t}$ the predicted mesh at time $t$,

[TABLE]

where $v_{i,t}$ is the ground truth vertex position at time $t$ and $\hat{v}_{i,t}$ its predicted counterpart and

[TABLE]

where $\vec{n}_{i}^{M_{t}}$ is the ground truth normal at vertex $v_{i}$ and time $t$ and $\vec{n}_{i}^{\hat{M}_{t}}$ its predicted counterpart.

For the regularization loss $\mathcal{L}^{I}$, it can be written as

[TABLE]

where $\mathcal{E}$ is the set of edge indices.

Also, note that we observed better results when adaptively adding the regularization after several epochs. Details will be given in the next section.

Unregistered training. Interestingly, the framework can be trained in a complete unsupervised setting in which the vertex-wise correspondence of each mesh in the training set is unavailable. For this, we rely on the chamfer distance (CD) for the reconstruction loss so that $\mathcal{L}^{rec}=\mathcal{L}^{CD}$ with

[TABLE]

so that

[TABLE]

It is worth noting that, to our knowledge, this is the first formulation of complete unsupervised prediction of complex motions for unregistered meshes.

4 Experiments

4.1 Experiments setup

Datasets. For movements with large deformations in place, we rely on the DFAUST dataset [8]. The original dataset contains many redundant shapes, so we resampled it to have one elementary movement for each sequence. The final dataset comprises 10 human subjects that perform between 3 and 5 simple movements each.

For longer range motions, we use an augmented version of the walking motions of AMASS [2] (ACCAD). From the original motions, we randomly modified the identity to improve diversity. The final dataset contains 7 identities with 3 walking motions, each in a different direction. We call this dataset WALK.

More details on the datasets are given in Tab. 1. Note that ScanMove operates with relatively few training samples compared to most other methods.

Metrics. We report the vertex-wise mean square error (MSE) between predicted and ground-truth sequences.

We also report the mean deviation between the predicted and ground-truth normal maps using a mean cosine dissimilarity (Cosim). The formulation of these metrics is similar to the corresponding loss functions introduced in Sec. 3.5.

Implementation details. All models are trained for 500 epochs using the Adam [30] optimizer. The learning rate starts from $10^{-3}$ and is progressively reduced with a Step LR scheduler of step size 5 and a decay rate of 0.99. We set $\lambda_{N}=10^{-4}$ and $\lambda_{I}=10^{-5}$ with $\lambda_{I}$ added after 400 epochs. Training and inferences were performed on a computer equipped with an Intel Xeon Gold 5218R CPU with 64Go RAM and a NVIDIA Quadro RTX 6000 graphic card.

The feature extractor is composed of 4 DiffusionNet [44] blocks predicting a per-vertex feature vector of size $f=64$. The motion embedding network is a shared multi-layer perceptron with 4 layers of size 128 with LeakyReLU activations followed by a maxpool aggregation to obtain a latent code of size $d=64$ (as a lightweight PoinNet encoder [15]) and end with 3 layers of bidirectional GRU to predict a vector time series of size $T\times d$ where $T$ is the arbitrary sequence length. The recursive deformation generator is a shallow MLP, shared by all vertices, with 4 layers of size 128 with LeakyReLU activations.

4.2 Large deformations in place

In this experiment, we evaluate the deformation predictions on DFAUST. The test set comprises two unseen identities executing similar (but not exactly the same) movements as seen in the training set. We compare the performance of ScanMove with other recent deformation models. Meshconv [64] and ArapReg [25] are convolutional graph neural networks. Note that these models cannot, by design, deform unregistered meshes. Neural Jacobian fields (NJF) [3] predict the Jacobian of the deformation and reconstruct the vertex positions of the deformed mesh by solving a Poisson equation. Finally, Spectral Meets Spatial (SMS) [13] is a shape interpolator network coupling spatial and spectral regularization for geometric consistency.

As our model uses a vectorized representation of the target motion, and for fair comparison, we train the other models to reconstruct the whole sequence. ScanMove outperforms the aforementioned state-of-the-art models for the MSE metric and comes close second for the Cosim metric. To further support these results, we show a qualitative comparison in Fig. 5.

Graph convolutional models (ARAPReg and Meshconv) struggle to reconstruct regions with small triangles such as hands, even with ARAP regularization: we believe this is due to the low amount of training samples in our training dataset compared to the one in the original papers. Similarly, NJF learns deformations from a PoinNet global encoder that shows poor efficiency when training data is scarce. Finally, SMS shows satisfying performance but was observed to struggle with the original DFAUST meshing which is non-isotropic making spectral regularization harder.

4.3 Walking and running motion generation

Next, we evaluate our method for ”long-range” motions such as walking. The test set comprises two identities walking in three different directions. Few comparable rig-free and correspondence-free methods are available: we modified the existing NJF [3] and 3DCODED [20] models to predict successive deformations of the source mesh to predict a whole deformation sequence.

Once again, ScanMove showed better performance with respect to the MSE metric. The Cosim metric is better for NJF as it is a first-order model in the sense that it has an inherent smoothing coming from solving the Poisson equation which smooth out the deformation and so does the normal map of the deformed mesh. We also show qualitative comparison in Fig. 5. Overall, ScanMove performs better with motion sequences thanks to its ”time-awareness” arising from the motion embedding and its recursive design acting on learned spatiotemporal features.

4.4 Applications

Rig-free motion transfer. In a similar way to [37], the proposed framework enables rig-free motion transfer. Some examples are showcased in the supplementary material videos, and we display a series of examples in Fig. 6 and Fig. 7. Note that, as shown in the figure, ScanMove is versatile enough to be adapted to different challenging shape categories such as scans from the FAUST dataset [7] and animals from the DeformingThings4D dataset [32].

Motion embedding. In addition to the prediction of time-dependent deformations, the proposed motion embedding network can be used once trained for other tasks. By design, the predicted vector representation of motions is robust against remeshing. To support this, in Fig. 8, we compute a multi-dimensional scaling (MDS) of the embedding vectors onto a 2D space and we do this for the original DFAUST parameterization (top row) and for a downsampled version (bottom row). Each point trajectory in the graphs corresponds to a motion. We highlight that the two graphs are almost identical and that similar movements share similar patterns in the embedding space: see for example the blue curves compared to the cyan curves which respectively correspond to raising arms above the head and raising arms in front of the body. On the other hand, raising a knee (in red) is clearly separated from the other two motions mentioned.

4.5 Ablation

To support our design choice, in Tab. 4, we report the performance metrics on the WALK dataset with different design variations.

In a complete unsupervised setting (using $\mathcal{L}^{CH}$), performance are not as good as in the registered setting (see Tab. 4) but to our knowledge, this is the first method to address complete unsupervised learning of complex motion prediction.

We also tested multiple candidates for the different component of the framework, namely the feature extractor, the motion frame encoder, the time consistency module and the deformation generator. While one could expect more advanced models such as Prism [6], transformers [54] or NJF [3] to show better performance, we found these model tricky to optimize correctly or prohibitively expensive for long range motions.

4.6 Computation time

Once trained, a major advantage for ScanMove compared to optimization methods is fast inference and a linear time complexity with respect to the resolution of the input mesh, defined as the number of vertices of the source mesh $\#V_{source}$. We report comparisons with an optimization based method [49] and a state-of-the-art robust deep deformation model [13] in Tab. 5.

4.7 Remeshing robustness

Robustness to a change of discretization is offered thanks to the robustness of each component of the framework. We evaluate said robustness on different remeshing of the DFAUST test set. In particular, we report the performance drop when evaluating our model on the original mesh parameterization (Original), a factor 2 downsampling (DS(2)), a factor 2 upsampling (US(2)) and variable density (VD) for which one half of the mesh is upsampled and the other half is downsampled. We define the relative difference $\Delta_{rel}$ of the metric $Met$ between the original parameterization $Orig$ and the corresponding remeshing $Remesh$.

[TABLE]

Results are reported in 6.With less than 5% in performance drop from the original training discretization, we validate the robustness of the proposed framework against changes in the mesh topology.

4.8 Tests with other models

Here, we report our test results using other candidates for the different parts of our framework. The numbers are obtained by training and testing the framework after replacing the designated component (first column) with the proposed model (second column) on the WALK dataset.

We observed worse results with more advanced models. For the self-attention (SA), this could be due to the recursive aspect in which we do not require long range dependencies. For the decoder, the tested models [3, 6] have not been used in recursive frameworks before and they might require more stability to train properly.

5 Limitations and future work

This work opens up interesting avenues for future research. For example, while our motion embedding network based on PointNet effectively captures extrinsic changes like pose variations addressed in this paper, integrating it with a complementary fine-grained encoder could strengthen its ability to model non-isometric, intrinsic deformations. Additionally, although the recursive deformation generator delivers coherent sequences, introducing corrective feedback mechanisms would make it even more robust for very long motions. In this work, we relied on the ability of the network to self correct during the training process but developing new approaches for periodic resetting of the deformation could be an interesting idea to mitigate error accumulation over time.

6 Conclusion

In this paper, we introduced ScanMove, a novel framework for predicting complex, time-dependent deformations of unregistered body meshes without reliance on skeletal rigging or explicit alignment in training data. Our method successfully couples a robust mesh feature extractor with a motion embedding network and a recursive deformation generator. Extensive experiments demonstrate that ScanMove predicts realistic deformations in various challenging scenarios. The ability of the method to capture both short- and long-range deformations, along with its versatility in applications like motion transfer and unsupervised learning, underlines its potential for advancing motion synthesis in fields like animation, environment modeling, and robotics. Overall, ScanMove offers strong foundations for future work in complex unsupervised settings.

Acknowledgments

This work benefited from the support of the project project 4DSHAPE ANR-24-CE23-5907 of the French National Research Agency (ANR).