Transfer Learning using Spectral Convolutional Autoencoders on Semi-Regular Surface Meshes

TL;DR

This paper introduces a spectral convolutional autoencoder for 3D surface meshes that generalizes deformation patterns to unseen shapes, enabling transfer learning and localized deformation analysis with superior reconstruction quality.

Contribution

The novel spectral CoSMA autoencoder handles meshes with different connectivity, improves generalization to unseen shapes, and localizes deformation patterns, surpassing state-of-the-art autoencoders.

Findings

40% lower transfer learning errors on unseen shapes

Superior reconstruction quality on new datasets

Enables localization of deformation patterns

Abstract

The underlying dynamics and patterns of 3D surface meshes deforming over time can be discovered by unsupervised learning, especially autoencoders, which calculate low-dimensional embeddings of the surfaces. To study the deformation patterns of unseen shapes by transfer learning, we want to train an autoencoder that can analyze new surface meshes without training a new network. Here, most state-of-the-art autoencoders cannot handle meshes of different connectivity and therefore have limited to no generalization capacities to new meshes. Also, reconstruction errors strongly increase in comparison to the errors for the training shapes. To address this, we propose a novel spectral CoSMA (Convolutional Semi-Regular Mesh Autoencoder) network. This patch-based approach is combined with a surface-aware training. It reconstructs surfaces not presented during training and generalizes the deformation behavior of the surfaces' patches. The novel approach reconstructs unseen meshes from different datasets in superior quality compared to state-of-the-art autoencoders that have been trained on these shapes. Our transfer learning errors on unseen shapes are 40% lower than those from models learned directly on the data. Furthermore, baseline autoencoders detect deformation patterns of unseen mesh sequences only for the whole shape. In contrast, due to the employed regional patches and stable reconstruction quality, we can localize where on the surfaces these deformation patterns manifest.