Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching

TL;DR

This paper introduces a self-supervised multimodal learning method that improves 3D shape matching across meshes and point clouds, achieving state-of-the-art results and strong cross-dataset generalization.

Contribution

It combines mesh-based functional map regularization with contrastive loss to unify shape matching across different data modalities without supervision.

Findings

Achieves state-of-the-art results on benchmark datasets.

Handles intramodal and cross-modal shape correspondences.

Demonstrates strong cross-dataset generalization.

Abstract

The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds. While point clouds are a common representation of raw real-world 3D data (e.g. from laser scanners), meshes encode rich and expressive topological information, but their creation typically requires some form of (often manual) curation. In turn, methods that purely rely on point clouds are unable to meet the matching quality of mesh-based methods that utilise the additional topological structure. In this work we close this gap by introducing a self-supervised multimodal learning strategy that combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data. Our shape matching approach allows to obtain intramodal correspondences for triangle meshes, complete point clouds, and partially observed point clouds, as well as correspondences across these data modalities. We demonstrate that our method achieves state-of-the-art results on several challenging benchmark datasets even in comparison to recent supervised methods, and that our method reaches previously unseen cross-dataset generalisation ability.