SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation

TL;DR

SRIF introduces a novel framework for semantic shape registration that leverages diffusion-based image morphing, flow estimation, and large vision models to achieve high-quality, semantically meaningful shape correspondences and interpolations.

Contribution

The paper presents a new semantic shape registration method combining diffusion models, dynamic 3D Gaussian splatting, and large vision models for richer semantic understanding.

Findings

Achieves high-quality dense correspondences on challenging shape pairs.

Provides smooth, semantically meaningful shape interpolations.

Outperforms existing methods in accuracy and semantic consistency.

Abstract

In this paper, we propose SRIF, a novel Semantic shape Registration framework based on diffusion-based Image morphing and Flow estimation. More concretely, given a pair of extrinsically aligned shapes, we first render them from multi-views, and then utilize an image interpolation framework based on diffusion models to generate sequences of intermediate images between them. The images are later fed into a dynamic 3D Gaussian splatting framework, with which we reconstruct and post-process for intermediate point clouds respecting the image morphing processing. In the end, tailored for the above, we propose a novel registration module to estimate continuous normalizing flow, which deforms source shape consistently towards the target, with intermediate point clouds as weak guidance. Our key insight is to leverage large vision models (LVMs) to associate shapes and therefore obtain much richer semantic information on the relationship between shapes than the ad-hoc feature extraction and alignment. As a consequence, SRIF achieves high-quality dense correspondences on challenging shape pairs, but also delivers smooth, semantically meaningful interpolation in between. Empirical evidence justifies the effectiveness and superiority of our method as well as specific design choices. The code is released at https://github.com/rqhuang88/SRIF.