Conjugate Product Graphs for Globally Optimal 2D-3D Shape Matching

TL;DR

This paper introduces a novel shape matching method using conjugate product graphs that achieves globally optimal, smooth, and realistic 2D-3D shape correspondences, even with partial data and minimal features.

Contribution

It proposes a new formalism based on conjugate product graphs that allows higher-order costs and local rigidity priors, improving shape matching quality and avoiding degenerate solutions.

Findings

Achieves globally optimal, smooth 2D-3D shape matching.

Handles partial shapes effectively.

Produces state-of-the-art results with minimal features.

Abstract

We consider the problem of finding a continuous and non-rigid matching between a 2D contour and a 3D mesh. While such problems can be solved to global optimality by finding a shortest path in the product graph between both shapes, existing solutions heavily rely on unrealistic prior assumptions to avoid degenerate solutions (e.g. knowledge to which region of the 3D shape each point of the 2D contour is matched). To address this, we propose a novel 2D-3D shape matching formalism based on the conjugate product graph between the 2D contour and the 3D shape. Doing so allows us for the first time to consider higher-order costs, i.e. defined for edge chains, as opposed to costs defined for single edges. This offers substantially more flexibility, which we utilise to incorporate a local rigidity prior. By doing so, we effectively circumvent degenerate solutions and thereby obtain smoother and more realistic matchings, even when using only a one-dimensional feature descriptor. Overall, our method finds globally optimal and continuous 2D-3D matchings, has the same asymptotic complexity as previous solutions, produces state-of-the-art results for shape matching and is even capable of matching partial shapes. Our code is publicly available (https://github.com/paul0noah/sm-2D3D).