COALESCE: Component Assembly by Learning to Synthesize Connections

TL;DR

COALESCE is a data-driven framework that automatically assembles 3D shape components by learning to synthesize natural and plausible connections, effectively handling mismatches and producing coherent objects.

Contribution

It introduces a novel deep learning-based method for component assembly that synthesizes joint connections, outperforming previous shape synthesis and completion approaches.

Findings

Outperforms prior deep models for 3D shape synthesis

Achieves more realistic and topologically meaningful joints

Automatically aligns and assembles diverse shape parts

Abstract

We introduce COALESCE, the first data-driven framework for component-based shape assembly which employs deep learning to synthesize part connections. To handle geometric and topological mismatches between parts, we remove the mismatched portions via erosion, and rely on a joint synthesis step, which is learned from data, to fill the gap and arrive at a natural and plausible part joint. Given a set of input parts extracted from different objects, COALESCE automatically aligns them and synthesizes plausible joints to connect the parts into a coherent 3D object represented by a mesh. The joint synthesis network, designed to focus on joint regions, reconstructs the surface between the parts by predicting an implicit shape representation that agrees with existing parts, while generating a smooth and topologically meaningful connection. We employ test-time optimization to further ensure that the synthesized joint region closely aligns with the input parts to create realistic component assemblies from diverse input parts. We demonstrate that our method significantly outperforms prior approaches including baseline deep models for 3D shape synthesis, as well as state-of-the-art methods for shape completion.