Leveraging Pretrained Diffusion Models for Zero-Shot Part Assembly

TL;DR

This paper introduces a zero-shot 3D part assembly method using pre-trained diffusion models as discriminators, eliminating the need for labeled data and outperforming supervised methods in constructing realistic 3D shapes.

Contribution

It presents a novel zero-shot assembly approach leveraging diffusion models, transforming the process into an ICP-like iteration and introducing a pushing-away strategy for robustness.

Findings

Outperforms supervised methods in experiments

Effectively guides part manipulation without labeled data

Enhances robustness with pushing-away strategy

Abstract

3D part assembly aims to understand part relationships and predict their 6-DoF poses to construct realistic 3D shapes, addressing the growing demand for autonomous assembly, which is crucial for robots. Existing methods mainly estimate the transformation of each part by training neural networks under supervision, which requires a substantial quantity of manually labeled data. However, the high cost of data collection and the immense variability of real-world shapes and parts make traditional methods impractical for large-scale applications. In this paper, we propose first a zero-shot part assembly method that utilizes pre-trained point cloud diffusion models as discriminators in the assembly process, guiding the manipulation of parts to form realistic shapes. Specifically, we theoretically demonstrate that utilizing a diffusion model for zero-shot part assembly can be transformed into an Iterative Closest Point (ICP) process. Then, we propose a novel pushing-away strategy to address the overlap parts, thereby further enhancing the robustness of the method. To verify our work, we conduct extensive experiments and quantitative comparisons to several strong baseline methods, demonstrating the effectiveness of the proposed approach, which even surpasses the supervised learning method. The code has been released on https://github.com/Ruiyuan-Zhang/Zero-Shot-Assembly.