DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data

TL;DR

DIPO is a new framework that uses dual images to controllably generate articulated 3D objects, leveraging a diffusion model and graph reasoning to improve accuracy and diversity.

Contribution

The paper introduces DIPO, a dual-image diffusion model with a graph reasoner, and a large-scale dataset PM-X for improved articulated object generation.

Findings

DIPO outperforms existing methods in generating articulated objects.

The PM-X dataset enhances model generalization to complex objects.

Dual-image input provides valuable motion information for better predictions.

Abstract

We present DIPO, a novel framework for the controllable generation of articulated 3D objects from a pair of images: one depicting the object in a resting state and the other in an articulated state. Compared to the single-image approach, our dual-image input imposes only a modest overhead for data collection, but at the same time provides important motion information, which is a reliable guide for predicting kinematic relationships between parts. Specifically, we propose a dual-image diffusion model that captures relationships between the image pair to generate part layouts and joint parameters. In addition, we introduce a Chain-of-Thought (CoT) based graph reasoner that explicitly infers part connectivity relationships. To further improve robustness and generalization on complex articulated objects, we develop a fully automated dataset expansion pipeline, name LEGO-Art, that enriches the diversity and complexity of PartNet-Mobility dataset. We propose PM-X, a large-scale dataset of complex articulated 3D objects, accompanied by rendered images, URDF annotations, and textual descriptions. Extensive experiments demonstrate that DIPO significantly outperforms existing baselines in both the resting state and the articulated state, while the proposed PM-X dataset further enhances generalization to diverse and structurally complex articulated objects. Our code and dataset will be released to the community upon publication.