A3D: Does Diffusion Dream about 3D Alignment?

TL;DR

This paper introduces A3D, a method for generating multiple semantically aligned 3D objects from text prompts by embedding them into a shared latent space and optimizing smooth, plausible transitions, improving applications like 3D asset design.

Contribution

A3D is the first approach to achieve aligned 3D object sets from text prompts by optimizing transitions in a shared latent space, ensuring smoothness and plausibility.

Findings

Effective alignment of 3D objects from text prompts.

Improved 3D editing and hybridization applications.

Demonstrated superiority over non-aligned generation methods.

Abstract

We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method. https://voyleg.github.io/a3d/