DeFormer: Integrating Transformers with Deformable Models for 3D Shape Abstraction from a Single Image

TL;DR

DeFormer is a novel Transformer-based approach that uses deformable models to accurately abstract complex 3D shapes from a single image with fewer primitives, enhancing detail and interpretability.

Contribution

It introduces a bi-channel Transformer architecture combined with deformable primitives for efficient 3D shape abstraction from single images.

Findings

Outperforms state-of-the-art in shape reconstruction accuracy

Uses fewer primitives for broader geometric coverage

Provides consistent semantic correspondences for interpretability

Abstract

Accurate 3D shape abstraction from a single 2D image is a long-standing problem in computer vision and graphics. By leveraging a set of primitives to represent the target shape, recent methods have achieved promising results. However, these methods either use a relatively large number of primitives or lack geometric flexibility due to the limited expressibility of the primitives. In this paper, we propose a novel bi-channel Transformer architecture, integrated with parameterized deformable models, termed DeFormer, to simultaneously estimate the global and local deformations of primitives. In this way, DeFormer can abstract complex object shapes while using a small number of primitives which offer a broader geometry coverage and finer details. Then, we introduce a force-driven dynamic fitting and a cycle-consistent re-projection loss to optimize the primitive parameters. Extensive experiments on ShapeNet across various settings show that DeFormer achieves better reconstruction accuracy over the state-of-the-art, and visualizes with consistent semantic correspondences for improved interpretability.