WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstraction

TL;DR

WIR3D introduces a method to abstract 3D shapes using visually meaningful Bezier curves optimized with CLIP guidance, enabling detailed shape representation and user-controlled feature manipulation.

Contribution

The paper presents a novel two-phase optimization approach for 3D shape abstraction using CLIP and localized keypoint loss, enhancing shape fidelity and user control.

Findings

Effective shape abstraction across diverse datasets

Enables intuitive shape deformation and feature control

Maintains high fidelity to original geometry and visual features

Abstract

In this work we present WIR3D, a technique for abstracting 3D shapes through a sparse set of visually meaningful curves in 3D. We optimize the parameters of Bezier curves such that they faithfully represent both the geometry and salient visual features (e.g. texture) of the shape from arbitrary viewpoints. We leverage the intermediate activations of a pre-trained foundation model (CLIP) to guide our optimization process. We divide our optimization into two phases: one for capturing the coarse geometry of the shape, and the other for representing fine-grained features. Our second phase supervision is spatially guided by a novel localized keypoint loss. This spatial guidance enables user control over abstracted features. We ensure fidelity to the original surface through a neural SDF loss, which allows the curves to be used as intuitive deformation handles. We successfully apply our method for shape abstraction over a broad dataset of shapes with varying complexity, geometric structure, and texture, and demonstrate downstream applications for feature control and shape deformation.