3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions

TL;DR

3D Highlighter is a flexible system that localizes semantic regions on 3D shapes based on text descriptions, including out-of-domain concepts, using a neural field guided by CLIP without requiring 3D datasets.

Contribution

It introduces a novel text-guided 3D shape localization method that handles out-of-domain concepts without 3D annotations, leveraging neural fields and CLIP.

Findings

Successfully localizes semantic regions on diverse 3D shapes.

Handles out-of-domain and non-obvious concepts effectively.

Operates without 3D datasets or annotations.

Abstract

We present 3D Highlighter, a technique for localizing semantic regions on a mesh using text as input. A key feature of our system is the ability to interpret "out-of-domain" localizations. Our system demonstrates the ability to reason about where to place non-obviously related concepts on an input 3D shape, such as adding clothing to a bare 3D animal model. Our method contextualizes the text description using a neural field and colors the corresponding region of the shape using a probability-weighted blend. Our neural optimization is guided by a pre-trained CLIP encoder, which bypasses the need for any 3D datasets or 3D annotations. Thus, 3D Highlighter is highly flexible, general, and capable of producing localizations on a myriad of input shapes. Our code is publicly available at https://github.com/threedle/3DHighlighter.