LLM-Guided Material Inference for 3D Point Clouds

TL;DR

This paper presents a novel two-stage LLM-based approach to infer material properties of 3D objects from point clouds, effectively bridging geometric and material understanding without task-specific training.

Contribution

It introduces a zero-shot, two-stage LLM method for material inference in 3D point clouds, leveraging language models as priors for semantic and material reasoning.

Findings

Achieves high semantic accuracy on 1,000 shapes

Demonstrates plausible material predictions without training

Validates approach using LLM-as-a-Judge in DeepEval

Abstract

Most existing 3D shape datasets and models focus solely on geometry, overlooking the material properties that determine how objects appear. We introduce a two-stage large language model (LLM) based method for inferring material composition directly from 3D point clouds with coarse segmentations. Our key insight is to decouple reasoning about what an object is from what it is made of. In the first stage, an LLM predicts the object's semantic; in the second stage, it assigns plausible materials to each geometric segment, conditioned on the inferred semantics. Both stages operate in a zero-shot manner, without task-specific training. Because existing datasets lack reliable material annotations, we evaluate our method using an LLM-as-a-Judge implemented in DeepEval. Across 1,000 shapes from Fusion/ABS and ShapeNet, our method achieves high semantic and material plausibility. These results demonstrate that language models can serve as general-purpose priors for bridging geometric reasoning and material understanding in 3D data.