MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Models

TL;DR

MatCLIP introduces a robust method for assigning realistic PBR materials to 3D models by extracting shape- and lighting-insensitive descriptors, enabling consistent material mapping from images or generated outputs.

Contribution

It extends Alpha-CLIP to create descriptors that bridge PBR representations with images, improving material assignment accuracy without explicit part relationships.

Findings

Achieves 76.6% top-1 accuracy, outperforming state-of-the-art methods.

Enables material assignment for large 3D shape datasets.

Provides a practical approach for consistent material mapping from images.

Abstract

Assigning realistic materials to 3D models remains a significant challenge in computer graphics. We propose MatCLIP, a novel method that extracts shape- and lighting-insensitive descriptors of Physically Based Rendering (PBR) materials to assign plausible textures to 3D objects based on images, such as the output of Latent Diffusion Models (LDMs) or photographs. Matching PBR materials to static images is challenging because the PBR representation captures the dynamic appearance of materials under varying viewing angles, shapes, and lighting conditions. By extending an Alpha-CLIP-based model on material renderings across diverse shapes and lighting, and encoding multiple viewing conditions for PBR materials, our approach generates descriptors that bridge the domains of PBR representations with photographs or renderings, including LDM outputs. This enables consistent material assignments without requiring explicit knowledge of material relationships between different parts of an object. MatCLIP achieves a top-1 classification accuracy of 76.6%, outperforming state-of-the-art methods such as PhotoShape and MatAtlas by over 15 percentage points on publicly available datasets. Our method can be used to construct material assignments for 3D shape datasets such as ShapeNet, 3DCoMPaT++, and Objaverse. All code and data will be released.