One Patch is All You Need: Joint Surface Material Reconstruction and Classification from Minimal Visual Cues

TL;DR

This paper introduces SMARC, a model that reconstructs and classifies surface materials from only a small, 10% image patch, outperforming existing methods in minimal visual input scenarios.

Contribution

SMARC is a novel unified model combining partial convolutional inpainting and classification, enabling effective surface understanding from extremely sparse visual cues.

Findings

SMARC achieves a PSNR of 17.55 dB on the Touch and Go dataset.

SMARC attains 85.10% accuracy in material classification.

Partial convolution enhances spatial reasoning with missing data.

Abstract

Understanding material surfaces from sparse visual cues is critical for applications in robotics, simulation, and material perception. However, most existing methods rely on dense or full-scene observations, limiting their effectiveness in constrained or partial view environment. To address this challenge, we introduce SMARC, a unified model for Surface MAterial Reconstruction and Classification from minimal visual input. By giving only a single 10% contiguous patch of the image, SMARC recognizes and reconstructs the full RGB surface while simultaneously classifying the material category. Our architecture combines a Partial Convolutional U-Net with a classification head, enabling both spatial inpainting and semantic understanding under extreme observation sparsity. We compared SMARC against five models including convolutional autoencoders [17], Vision Transformer (ViT) [13], Masked Autoencoder (MAE) [5], Swin Transformer [9], and DETR [2] using Touch and Go dataset [16] of real-world surface textures. SMARC achieves state-of-the-art results with a PSNR of 17.55 dB and a material classification accuracy of 85.10%. Our findings highlight the advantages of partial convolution in spatial reasoning under missing data and establish a strong foundation for minimal-vision surface understanding.