Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis

TL;DR

This paper introduces Neural Textured Deformable Meshes that leverage analysis-by-synthesis with neural features for robust 3D object analysis, outperforming traditional neural networks especially in out-of-distribution scenarios.

Contribution

It proposes a novel deformable mesh model with neural textures optimized via differentiable rendering for robust 3D analysis-by-synthesis.

Findings

More robust than conventional neural networks under occlusion and domain shift.

Competitive performance on standard benchmarks.

Effective in real-world and out-of-distribution scenarios.

Abstract

Human vision demonstrates higher robustness than current AI algorithms under out-of-distribution scenarios. It has been conjectured such robustness benefits from performing analysis-by-synthesis. Our paper formulates triple vision tasks in a consistent manner using approximate analysis-by-synthesis by render-and-compare algorithms on neural features. In this work, we introduce Neural Textured Deformable Meshes, which involve the object model with deformable geometry that allows optimization on both camera parameters and object geometries. The deformable mesh is parameterized as a neural field, and covered by whole-surface neural texture maps, which are trained to have spatial discriminability. During inference, we extract the feature map of the test image and subsequently optimize the 3D pose and shape parameters of our model using differentiable rendering to best reconstruct the target feature map. We show that our analysis-by-synthesis is much more robust than conventional neural networks when evaluated on real-world images and even in challenging out-of-distribution scenarios, such as occlusion and domain shift. Our algorithms are competitive with standard algorithms when tested on conventional performance measures.