Approaching human 3D shape perception with neurally mappable models

TL;DR

This paper investigates how 3D neural field models, trained with multi-view learning, can better replicate human 3D shape perception and matching across viewpoints, highlighting the importance of training methods for human-like performance.

Contribution

It demonstrates that 3D neural field models trained with multi-view objectives align more closely with human shape perception, advancing understanding of computational mechanisms behind 3D shape inference.

Findings

3D-LFN models support human-like 3D shape matching

Multi-view training improves model-human alignment

Models partially generalize to new categories but still differ from humans

Abstract

Humans effortlessly infer the 3D shape of objects. What computations underlie this ability? Although various computational models have been proposed, none of them capture the human ability to match object shape across viewpoints. Here, we ask whether and how this gap might be closed. We begin with a relatively novel class of computational models, 3D neural fields, which encapsulate the basic principles of classic analysis-by-synthesis in a deep neural network (DNN). First, we find that a 3D Light Field Network (3D-LFN) supports 3D matching judgments well aligned to humans for within-category comparisons, adversarially-defined comparisons that accentuate the 3D failure cases of standard DNN models, and adversarially-defined comparisons for algorithmically generated shapes with no category structure. We then investigate the source of the 3D-LFN's ability to achieve human-aligned performance through a series of computational experiments. Exposure to multiple viewpoints of objects during training and a multi-view learning objective are the primary factors behind model-human alignment; even conventional DNN architectures come much closer to human behavior when trained with multi-view objectives. Finally, we find that while the models trained with multi-view learning objectives are able to partially generalize to new object categories, they fall short of human alignment. This work provides a foundation for understanding human shape inferences within neurally mappable computational architectures.