Deep neural networks trained for estimating reflectance and illumination achieve lightness constancy differently than human observers

TL;DR

This paper compares how humans and deep neural networks estimate surface reflectance under varying lighting conditions, finding that the networks perform better and use different cues.

Contribution

The study reveals that deep neural networks achieve lightness constancy differently than humans, relying on shading and shadows rather than local contrast.

Findings

Convolutional neural networks outperformed humans in lightness constancy tasks.

Networks relied on shading and shadows, while humans were most affected by loss of local contrast.

Networks could learn to exploit noise artifacts in ray-traced images.

Abstract

Lightness constancy, the ability to create perceptual representations that are strongly correlated with surface reflectance despite variations in lighting and context, is a challenging computational problem. Indeed, it has proven difficult to develop image-computable models of how human vision achieves a substantial degree of lightness constancy in complex scenes. Recently, convolutional neural networks have been developed that are proficient at estimating reflectance, but little is known about how they achieve this, or whether they are good models of human vision. We examined this question by training a convolutional neural network to estimate reflectance and illumination in a computer-rendered virtual world, and evaluating both the convolutional neural network and human observers in a lightness matching task. In several conditions, we eliminated cues potentially supporting lightness constancy: local contrast, shading, shadows, and all contextual cues. We found that the network achieved a high degree of lightness constancy, outperforming human observers. However, we also found that eliminating cues affected the convolutional neural network and humans very differently. Humans were most affected when local contrast cues were made uninformative, whereas the convolutional neural network mostly relied on shading and shadows. In a follow-up experiment, we found that the convolutional neural network could learn to exploit noise artifacts typically associated with ray tracing and correlated with illuminance, with potential implications for the many studies relying on ray-traced images. We conclude that convolutional neural networks can learn an effective, global strategy of estimating lightness, which is closer to an optimal strategy for the ensemble of scenes we studied than the computation used by human vision.

Figures