Human-Aligned Evaluation of a Pixel-wise DNN Color Constancy Model

TL;DR

This study compares a deep neural network's color constancy performance with human behavior in virtual reality, revealing strong similarities and condition-dependent performance declines, advancing understanding of computational and human color perception mechanisms.

Contribution

It introduces a method to evaluate a DNN's color constancy against human performance using the same task and conditions, highlighting model-human correspondence.

Findings

Both model and humans achieved high constancy in baseline conditions.

Performance declined similarly when local cues were removed.

Model behavior closely matched human responses across conditions.

Abstract

We previously investigated color constancy in photorealistic virtual reality (VR) and developed a Deep Neural Network (DNN) that predicts reflectance from rendered images. Here, we combine both approaches to compare and study a model and human performance with respect to established color constancy mechanisms: local surround, maximum flux and spatial mean. Rather than evaluating the model against physical ground truth, model performance was assessed using the same achromatic object selection task employed in the human experiments. The model, a ResNet based U-Net from our previous work, was pre-trained on rendered images to predict surface reflectance. We then applied transfer learning, fine-tuning only the network's decoder on images from the baseline VR condition. To parallel the human experiment, the model's output was used to perform the same achromatic object selection task across all conditions. Results show a strong correspondence between the model and human behavior. Both achieved high constancy under baseline conditions and showed similar, condition-dependent performance declines when the local surround or spatial mean color cues were removed.