Do computer vision foundation models learn the low-level characteristics of the human visual system?

TL;DR

This study evaluates whether computer vision foundation models mimic low-level human visual system characteristics, finding that some models like DINOv2 show notable similarities, especially in contrast masking, but overall differences remain.

Contribution

The paper introduces a protocol to compare foundation models' low-level visual characteristics with human vision, revealing partial similarities and differences among models.

Findings

DINOv2 shows the closest resemblance to human contrast masking.

Foundation models exhibit less sensitivity to low contrast.

Responses to contrast across frequencies are irregular in models.

Abstract

Computer vision foundation models, such as DINO or OpenCLIP, are trained in a self-supervised manner on large image datasets. Analogously, substantial evidence suggests that the human visual system (HVS) is influenced by the statistical distribution of colors and patterns in the natural world, characteristics also present in the training data of foundation models. The question we address in this paper is whether foundation models trained on natural images mimic some of the low-level characteristics of the human visual system, such as contrast detection, contrast masking, and contrast constancy. Specifically, we designed a protocol comprising nine test types to evaluate the image encoders of 45 foundation and generative models. Our results indicate that some foundation models (e.g., DINO, DINOv2, and OpenCLIP), share some of the characteristics of human vision, but other models show little resemblance. Foundation models tend to show smaller sensitivity to low contrast and rather irregular responses to contrast across frequencies. The foundation models show the best agreement with human data in terms of contrast masking. Our findings suggest that human vision and computer vision may take both similar and different paths when learning to interpret images of the real world. Overall, while differences remain, foundation models trained on vision tasks start to align with low-level human vision, with DINOv2 showing the closest resemblance.