Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment

TL;DR

This paper investigates how Vision Transformers' size, training data, and augmentation techniques affect their perceptual similarity to humans, revealing a trade-off between model complexity and human-like perception.

Contribution

It systematically evaluates factors influencing ViT perceptual alignment with humans, highlighting the impact of model size, data diversity, and training strategies.

Findings

Larger models show lower perceptual alignment.

Increased dataset diversity has minimal effect.

More data augmentation and regularization decrease alignment.

Abstract

Vision Transformers (ViTs) achieve remarkable performance in image recognition tasks, yet their alignment with human perception remains largely unexplored. This study systematically analyzes how model size, dataset size, data augmentation and regularization impact ViT perceptual alignment with human judgments on the TID2013 dataset. Our findings confirm that larger models exhibit lower perceptual alignment, consistent with previous works. Increasing dataset diversity has a minimal impact, but exposing models to the same images more times reduces alignment. Stronger data augmentation and regularization further decrease alignment, especially in models exposed to repeated training cycles. These results highlight a trade-off between model complexity, training strategies, and alignment with human perception, raising important considerations for applications requiring human-like visual understanding.