An extremely coarse feedback signal is sufficient for learning human-aligned visual representations

TL;DR

This study shows that neural networks trained with coarse, broad categories develop visual representations that align closely with human perception and brain data, challenging the need for fine-grained supervision.

Contribution

The paper systematically investigates how the granularity of supervisory signals affects brain-like visual representation learning, revealing coarse signals are surprisingly effective.

Findings

Networks trained on as few as 8 categories match or exceed neural alignment of fine-grained models.

Coarse training improves alignment with human perceptual similarity more than fine supervision.

Coarse feedback signals can produce human-like visual representations in neural networks.

Abstract

Artificial neural networks trained on visual tasks develop internal representations resembling those of the primate visual system, a discovery that has guided a decade of computational neuroscience. Research on building brain-aligned models has progressively embraced finer-grained supervisory signals, from object classification to contrastive self-supervised objectives that maximize distinctions among individual images, yet the role of supervisory signal granularity on brain alignment remains largely unexamined. Here we systematically investigate how the coarseness of a learning signal shapes representational alignment with human vision. We parametrically vary the level of signal granularity using a data-driven approach that partitions a set of training images into varied numbers of categories (2, 4, 8, 16, ..., 64) via PCA-based splits of pretrained embeddings. We train hundreds of neural networks across convolutional and transformer architectures on these coarse classification tasks and compare their representations to macaque electrophysiology recordings and human fMRI responses. We find that networks trained to distinguish as few as 8 broad categories learn representations that match or exceed the neural alignment of models distinguishing 1,000-classes. Even more strikingly, these coarsely trained networks align more closely with human perceptual similarity judgments than all other models evaluated, including networks trained with fine-grained supervision or self-supervision as well as leading large-scale vision models. These results demonstrate that human-like visual representations emerge from remarkably coarse feedback, reframing what learning signals vision may require and opening a path toward building AI systems that are more aligned with human perception.