Aligning Machine and Human Visual Representations across Abstraction Levels

TL;DR

This paper proposes a method to align neural network representations with human conceptual hierarchies by finetuning models with human judgment data, improving their alignment with human behavior and robustness.

Contribution

The authors introduce a novel finetuning approach using a teacher model trained on human judgments to enhance model alignment with human conceptual structures.

Findings

Human-aligned models better match human similarity judgments

Aligned models show improved out-of-distribution robustness

Enhanced models perform better on diverse machine learning tasks

Abstract

Deep neural networks have achieved success across a wide range of applications, including as models of human behavior and neural representations in vision tasks. However, neural network training and human learning differ in fundamental ways, and neural networks often fail to generalize as robustly as humans do raising questions regarding the similarity of their underlying representations. What is missing for modern learning systems to exhibit more human-aligned behavior? We highlight a key misalignment between vision models and humans: whereas human conceptual knowledge is hierarchically organized from fine- to coarse-scale distinctions, model representations do not accurately capture all these levels of abstraction. To address this misalignment, we first train a teacher model to imitate human judgments, then transfer human-aligned structure from its representations to refine the representations of pretrained state-of-the-art vision foundation models via finetuning. These human-aligned models more accurately approximate human behavior and uncertainty across a wide range of similarity tasks, including a new dataset of human judgments spanning multiple levels of semantic abstractions. They also perform better on a diverse set of machine learning tasks, increasing generalization and out-of-distribution robustness. Thus, infusing neural networks with additional human knowledge yields a best-of-both-worlds representation that is both more consistent with human cognitive judgments and more practically useful, thus paving the way toward more robust, interpretable, and human-aligned artificial intelligence systems.