Adapting Deep Network Features to Capture Psychological Representations

TL;DR

This paper investigates how deep neural network features relate to human psychological representations, finding that while they explain much of human similarity judgments, they miss some qualitative distinctions, which can be addressed through adaptation.

Contribution

The authors develop a method to adapt deep network features to better align with human similarity judgments, enhancing their psychological relevance.

Findings

Deep features explain significant variance in human similarity judgments.

Certain qualitative distinctions in human representations are not captured by standard deep features.

Adapting deep features improves alignment with human psychological representations.

Abstract

Deep neural networks have become increasingly successful at solving classic perception problems such as object recognition, semantic segmentation, and scene understanding, often reaching or surpassing human-level accuracy. This success is due in part to the ability of DNNs to learn useful representations of high-dimensional inputs, a problem that humans must also solve. We examine the relationship between the representations learned by these networks and human psychological representations recovered from similarity judgments. We find that deep features learned in service of object classification account for a significant amount of the variance in human similarity judgments for a set of animal images. However, these features do not capture some qualitative distinctions that are a key part of human representations. To remedy this, we develop a method for adapting deep features to align with human similarity judgments, resulting in image representations that can potentially be used to extend the scope of psychological experiments.