Revealing interpretable object representations from human behavior

TL;DR

This paper develops interpretable, sparse object representations from human behavioral data, revealing underlying conceptual dimensions like taxonomy and functionality, and demonstrating their predictive power across various human judgments.

Contribution

It introduces a method to derive low-dimensional, interpretable object representations from behavioral judgments, capturing key conceptual features and generalizing across tasks.

Findings

Representations predict most of the explainable variance in judgments.

Dimensions are highly reproducible and interpretable.

Embeddings generalize to categorization, typicality, and feature ratings.

Abstract

To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the explainable variance in human behavioral judgments. Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership, functionality, and perceptual attributes. We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior, including categorization, typicality judgments, and feature ratings, suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task.