Connecting Concept Convexity and Human-Machine Alignment in Deep Neural Networks

TL;DR

This paper explores the relationship between convexity in neural network representations and human-machine alignment, revealing correlations and complex interactions that inform interpretability and reliability of AI systems.

Contribution

It introduces the first analysis linking convexity in neural representations with human alignment, based on behavioral data in vision transformer models.

Findings

Convex regions in neural latent spaces correlate with human-defined categories.

Fine-tuning increases convexity but has inconsistent effects on alignment.

The study provides initial insights into the relationship between neural convexity and human cognition.

Abstract

Understanding how neural networks align with human cognitive processes is a crucial step toward developing more interpretable and reliable AI systems. Motivated by theories of human cognition, this study examines the relationship between \emph{convexity} in neural network representations and \emph{human-machine alignment} based on behavioral data. We identify a correlation between these two dimensions in pretrained and fine-tuned vision transformer models. Our findings suggest that the convex regions formed in latent spaces of neural networks to some extent align with human-defined categories and reflect the similarity relations humans use in cognitive tasks. While optimizing for alignment generally enhances convexity, increasing convexity through fine-tuning yields inconsistent effects on alignment, which suggests a complex relationship between the two. This study presents a first step toward understanding the relationship between the convexity of latent representations and human-machine alignment.