Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry

TL;DR

This paper investigates the internal concepts learned by DINOv2, revealing how different tasks utilize these concepts and proposing a geometric interpretation based on Minkowski spaces.

Contribution

It introduces the Minkowski Representation Hypothesis and provides a detailed analysis of the learned concepts' geometry and their functional roles.

Findings

Downstream tasks recruit distinct concept groups, such as negations for classification and boundary detectors for segmentation.

Representations are partly dense and organized beyond simple sparsity, resembling convex mixtures of archetypes.

Tokens form low-dimensional, locally connected sets that can be interpreted through Minkowski geometry.

Abstract

DINOv2 is routinely deployed to recognize objects, scenes, and actions; yet the nature of what it perceives remains unknown. As a working baseline, we adopt the Linear Representation Hypothesis (LRH) and operationalize it using SAEs, producing a 32,000-unit dictionary that serves as the interpretability backbone of our study, which unfolds in three parts. In the first part, we analyze how different downstream tasks recruit concepts from our learned dictionary, revealing functional specialization: classification exploits "Elsewhere" concepts that fire everywhere except on target objects, implementing learned negations; segmentation relies on boundary detectors forming coherent subspaces; depth estimation draws on three distinct monocular depth cues matching visual neuroscience principles. Following these functional results, we analyze the geometry and statistics of the concepts learned by the SAE. We found that representations are partly dense rather than strictly sparse. The dictionary evolves toward greater coherence and departs from maximally orthogonal ideals (Grassmannian frames). Within an image, tokens occupy a low dimensional, locally connected set persisting after removing position. These signs suggest representations are organized beyond linear sparsity alone. Synthesizing these observations, we propose a refined view: tokens are formed by combining convex mixtures of archetypes (e.g., a rabbit among animals, brown among colors, fluffy among textures). This structure is grounded in Gardenfors' conceptual spaces and in the model's mechanism as multi-head attention produces sums of convex mixtures, defining regions bounded by archetypes. We introduce the Minkowski Representation Hypothesis (MRH) and examine its empirical signatures and implications for interpreting vision-transformer representations.