The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers

TL;DR

This paper reveals that the 'Inductive Bottleneck' in Vision Transformers is a data-driven phenomenon where the network adapts its representational complexity based on task semantic requirements, rather than an architectural feature.

Contribution

It demonstrates that the emergence of the bottleneck is dependent on data complexity and semantic abstraction, challenging the notion that it is solely due to architecture.

Findings

Bottleneck depth correlates with dataset semantic complexity.

Texture-heavy datasets maintain high-rank representations.

Object-centric datasets induce a semantic-focused bottleneck.

Abstract

Vision Transformers (ViTs) lack the hierarchical inductive biases inherent to Convolutional Neural Networks (CNNs), theoretically allowing them to maintain high-dimensional representations throughout all layers. However, recent observations suggest ViTs often spontaneously manifest a "U-shaped" entropy profile-compressing information in middle layers before expanding it for the final classification. In this work, we demonstrate that this "Inductive Bottleneck" is not an architectural artifact, but a data-dependent adaptation. By analyzing the layer-wise Effective Encoding Dimension (EED) of DINO-trained ViTs across datasets of varying compositional complexity (UC Merced, Tiny ImageNet, and CIFAR-100), we show that the depth of the bottleneck correlates strongly with the semantic abstraction required by the task. We find that while texture-heavy datasets preserve high-rank representations throughout, object-centric datasets drive the network to dampen high-frequency information in middle layers, effectively "learning" a bottleneck to isolate semantic features.