Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

TL;DR

This paper explores how superposition and interference affect low-parameter vision models, proposing design improvements to reduce interference, thereby enhancing efficiency and accuracy on large-scale datasets like ImageNet.

Contribution

It introduces the NoDepth Bottleneck architecture, a novel design that reduces interference in low-parameter networks, improving scalability and performance.

Findings

Limiting interference enhances low-parameter network accuracy.

Key bottleneck design elements reduce neuron superposition.

NoDepth Bottleneck achieves robust ImageNet performance.

Abstract

The paper investigates the performance of state-of-the-art low-parameter deep neural networks for computer vision, focusing on bottleneck architectures and their behavior using superlinear activation functions. We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics. Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters). We identify key design elements that reduce interference by examining various bottleneck architectures, leading to a more efficient neural network. Consequently, we propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset. These findings contribute to more efficient and scalable neural networks for the low-parameter range and advance the understanding of bottlenecks in computer vision. https://caiac.pubpub.org/pub/3dh6rsel