Attention Via Convolutional Nearest Neighbors

TL;DR

This paper introduces ConvNN, a unified framework that models both convolution and self-attention as neighbor selection and aggregation, enabling exploration of intermediate architectures and improving vision tasks.

Contribution

The paper formalizes convolution and attention as special cases of neighbor aggregation within a single framework, allowing systematic interpolation between them.

Findings

ConvNN improves classification accuracy on CIFAR-10 and CIFAR-100.

Interpolating between convolution and attention provides regularization benefits.

ConvNN outperforms standard attention in vision transformer models.

Abstract

The shift from Convolutional Neural Networks to Transformers has reshaped computer vision, yet these two architectural families are typically viewed as fundamentally distinct. We argue that convolution and self-attention, despite their apparent differences, can be unified within a single k-nearest neighbor aggregation framework. The critical insight is that both operations are special cases of neighbor selection and aggregation; convolution selects neighbors by spatial proximity, while attention selects by feature similarity, revealing they exist on a continuous spectrum. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection. Crucially, ConvNN serves as a drop-in replacement for convolutional and attention layers, enabling systematic exploration of the intermediate spectrum between these two extremes. We validate the framework's coherence on CIFAR-10 and CIFAR-100 classification tasks across two complementary architectures: (1) Hybrid branching in VGG improves accuracy on both CIFAR datasets by combining spatial-proximity and feature-similarity selection; and (2) ConvNN in ViT outperforms standard attention and other attention variants on both datasets. Extensive ablations on $k$ values and architectural variants reveal that interpolating along this spectrum provides regularization benefits by balancing local and global receptive fields. Our work provides a unifying framework that dissolves the apparent distinction between convolution and attention, with implications for designing more principled and interpretable vision architectures.