GViT: Representing Images as Gaussians for Visual Recognition

TL;DR

GViT introduces a novel image representation using learnable 2D Gaussians instead of patches, enabling efficient classification that matches traditional ViT performance on ImageNet.

Contribution

The paper proposes a Gaussian-based input representation for ViT, jointly optimized with classification and guided by gradients, offering a compact and effective alternative to patch grids.

Findings

Achieves 76.9% top-1 accuracy on ImageNet-1k with ViT-B.

Gaussian representations closely match patch-based ViT performance.

Guided optimization improves class-salient region focus.

Abstract

We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose positions, scales, orientations, colors, and opacities are optimized jointly with a ViT classifier trained on top of these representations. We reuse the classifier gradients as constructive guidance, steering the Gaussians toward class-salient regions while a differentiable renderer optimizes an image reconstruction loss. We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT, reaching a 76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.