SparseFormer: Sparse Visual Recognition via Limited Latent Tokens

TL;DR

SparseFormer introduces a sparse visual recognition model that uses limited latent tokens to efficiently process images, achieving comparable accuracy to dense models with lower computational costs and extending to video classification.

Contribution

The paper presents SparseFormer, a novel sparse neural architecture that mimics human visual sparsity by representing images with few tokens, reducing computation while maintaining performance.

Findings

Achieves ImageNet classification performance comparable to dense models.

Offers a better accuracy-throughput tradeoff.

Extensible to video classification with lower computational costs.

Abstract

Human visual recognition is a sparse process, where only a few salient visual cues are attended to rather than traversing every detail uniformly. However, most current vision networks follow a dense paradigm, processing every single visual unit (e.g,, pixel or patch) in a uniform manner. In this paper, we challenge this dense paradigm and present a new method, coined SparseFormer, to imitate human's sparse visual recognition in an end-to-end manner. SparseFormer learns to represent images using a highly limited number of tokens (down to 49) in the latent space with sparse feature sampling procedure instead of processing dense units in the original pixel space. Therefore, SparseFormer circumvents most of dense operations on the image space and has much lower computational costs. Experiments on the ImageNet classification benchmark dataset show that SparseFormer achieves performance on par with canonical or well-established models while offering better accuracy-throughput tradeoff. Moreover, the design of our network can be easily extended to the video classification with promising performance at lower computational costs. We hope that our work can provide an alternative way for visual modeling and inspire further research on sparse neural architectures. The code will be publicly available at https://github.com/showlab/sparseformer