Dynamic Query Selection for Fast Visual Perceiver

TL;DR

This paper proposes a method to improve the efficiency of Perceiver models in vision tasks by dynamically reducing the number of query tokens during inference, aiming to decrease complexity without significantly sacrificing accuracy.

Contribution

It introduces a dynamic query selection strategy for Perceiver models to enhance inference speed while maintaining performance, addressing a less-explored aspect of model efficiency.

Findings

Reduced inference time with minimal accuracy loss

Effective dynamic query selection method demonstrated

Maintained competitive performance on vision benchmarks

Abstract

Transformers have been matching deep convolutional networks for vision architectures in recent works. Most work is focused on getting the best results on large-scale benchmarks, and scaling laws seem to be the most successful strategy: bigger models, more data, and longer training result in higher performance. However, the reduction of network complexity and inference time remains under-explored. The Perceiver model offers a solution to this problem: by first performing a Cross-attention with a fixed number Q of latent query tokens, the complexity of the L-layers Transformer network that follows is bounded by O(LQ^2). In this work, we explore how to make Perceivers even more efficient, by reducing the number of queries Q during inference while limiting the accuracy drop.