Dynamic Perceiver for Efficient Visual Recognition

TL;DR

The paper introduces Dynamic Perceiver, a dual-branch architecture that enhances early exiting in deep networks by decoupling feature extraction from classification, leading to improved inference efficiency across various tasks and backbones.

Contribution

It proposes a novel dual-branch architecture with bi-directional cross-attention for early exiting, overcoming limitations of linear classifiers at intermediate layers.

Findings

Significant improvement in inference efficiency across multiple tasks.

Outperforms existing methods on CPU and GPU platforms.

Versatile framework adaptable to various architectures.

Abstract

Early exiting has become a promising approach to improving the inference efficiency of deep networks. By structuring models with multiple classifiers (exits), predictions for ``easy'' samples can be generated at earlier exits, negating the need for executing deeper layers. Current multi-exit networks typically implement linear classifiers at intermediate layers, compelling low-level features to encapsulate high-level semantics. This sub-optimal design invariably undermines the performance of later exits. In this paper, we propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task with a novel dual-branch architecture. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Bi-directional cross-attention layers are established to progressively fuse the information of both branches. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features. Dyn-Perceiver constitutes a versatile and adaptable framework that can be built upon various architectures. Experiments on image classification, action recognition, and object detection demonstrate that our method significantly improves the inference efficiency of different backbones, outperforming numerous competitive approaches across a broad range of computational budgets. Evaluation on both CPU and GPU platforms substantiate the superior practical efficiency of Dyn-Perceiver. Code is available at https://www.github.com/LeapLabTHU/Dynamic_Perceiver.