Focus Your Attention: Towards Data-Intuitive Lightweight Vision Transformers

TL;DR

This paper introduces a lightweight, energy-efficient vision transformer architecture using novel patch pooling and attention modules, achieving comparable accuracy to state-of-the-art models while significantly reducing computational costs.

Contribution

The paper proposes the Super-Pixel Based Patch Pooling and Light Latent Attention modules, enabling efficient, task-specific vision transformers with reduced complexity and improved training speed.

Findings

Significant reduction in computational complexity and energy consumption.

Achieves comparable accuracy to state-of-the-art models.

Enhanced training efficiency and convergence speed.

Abstract

The evolution of Vision Transformers has led to their widespread adaptation to different domains. Despite large-scale success, there remain significant challenges including their reliance on extensive computational and memory resources for pre-training on huge datasets as well as difficulties in task-specific transfer learning. These limitations coupled with energy inefficiencies mainly arise due to the computation-intensive self-attention mechanism. To address these issues, we propose a novel Super-Pixel Based Patch Pooling (SPPP) technique that generates context-aware, semantically rich, patch embeddings to effectively reduce the architectural complexity and improve efficiency. Additionally, we introduce the Light Latent Attention (LLA) module in our pipeline by integrating latent tokens into the attention mechanism allowing cross-attention operations to significantly reduce the time and space complexity of the attention module. By leveraging the data-intuitive patch embeddings coupled with dynamic positional encodings, our approach adaptively modulates the cross-attention process to focus on informative regions while maintaining the global semantic structure. This targeted attention improves training efficiency and accelerates convergence. Notably, the SPPP module is lightweight and can be easily integrated into existing transformer architectures. Extensive experiments demonstrate that our proposed architecture provides significant improvements in terms of computational efficiency while achieving comparable results with the state-of-the-art approaches, highlighting its potential for energy-efficient transformers suitable for edge deployment. (The code is available on our GitHub repository: https://github.com/zser092/Focused-Attention-ViT).