DA$^{\textbf{2}}$-Net : Diverse & Adaptive Attention Convolutional Neural Network

TL;DR

DA$^{2}$-Net introduces a biologically inspired attention mechanism that explicitly captures diverse features and adaptively emphasizes the most informative ones, significantly improving CNN performance with minimal additional computational cost.

Contribution

It proposes a novel attention module for CNNs that enhances feature diversity and adaptivity, easily integrable with existing architectures.

Findings

Significant accuracy improvements on CIFAR100, SVHN, and ImageNet.

Minimal increase in computational overhead.

Effective across various CNN architectures.

Abstract

Standard Convolutional Neural Network (CNN) designs rarely focus on the importance of explicitly capturing diverse features to enhance the network's performance. Instead, most existing methods follow an indirect approach of increasing or tuning the networks' depth and width, which in many cases significantly increases the computational cost. Inspired by a biological visual system, we propose a Diverse and Adaptive Attention Convolutional Network (DA$^{2}$-Net), which enables any feed-forward CNNs to explicitly capture diverse features and adaptively select and emphasize the most informative features to efficiently boost the network's performance. DA$^{2}$-Net incurs negligible computational overhead and it is designed to be easily integrated with any CNN architecture. We extensively evaluated DA$^{2}$-Net on benchmark datasets, including CIFAR100, SVHN, and ImageNet, with various CNN architectures. The experimental results show DA$^{2}$-Net provides a significant performance improvement with very minimal computational overhead.