Lightweight Channel Attention for Efficient CNNs

TL;DR

This paper empirically compares various channel attention modules in CNNs, proposing a lightweight LCA method that balances accuracy and efficiency for resource-constrained deployment.

Contribution

It introduces the Lite Channel Attention (LCA) module, which reduces parameters while maintaining competitive accuracy, and provides comprehensive benchmarks for attention modules in CNNs.

Findings

LCA achieves 94.68% accuracy on ResNet 18

LCA matches ECA in parameter efficiency

LCA maintains low inference latency

Abstract

Attention mechanisms have become integral to modern convolutional neural networks (CNNs), delivering notable performance improvements with minimal computational overhead. However, the efficiency accuracy trade off of different channel attention designs remains underexplored. This work presents an empirical study comparing Squeeze and Excitation (SE), Efficient Channel Attention (ECA), and a proposed Lite Channel Attention (LCA) module across ResNet 18 and MobileNetV2 architectures on CIFAR 10. LCA employs adaptive one dimensional convolutions with grouped operations to reduce parameter usage while preserving effective attention behavior. Experimental results show that LCA achieves competitive accuracy, reaching 94.68 percent on ResNet 18 and 93.10 percent on MobileNetV2, while matching ECA in parameter efficiency and maintaining favorable inference latency. Comprehensive benchmarks including FLOPs, parameter counts, and GPU latency measurements are provided, offering practical insights for deploying attention enhanced CNNs in resource constrained environments.