FastBoost: Progressive Attention with Dynamic Scaling for Efficient Deep Learning

TL;DR

FastBoost introduces a novel dynamically scaled attention mechanism that significantly improves efficiency and accuracy in deep learning models, enabling deployment on resource-constrained devices.

Contribution

The paper proposes DSPA, a new attention mechanism with adaptive fusion, phase scaling, and residual adaptation, achieving state-of-the-art results with fewer parameters.

Findings

CIFAR-10 accuracy of 95.57% with 0.85M parameters.

CIFAR-100 accuracy of 81.37% with 0.92M parameters.

2.1x parameter reduction over MobileNetV3 with improved accuracy.

Abstract

We present FastBoost, a parameter-efficient neural architecture that achieves state-of-the-art performance on CIFAR benchmarks through a novel Dynamically Scaled Progressive Attention (DSPA) mechanism. Our design establishes new efficiency frontiers with: CIFAR-10: 95.57% accuracy (0.85M parameters) and 93.80% (0.37M parameters) CIFAR-100: 81.37% accuracy (0.92M parameters) and 74.85% (0.44M parameters) The breakthrough stems from three fundamental innovations in DSPA: (1) Adaptive Fusion: Learnt channel-spatial attention blending with dynamic weights. (2) Phase Scaling: Training-stage-aware intensity modulation (from 0.5 to 1.0). (3) Residual Adaptation: Self-optimized skip connections (gamma from 0.5 to 0.72). By integrating DSPA with enhanced MBConv blocks, FastBoost achieves a 2.1 times parameter reduction over MobileNetV3 while improving accuracy by +3.2 percentage points on CIFAR-10. The architecture features dual attention pathways with real-time weight adjustment, cascaded refinement layers (increasing gradient flow by 12.7%), and a hardware-friendly design (0.28G FLOPs). This co-optimization of dynamic attention and efficient convolution operations demonstrates unprecedented parameter-accuracy trade-offs, enabling deployment in resource-constrained edge devices without accuracy degradation.