One Size Does Not Fit All: Architecture-Aware Adaptive Batch Scheduling with DEBA

TL;DR

DEBA is an architecture-aware adaptive batch scheduling method that monitors gradient metrics to optimize training speed and accuracy across diverse neural network architectures, revealing that adaptation efficacy varies significantly with architecture.

Contribution

We introduce DEBA, a novel architecture-aware adaptive batch scheduler that uses gradient metrics to improve training efficiency, demonstrating that adaptation strategies must be tailored to specific architectures.

Findings

Lightweight architectures achieve 45-62% speedup with accuracy gains of 1-7%.

ResNet-18 gains 2.4-4.0% in accuracy and 36-43% speedup.

ViT-B16 shows minimal speedup (~6%) despite stable accuracy.

Abstract

Adaptive batch size methods aim to accelerate neural network training, but existing approaches apply identical adaptation strategies across all architectures, assuming a one-size-fits-all solution. We introduce DEBA (Dynamic Efficient Batch Adaptation), an adaptive batch scheduler that monitors gradient variance, gradient norm variation and loss variation to guide batch size adaptations. Through systematic evaluation across six architectures (ResNet-18/50, DenseNet-121, EfficientNet-B0, MobileNet-V3, ViT-B16) on CIFAR-10 and CIFAR-100, with five random seeds per configuration, we demonstrate that the architecture fundamentally determines adaptation efficacy. Our findings reveal that: (1) lightweight and medium-depth architectures (MobileNet-V3, DenseNet-121, EfficientNet-B0) achieve a 45-62% training speedup with simultaneous accuracy improvements of 1-7%; (2) shallow residual networks (ResNet-18) show consistent gains of +2.4 - 4.0% in accuracy, 36 - 43% in speedup, while deep residual networks (ResNet-50) exhibit high variance and occasional degradation; (3) already-stable architectures (ViT-B16) show minimal speedup (6%) despite maintaining accuracy, indicating that adaptation benefits vary with baseline optimization characteristics. We introduce a baseline characterization framework using gradient stability metrics (stability score, gradient norm variation) that predicts which architectures will benefit from adaptive scheduling. Our ablation studies reveal critical design choices often overlooked in prior work: sliding window statistics (vs. full history) and sufficient cooldown periods (5+ epochs) between adaptations are essential for success. This work challenges the prevailing assumption that adaptive methods generalize across architectures and provides the first systematic evidence that batch size adaptation requires an architecture-aware design.