Hyper-Learning for Gradient-Based Batch Size Adaptation

TL;DR

This paper introduces Arbiter, a hyper-learning algorithm that adaptively schedules batch sizes during neural network training by learning from gradient responses, improving flexibility and effectiveness over heuristic methods.

Contribution

Arbiter employs hyper-learning to optimize batch size scheduling without unrolled optimization or hypernetworks, enabling adaptive, task-agnostic batch size control.

Findings

Arbiter effectively schedules batch sizes as a standalone method.

It enhances fixed batch size schedules with greater flexibility.

It reduces variance during stochastic meta-optimization of learning rates.

Abstract

Scheduling the batch size to increase is an effective strategy to control gradient noise when training deep neural networks. Current approaches implement scheduling heuristics that neglect structure within the optimization procedure, limiting their flexibility to the training dynamics and capacity to discern the impact of their adaptations on generalization. We introduce Arbiter as a new hyperparameter optimization algorithm to perform batch size adaptations for learnable scheduling heuristics using gradients from a meta-objective function, which overcomes previous heuristic constraints by enforcing a novel learning process called hyper-learning. With hyper-learning, Arbiter formulates a neural network agent to generate optimal batch size samples for an inner deep network by learning an adaptive heuristic through observing concomitant responses over T inner descent steps. Arbiter avoids unrolled optimization, and does not require hypernetworks to facilitate gradients, making it reasonably cheap, simple to implement, and versatile to different tasks. We demonstrate Arbiter's effectiveness in several illustrative experiments: to act as a stand-alone batch size scheduler; to complement fixed batch size schedules with greater flexibility; and to promote variance reduction during stochastic meta-optimization of the learning rate.