Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks

TL;DR

This paper introduces SMGD, a new discrete optimization algorithm tailored for training low-bit neural networks efficiently in memory-constrained environments, with theoretical and numerical validation.

Contribution

The paper presents SMGD, a novel stochastic Markov gradient descent method specifically designed for training quantized neural networks under limited memory conditions.

Findings

Theoretical guarantees for SMGD's performance.

Numerical results demonstrating effectiveness of SMGD.

Applicable to highly memory-constrained training scenarios.

Abstract

The massive size of modern neural networks has motivated substantial recent interest in neural network quantization. We introduce Stochastic Markov Gradient Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. The SMGD algorithm is designed for settings where memory is highly constrained during training. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.