Convergence for Discrete Parameter Update Schemes

TL;DR

This paper introduces a novel discrete update scheme for training deep learning models, providing convergence guarantees and demonstrating its effectiveness, which could lead to more efficient training especially for models with discrete structures.

Contribution

It proposes a new class of discrete update rules for training, with theoretical convergence guarantees and empirical validation, diverging from traditional quantisation methods.

Findings

Convergence guarantees for a broad class of discrete update schemes.

Empirical validation of a multinomial update rule.

Potential for more efficient training of discrete-structure models.

Abstract

Modern deep learning models require immense computational resources, motivating research into low-precision training. Quantised training addresses this by representing training components in low-bit integers, but typically relies on discretising real-valued updates. We introduce an alternative approach where the update rule itself is discrete, avoiding the quantisation of continuous updates by design. We establish convergence guarantees for a general class of such discrete schemes, and present a multinomial update rule as a concrete example, supported by empirical evaluation. This perspective opens new avenues for efficient training, particularly for models with inherently discrete structure.