MOGNET: A Mux-residual quantized Network leveraging Online-Generated weights

TL;DR

MOGNET is a compact, resource-efficient neural network architecture that uses online-generated weights and low-precision quantization to achieve higher accuracy within tiny memory budgets.

Contribution

It introduces a novel Mux-residual quantized network with online-generated weights and a new weight ternarization method for resource-constrained hardware.

Findings

Achieves up to 1% higher accuracy than recent methods at similar or smaller model size.

Operates effectively within a sub-2Mb memory budget.

Utilizes online-generated weights and low-precision quantization for efficiency.

Abstract

This paper presents a compact model architecture called MOGNET, compatible with a resource-limited hardware. MOGNET uses a streamlined Convolutional factorization block based on a combination of 2 point-wise (1x1) convolutions with a group-wise convolution in-between. To further limit the overall model size and reduce the on-chip required memory, the second point-wise convolution's parameters are on-line generated by a Cellular Automaton structure. In addition, MOGNET enables the use of low-precision weights and activations, by taking advantage of a Multiplexer mechanism with a proper Bitshift rescaling for integrating residual paths without increasing the hardware-related complexity. To efficiently train this model we also introduce a novel weight ternarization method favoring the balance between quantized levels. Experimental results show that given tiny memory budget (sub-2Mb), MOGNET can achieve higher accuracy with a clear gap up to 1% at a similar or even lower model size compared to recent state-of-the-art methods.