# HadaNets: Flexible Quantization Strategies for Neural Networks

## TL;DR

HadaNets propose a novel tensor quantization method pairing full precision and binary tensors via Hadamard products, enabling efficient training and significant model compression for neural networks on resource-constrained devices.

## Contribution

The paper introduces a flexible train-from-scratch quantization scheme that preserves parameter count and improves performance over existing reduced precision models.

## Key findings

- Reduces ResNet-18 size by 7.43 times without extra compression techniques.
- Achieves a 10-fold speedup in matrix multiplication using Hadamard Binary Matrix Multiply.
- Outperforms XNOR-Nets without additional train-time memory overhead.

## Abstract

On-board processing elements on UAVs are currently inadequate for training and inference of Deep Neural Networks. This is largely due to the energy consumption of memory accesses in such a network. HadaNets introduce a flexible train-from-scratch tensor quantization scheme by pairing a full precision tensor to a binary tensor in the form of a Hadamard product. Unlike wider reduced precision neural network models, we preserve the train-time parameter count, thus out-performing XNOR-Nets without a train-time memory penalty. Such training routines could see great utility in semi-supervised online learning tasks. Our method also offers advantages in model compression, as we reduce the model size of ResNet-18 by 7.43 times with respect to a full precision model without utilizing any other compression techniques. We also demonstrate a 'Hadamard Binary Matrix Multiply' kernel, which delivers a 10-fold increase in performance over full precision matrix multiplication with a similarly optimized kernel.