Neural Network Training with Approximate Logarithmic Computations

TL;DR

This paper introduces a log-domain neural network training method using approximate operations to reduce computational complexity, suitable for edge devices, with minimal accuracy loss.

Contribution

It presents an end-to-end log-domain training scheme with fixed-point data, leveraging hardware-friendly approximations for efficient neural network training.

Findings

16-bit log-based training achieves within 1% accuracy of floating-point baselines

The method reduces multiplication operations in training and inference

Applicable to common datasets with minimal accuracy degradation

Abstract

The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduce implementation complexity. We implement the entire training procedure in the log-domain, with fixed-point data representations. This training procedure is inspired by hardware-friendly approximations of log-domain addition which are based on look-up tables and bit-shifts. We show that our 16-bit log-based training can achieve classification accuracy within approximately 1% of the equivalent floating-point baselines for a number of commonly used datasets.