Differentiable Weightless Neural Networks

TL;DR

The paper introduces Differentiable Weightless Neural Networks (DWN), a novel model using lookup tables and a new differentiation technique, demonstrating superior performance in edge computing hardware and tabular data tasks.

Contribution

It presents DWNs with a new training method and regularization techniques, advancing neural network efficiency and accuracy for edge devices and tabular data processing.

Findings

Superior latency, throughput, and energy efficiency on FPGA hardware.

Better accuracy than XGBoost on low-power microcontrollers.

Outperforms small models in accuracy and hardware area on ultra-low-cost chips.

Abstract

We introduce the Differentiable Weightless Neural Network (DWN), a model based on interconnected lookup tables. Training of DWNs is enabled by a novel Extended Finite Difference technique for approximate differentiation of binary values. We propose Learnable Mapping, Learnable Reduction, and Spectral Regularization to further improve the accuracy and efficiency of these models. We evaluate DWNs in three edge computing contexts: (1) an FPGA-based hardware accelerator, where they demonstrate superior latency, throughput, energy efficiency, and model area compared to state-of-the-art solutions, (2) a low-power microcontroller, where they achieve preferable accuracy to XGBoost while subject to stringent memory constraints, and (3) ultra-low-cost chips, where they consistently outperform small models in both accuracy and projected hardware area. DWNs also compare favorably against leading approaches for tabular datasets, with higher average rank. Overall, our work positions DWNs as a pioneering solution for edge-compatible high-throughput neural networks.