Functional Hashing for Compressing Neural Networks

TL;DR

FunHashNN is a novel neural network compression method using multiple hash functions and a small reconstruction network, achieving high compression with minimal accuracy loss.

Contribution

Introduces FunHashNN, a new structure for neural network compression that improves upon HashedNets by using multiple hash functions and a reconstruction network.

Findings

Achieves high compression ratios on benchmark datasets.

Maintains prediction accuracy with significant model size reduction.

Extensible with dual space hashing and multi-hops.

Abstract

As the complexity of deep neural networks (DNNs) trend to grow to absorb the increasing sizes of data, memory and energy consumption has been receiving more and more attentions for industrial applications, especially on mobile devices. This paper presents a novel structure based on functional hashing to compress DNNs, namely FunHashNN. For each entry in a deep net, FunHashNN uses multiple low-cost hash functions to fetch values in the compression space, and then employs a small reconstruction network to recover that entry. The reconstruction network is plugged into the whole network and trained jointly. FunHashNN includes the recently proposed HashedNets as a degenerated case, and benefits from larger value capacity and less reconstruction loss. We further discuss extensions with dual space hashing and multi-hops. On several benchmark datasets, FunHashNN demonstrates high compression ratios with little loss on prediction accuracy.