Balanced and Deterministic Weight-sharing Helps Network Performance

TL;DR

This paper introduces ArbNets, a flexible framework for weight-sharing in neural networks, demonstrating that balanced and deterministic weight-sharing improves network performance through novel hash functions and extensive experiments.

Contribution

It generalizes existing weight-sharing methods into ArbNets, introduces new hash functions, and empirically shows that balanced deterministic weight-sharing enhances neural network performance.

Findings

Balanced weight-sharing improves accuracy.

Deterministic hash functions outperform random ones.

New hash functions effectively facilitate weight-sharing.

Abstract

Weight-sharing plays a significant role in the success of many deep neural networks, by increasing memory efficiency and incorporating useful inductive priors about the problem into the network. But understanding how weight-sharing can be used effectively in general is a topic that has not been studied extensively. Chen et al. [2015] proposed HashedNets, which augments a multi-layer perceptron with a hash table, as a method for neural network compression. We generalize this method into a framework (ArbNets) that allows for efficient arbitrary weight-sharing, and use it to study the role of weight-sharing in neural networks. We show that common neural networks can be expressed as ArbNets with different hash functions. We also present two novel hash functions, the Dirichlet hash and the Neighborhood hash, and use them to demonstrate experimentally that balanced and deterministic weight-sharing helps with the performance of a neural network.