Artificial Neural Networks generated by Low Discrepancy Sequences

TL;DR

This paper introduces a novel method for initializing and training sparse artificial neural networks using low discrepancy sequences, achieving comparable accuracy to dense networks with reduced computational costs.

Contribution

The paper proposes a deterministic approach to generate sparse neural networks via low discrepancy sequences, enabling efficient training and hardware-friendly memory access.

Findings

Sparse networks trained from scratch perform comparably to dense networks.

Using low discrepancy sequences improves hardware efficiency by avoiding memory bank conflicts.

The method reduces computational complexity while maintaining high accuracy.

Abstract

Artificial neural networks can be represented by paths. Generated as random walks on a dense network graph, we find that the resulting sparse networks allow for deterministic initialization and even weights with fixed sign. Such networks can be trained sparse from scratch, avoiding the expensive procedure of training a dense network and compressing it afterwards. Although sparse, weights are accessed as contiguous blocks of memory. In addition, enumerating the paths using deterministic low discrepancy sequences, for example the Sobol' sequence, amounts to connecting the layers of neural units by progressive permutations, which naturally avoids bank conflicts in parallel computer hardware. We demonstrate that the artificial neural networks generated by low discrepancy sequences can achieve an accuracy within reach of their dense counterparts at a much lower computational complexity.