Pruning Increases Orderedness in Recurrent Computation

TL;DR

This paper shows that pruning can induce directionality in neural networks, increasing orderedness in recurrent computation without sacrificing performance, suggesting directionality is an advantageous bias rather than a necessity.

Contribution

It demonstrates that directionality in neural networks can be induced through pruning, challenging the idea that it must be hard-wired and highlighting its role as an inductive bias.

Findings

Pruning increases topological ordering in neural networks.

Induced directionality does not impair network performance.

Directionality can be learned via gradient descent and sparsification.

Abstract

Inspired by the prevalence of recurrent circuits in biological brains, we investigate the degree to which directionality is a helpful inductive bias for artificial neural networks. Taking directionality as topologically-ordered information flow between neurons, we formalise a perceptron layer with all-to-all connections (mathematically equivalent to a weight-tied recurrent neural network) and demonstrate that directionality, a hallmark of modern feed-forward networks, can be induced rather than hard-wired by applying appropriate pruning techniques. Across different random seeds our pruning schemes successfully induce greater topological ordering in information flow between neurons without compromising performance, suggesting that directionality is not a prerequisite for learning, but may be an advantageous inductive bias discoverable by gradient descent and sparsification.