Representations learnt by SGD and Adaptive learning rules: Conditions that vary sparsity and selectivity in neural networks

TL;DR

This paper explores how different training conditions, like learning rate and batch size, naturally increase sparsity and selectivity in neural networks, providing insights relevant to both neuroscience and machine learning.

Contribution

It identifies specific training conditions that promote sparsity and selectivity in neural networks, a novel investigation into their natural occurrence.

Findings

Large learning rate increases sparsity and selectivity.

Lowering batch size enhances sparsity and selectivity.

Sparsity and selectivity relate to test accuracy.

Abstract

From the point of view of the human brain, continual learning can perform various tasks without mutual interference. An effective way to reduce mutual interference can be found in sparsity and selectivity of neurons. According to Aljundi et al. and Hadsell et al., imposing sparsity at the representational level is advantageous for continual learning because sparse neuronal activations encourage less overlap between parameters, resulting in less interference. Similarly, highly selective neural networks are likely to induce less interference since particular response in neurons will reduce the chance of overlap with other parameters. Considering that the human brain performs continual learning over the lifespan, finding conditions where sparsity and selectivity naturally arises may provide insight for understanding how the brain functions. This paper investigates various conditions that naturally increase sparsity and selectivity in a neural network. This paper tested different optimizers with Hoyer's sparsity metric and CCMAS selectivity metric in MNIST classification task. It is essential to note that investigations on the natural occurrence of sparsity and selectivity concerning various conditions have not been acknowledged in any sector of neuroscience nor machine learning until this day. This paper found that particular conditions increase sparsity and selectivity such as applying a large learning rate and lowering a batch size. In addition to the relationship between the condition, sparsity, and selectivity, the following will be discussed based on empirical analysis: 1. The relationship between sparsity and selectivity and 2. The relationship between test accuracy, sparsity, and selectivity.