Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

TL;DR

This paper introduces a neuro-inspired deep neural network approach that promotes sparse, strong activations through layer-wise Hebbian and anti-Hebbian costs, improving robustness and interpretability.

Contribution

It proposes a novel training method combining Hebbian principles with divisive normalization, leading to sparser activations and increased robustness without extensive retraining.

Findings

Sparser activations with slight accuracy trade-off

Enhanced robustness to noise

Improved resistance to adversarial attacks

Abstract

While end-to-end training of Deep Neural Networks (DNNs) yields state of the art performance in an increasing array of applications, it does not provide insight into, or control over, the features being extracted. We report here on a promising neuro-inspired approach to DNNs with sparser and stronger activations. We use standard stochastic gradient training, supplementing the end-to-end discriminative cost function with layer-wise costs promoting Hebbian ("fire together," "wire together") updates for highly active neurons, and anti-Hebbian updates for the remaining neurons. Instead of batch norm, we use divisive normalization of activations (suppressing weak outputs using strong outputs), along with implicit $\ell_2$ normalization of neuronal weights. Experiments with standard image classification tasks on CIFAR-10 demonstrate that, relative to baseline end-to-end trained architectures, our proposed architecture (a) leads to sparser activations (with only a slight compromise on accuracy), (b) exhibits more robustness to noise (without being trained on noisy data), (c) exhibits more robustness to adversarial perturbations (without adversarial training).