Different Spectral Representations in Optimized Artificial Neural Networks and Brains

TL;DR

This study systematically explores how spectral regularizers with different power-law exponents affect neural network performance and robustness, revealing that higher exponents can improve accuracy and adversarial robustness, challenging the idea that brain-like spectra are always optimal.

Contribution

It is the first systematic investigation of how varying spectral regularizers influence ANN performance and robustness across different tasks and architectures.

Findings

Higher spectral exponents (around 2-3) improve accuracy and robustness.

Brain-like spectral properties ($eta \,\sim\, 1$) are not always optimal.

Spectral regularizers can be tuned to optimize specific performance metrics.

Abstract

Recent studies suggest that artificial neural networks (ANNs) that match the spectral properties of the mammalian visual cortex -- namely, the $\sim 1/n$ eigenspectrum of the covariance matrix of neural activities -- achieve higher object recognition performance and robustness to adversarial attacks than those that do not. To our knowledge, however, no previous work systematically explored how modifying the ANN's spectral properties affects performance. To fill this gap, we performed a systematic search over spectral regularizers, forcing the ANN's eigenspectrum to follow $1/n^\alpha$ power laws with different exponents $\alpha$. We found that larger powers (around 2--3) lead to better validation accuracy and more robustness to adversarial attacks on dense networks. This surprising finding applied to both shallow and deep networks and it overturns the notion that the brain-like spectrum (corresponding to $\alpha \sim 1$) always optimizes ANN performance and/or robustness. For convolutional networks, the best $\alpha$ values depend on the task complexity and evaluation metric: lower $\alpha$ values optimized validation accuracy and robustness to adversarial attack for networks performing a simple object recognition task (categorizing MNIST images of handwritten digits); for a more complex task (categorizing CIFAR-10 natural images), we found that lower $\alpha$ values optimized validation accuracy whereas higher $\alpha$ values optimized adversarial robustness. These results have two main implications. First, they cast doubt on the notion that brain-like spectral properties ($\alpha \sim 1$) \emph{always} optimize ANN performance. Second, they demonstrate the potential for fine-tuned spectral regularizers to optimize a chosen design metric, i.e., accuracy and/or robustness.