Spectral Neural Networks: Approximation Theory and Optimization Landscape

TL;DR

This paper explores the theoretical foundations of Spectral Neural Networks, focusing on the tradeoff between network size and spectral information learning, and analyzing the complex optimization landscape involved in training.

Contribution

It provides the first quantitative analysis of the neuron-spectral information tradeoff and investigates the non-convex optimization landscape of SNNs.

Findings

Quantitative insights into neuron-spectral information tradeoff

Analysis of the non-convex optimization landscape of SNNs

Understanding of training dynamics and convergence challenges

Abstract

There is a large variety of machine learning methodologies that are based on the extraction of spectral geometric information from data. However, the implementations of many of these methods often depend on traditional eigensolvers, which present limitations when applied in practical online big data scenarios. To address some of these challenges, researchers have proposed different strategies for training neural networks as alternatives to traditional eigensolvers, with one such approach known as Spectral Neural Network (SNN). In this paper, we investigate key theoretical aspects of SNN. First, we present quantitative insights into the tradeoff between the number of neurons and the amount of spectral geometric information a neural network learns. Second, we initiate a theoretical exploration of the optimization landscape of SNN's objective to shed light on the training dynamics of SNN. Unlike typical studies of convergence to global solutions of NN training dynamics, SNN presents an additional complexity due to its non-convex ambient loss function.