Training Deep Fourier Neural Networks To Fit Time-Series Data

TL;DR

This paper introduces a novel deep Fourier neural network training method for time-series data, utilizing Fourier-based initialization, dynamic parameter tuning, and regularization to enhance modeling, generalization, and extrapolation of nonlinear trends.

Contribution

It proposes a new training approach for deep Fourier neural networks with sinusoidal activations, including Fourier initialization and adaptive regularization for improved time-series modeling.

Findings

Effective extrapolation of nonlinear trends demonstrated

Deeper layers model sequences with fewer sinusoid units

Non-uniform regularization enhances generalization

Abstract

We present a method for training a deep neural network containing sinusoidal activation functions to fit to time-series data. Weights are initialized using a fast Fourier transform, then trained with regularization to improve generalization. A simple dynamic parameter tuning method is employed to adjust both the learning rate and regularization term, such that stability and efficient training are both achieved. We show how deeper layers can be utilized to model the observed sequence using a sparser set of sinusoid units, and how non-uniform regularization can improve generalization by promoting the shifting of weight toward simpler units. The method is demonstrated with time-series problems to show that it leads to effective extrapolation of nonlinear trends.