Deep Oscillatory Neural Network

TL;DR

The paper introduces the Deep Oscillatory Neural Network (DONN), a brain-inspired model with oscillatory internal dynamics, demonstrating competitive performance on signal and image processing benchmarks.

Contribution

It presents a novel neural network architecture with oscillatory neurons, including complex-valued weights and activations, and extends it to convolutional networks for improved processing.

Findings

Performance comparable or superior to existing models on benchmarks

Incorporates brain-like oscillatory activity into deep neural networks

Proposes complex-valued neural components and training methods

Abstract

We propose a novel, brain-inspired deep neural network model known as the Deep Oscillatory Neural Network (DONN). Deep neural networks like the Recurrent Neural Networks indeed possess sequence processing capabilities but the internal states of the network are not designed to exhibit brain-like oscillatory activity. With this motivation, the DONN is designed to have oscillatory internal dynamics. Neurons of the DONN are either nonlinear neural oscillators or traditional neurons with sigmoidal or ReLU activation. The neural oscillator used in the model is the Hopf oscillator, with the dynamics described in the complex domain. Input can be presented to the neural oscillator in three possible modes. The sigmoid and ReLU neurons also use complex-valued extensions. All the weight stages are also complex-valued. Training follows the general principle of weight change by minimizing the output error and therefore has an overall resemblance to complex backpropagation. A generalization of DONN to convolutional networks known as the Oscillatory Convolutional Neural Network is also proposed. The two proposed oscillatory networks are applied to a variety of benchmark problems in signal and image/video processing. The performance of the proposed models is either comparable or superior to published results on the same data sets.