Efficient Time-Series Approximation with Linear Recurrent Neural Networks: Architecture Learning and Predictive Power

TL;DR

This paper introduces a method for efficiently approximating time-series data using linear recurrent neural networks, which can learn architectures and predict future values without backpropagation, demonstrated on various case studies.

Contribution

The paper presents a novel approach to learn both the weights and architecture of linear RNNs by spectral analysis, enabling significant size reduction and improved prediction.

Findings

LRNNs can approximate any time-dependent function f(t).

LRNNs outperform state-of-the-art on superimposed oscillators.

Network size can be reduced by spectral analysis of the transition matrix.

Abstract

Recurrent neural networks are a powerful means to cope with time series. We show how autoregressive linear, i.e., linearly activated recurrent neural networks (LRNNs) can approximate any time-dependent function f(t). The approximation can effectively be learned by simply solving a linear equation system; no backpropagation or similar methods are needed. Furthermore, and this is the main contribution of this paper, the size of an LRNN can be reduced significantly in one step after inspecting the spectrum of the network transition matrix, i.e., its eigenvalues, by taking only the most relevant components. Therefore, in contrast to other approaches, we do not only learn network weights but also the network architecture. LRNNs have interesting properties: They end up in ellipse trajectories in the long run and allow the prediction of further values and compact representations of functions. We demonstrate this by several case studies, among them multiple superimposed oscillators (MSO), robotic soccer (RoboCup), and stock price prediction. LRNNs outperform the previous state-of-the-art for the MSO task with a minimal number of units.