Optimal Neural Network Feature Selection for Spatial-Temporal Forecasting

TL;DR

This paper empirically demonstrates how to optimally select input features for neural networks in spatial-temporal forecasting by applying dynamical systems theory, specifically non-linear embedding theorems, across various systems.

Contribution

It introduces a method for constructing optimal neural network input representations based on non-linear embedding theorems, validated through extensive simulations and real data.

Findings

Optimal input features form a grid with lags from mutual information minima.

The embedding dimension determines the number of points in space/time.

The proposed method performs near-optimally across diverse systems.

Abstract

In this paper, we show empirical evidence on how to construct the optimal feature selection or input representation used by the input layer of a feedforward neural network for the propose of forecasting spatial-temporal signals. The approach is based on results from dynamical systems theory, namely the non-linear embedding theorems. We demonstrate it for a variety of spatial-temporal signals, with one spatial and one temporal dimensions, and show that the optimal input layer representation consists of a grid, with spatial/temporal lags determined by the minimum of the mutual information of the spatial/temporal signals and the number of points taken in space/time decided by the embedding dimension of the signal. We present evidence of this proposal by running a Monte Carlo simulation of several combinations of input layer feature designs and show that the one predicted by the non-linear embedding theorems seems to be optimal or close of optimal. In total we show evidence in four unrelated systems: a series of coupled Henon maps; a series of couple Ordinary Differential Equations (Lorenz-96) phenomenologically modelling atmospheric dynamics; the Kuramoto-Sivashinsky equation, a partial differential equation used in studies of instabilities in laminar flame fronts and finally real physical data from sunspot areas in the Sun (in latitude and time) from 1874 to 2015.