Sparse Deep Learning for Time Series Data: Theory and Applications

TL;DR

This paper develops a theoretical framework for sparse deep learning with dependent data like time series, demonstrating consistent estimation, asymptotic normality, and improved uncertainty quantification over existing methods.

Contribution

It introduces a theory for sparse RNNs with dependent data, showing their consistent estimation, asymptotic properties, and superior performance in uncertainty quantification and model compression.

Findings

Sparse RNNs can be consistently estimated.

Predictions are asymptotically normally distributed.

Outperforms state-of-the-art methods in uncertainty quantification.

Abstract

Sparse deep learning has become a popular technique for improving the performance of deep neural networks in areas such as uncertainty quantification, variable selection, and large-scale network compression. However, most existing research has focused on problems where the observations are independent and identically distributed (i.i.d.), and there has been little work on the problems where the observations are dependent, such as time series data and sequential data in natural language processing. This paper aims to address this gap by studying the theory for sparse deep learning with dependent data. We show that sparse recurrent neural networks (RNNs) can be consistently estimated, and their predictions are asymptotically normally distributed under appropriate assumptions, enabling the prediction uncertainty to be correctly quantified. Our numerical results show that sparse deep learning outperforms state-of-the-art methods, such as conformal predictions, in prediction uncertainty quantification for time series data. Furthermore, our results indicate that the proposed method can consistently identify the autoregressive order for time series data and outperform existing methods in large-scale model compression. Our proposed method has important practical implications in fields such as finance, healthcare, and energy, where both accurate point estimates and prediction uncertainty quantification are of concern.