A Survey of Deep Learning and Foundation Models for Time Series Forecasting

TL;DR

This survey reviews recent advances in deep learning and foundation models for time series forecasting, highlighting architectural innovations, challenges, and the integration of scientific knowledge to improve model performance.

Contribution

It provides a comprehensive overview of state-of-the-art deep learning techniques and discusses future directions for incorporating knowledge into time series forecasting models.

Findings

Deep learning models are increasingly effective with architectural advances.

Foundation models enable better understanding and transfer of knowledge.

Challenges remain in interpretability and data scarcity for pandemic prediction.

Abstract

Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.