TimeSynth: A Framework for Uncovering Systematic Biases in Time Series Forecasting

TL;DR

TimeSynth is a framework that systematically evaluates time series forecasting models, revealing biases in linear models and advantages of nonlinear models like CNNs and Transformers, especially with complex signals.

Contribution

It introduces a structured, realistic benchmarking framework that isolates model performance factors and challenges prior claims of linear model dominance in time series forecasting.

Findings

Linear models tend to collapse to simple oscillations regardless of complexity.

Nonlinear models, especially CNNs and Transformers, outperform linear models on complex signals.

The framework assesses robustness under distribution and noise shifts.

Abstract

Time series forecasting is a fundamental tool with wide ranging applications, yet recent debates question whether complex nonlinear architectures truly outperform simple linear models. Prior claims of dominance of the linear model often stem from benchmarks that lack diverse temporal dynamics and employ biased evaluation protocols. We revisit this debate through TimeSynth, a structured framework that emulates key properties of real world time series,including non-stationarity, periodicity, trends, and phase modulation by creating synthesized signals whose parameters are derived from real-world time series. Evaluating four model families Linear, Multi Layer Perceptrons (MLP), Convolutional Neural Networks (CNNs), and Transformers, we find a systematic bias in linear models: they collapse to simple oscillation regardless of signal complexity. Nonlinear models avoid this collapse and gain clear advantages as signal complexity increases. Notably, Transformers and CNN based models exhibit slightly greater adaptability to complex modulated signals compared to MLPs. Beyond clean forecasting, the framework highlights robustness differences under distribution and noise shifts and removes biases of prior benchmarks by using independent instances for train, test, and validation for each signal family. Collectively, TimeSynth provides a principled foundation for understanding when different forecasting approaches succeed or fail, moving beyond oversimplified claims of model equivalence.