Are Data Embeddings effective in time series forecasting?

TL;DR

This study critically evaluates the effectiveness of data embedding layers in time series forecasting models, revealing that removing them often maintains or improves accuracy and efficiency across multiple models and datasets.

Contribution

The paper provides extensive ablation studies demonstrating that data embedding layers may be unnecessary or even detrimental in many time series forecasting scenarios.

Findings

Removing embedding layers does not degrade performance.

In many cases, removing embeddings improves accuracy.

Eliminating embeddings enhances computational efficiency.

Abstract

Time series forecasting plays a crucial role in many real-world applications, and numerous complex forecasting models have been proposed in recent years. Despite their architectural innovations, most state-of-the-art models report only marginal improvements -- typically just a few thousandths in standard error metrics. These models often incorporate complex data embedding layers to transform raw inputs into higher-dimensional representations to enhance accuracy. But are data embedding techniques actually effective in time series forecasting? Through extensive ablation studies across fifteen state-of-the-art models and four benchmark datasets, we find that removing data embedding layers from many state-of-the-art models does not degrade forecasting performance. In many cases, it improves both accuracy and computational efficiency. The gains from removing embedding layers often exceed the performance differences typically reported between competing models. Code available at: https://github.com/neuripsdataembedidng/DataEmbedding