The Procrustean Bed of Time Series: The Optimization Bias in Point-wise Loss Functions

TL;DR

This paper identifies and formalizes the systematic bias introduced by point-wise loss functions in time series forecasting, showing it depends on data properties and proposing debiasing methods that improve predictive accuracy.

Contribution

It formalizes the optimization bias in point-wise loss functions, derives theoretical bounds, and proposes a debiasing approach that enhances forecasting performance across multiple datasets.

Findings

Bias is governed by sequence length and SSNR, independent of model architecture.

The proposed debiasing method reduces MSE/MAE by over 5% on average.

Theoretical bounds match empirical observations of bias and improvements.

Abstract

Intuitively, a more deterministic time series should be easier to forecast. However, point-wise loss functions (e.g., MSE and MAE), serving as differentiable surrogates for the ideal optimization target, score each timestamp independently and therefore disregard temporal dependence. This mismatch induces a systematic optimization bias that cannot be eliminated merely by improving model expressiveness or optimizer. To formalize this issue, we define the Expectation of Optimization Bias (EOB) as the Kullback--Leibler divergence between the true joint distribution and the factorized i.i.d. surrogate induced by the point-wise paradigm. Under covariance-stationary Gaussian assumptions, we derive closed-form expressions for the stochastic component of EOB, establishing it as an irreducible lower bound on the total bias in linear systems, and further extend it to nonlinear regimes through a Gaussian mixture model lower bound. Crucially, we prove this bias is governed intrinsically by two data properties, i.e., sequence length and Structural Signal-to-Noise Ratio (SSNR), regardless of specific model architecture, optimizer, or point-wise loss forms. This theory motivates a principled debiasing program based on sequence length reduction and structural orthogonalization, which we instantiate through DFT/DWT combined with a novel harmonized $\ell_p$ norm. Extensive experiments validate the predicted SSNR--horizon dynamics, resolve the classic trigonometric fitting failure as an objective-induced pathology, and demonstrate substantial plug-and-play gains. Notably, on iTransformer, our proposed objective reduces average MSE/MAE by 5.2%/5.0% in forecasting across 11 datasets and by 27.4%/19.4% in imputation across 9 datasets.