Benign Overfitting in Time Series Linear Models with Over-Parameterization

TL;DR

This paper investigates benign overfitting in over-parameterized linear models applied to dependent time-series data, providing theoretical risk bounds and emphasizing the role of temporal covariance coherence.

Contribution

It extends analysis of over-parameterized linear models to dependent time-series data, deriving non-asymptotic risk bounds considering temporal covariance coherence.

Findings

Risk bounds depend on temporal covariance coherence

Convergence rate influenced by temporal covariance coherence

Applicable to various dependent time-series processes

Abstract

The success of large-scale models in recent years has increased the importance of statistical models with numerous parameters. Several studies have analyzed over-parameterized linear models with high-dimensional data, which may not be sparse; however, existing results rely on the assumption of sample independence. In this study, we analyze a linear regression model with dependent time-series data in an over-parameterized setting. We consider an estimator using interpolation and develop a theory for the excess risk of the estimator. Then, we derive non-asymptotic risk bounds for the estimator for cases with dependent data. This analysis reveals that the coherence of the temporal covariance plays a key role; the risk bound is influenced by the product of temporal covariance matrices at different time steps. Moreover, we show the convergence rate of the risk bound and demonstrate that it is also influenced by the coherence of the temporal covariance. Finally, we provide several examples of specific dependent processes applicable to our setting.