Fast Rates for Nonstationary Weighted Risk Minimization

TL;DR

This paper analyzes the out-of-sample prediction error of weighted empirical risk minimization under nonstationary data, providing bounds that adapt to distribution drift and data dependence.

Contribution

It offers a general decomposition of excess risk, oracle inequalities under mixing conditions, and applies these to various models, achieving near-optimal rates.

Findings

Provides a decomposition of excess risk into learning and drift error terms.

Establishes oracle inequalities for the learning error under mixing conditions.

Achieves minimax-optimal rates in stationary and unweighted cases.

Abstract

Weighted empirical risk minimization is a common approach to prediction under distribution drift. This article studies its out-of-sample prediction error under nonstationarity. We provide a general decomposition of the excess risk into a learning term and an error term associated with distribution drift, and prove oracle inequalities for the learning error under mixing conditions. The learning bound holds uniformly over arbitrary weight classes and accounts for the effective sample size induced by the weight vector, the complexity of the weight and hypothesis classes, and potential data dependence. We illustrate the applicability and sharpness of our results in (auto-) regression problems with linear models, basis approximations, and neural networks, recovering minimax-optimal rates (up to logarithmic factors) when specialized to unweighted and stationary settings.