Adaptive Estimation and Learning under Temporal Distribution Shift

TL;DR

This paper introduces a wavelet soft-thresholding estimator for accurate groundtruth estimation under temporal distribution shift, connecting non-stationarity with wavelet sparsity, and extends its application to classification and signal denoising.

Contribution

It develops a novel wavelet-based estimation method that adapts to unknown temporal shifts and links non-stationarity with wavelet sparsity, improving estimation accuracy.

Findings

Optimal error bounds for groundtruth estimation under shift.

Validated the estimator through numerical experiments.

Derived new algorithms for total-variation denoising.

Abstract

In this paper, we study the problem of estimation and learning under temporal distribution shift. Consider an observation sequence of length $n$, which is a noisy realization of a time-varying groundtruth sequence. Our focus is to develop methods to estimate the groundtruth at the final time-step while providing sharp point-wise estimation error rates. We show that, without prior knowledge on the level of temporal shift, a wavelet soft-thresholding estimator provides an optimal estimation error bound for the groundtruth. Our proposed estimation method generalizes existing researches Mazzetto and Upfal (2023) by establishing a connection between the sequence's non-stationarity level and the sparsity in the wavelet-transformed domain. Our theoretical findings are validated by numerical experiments. Additionally, we applied the estimator to derive sparsity-aware excess risk bounds for binary classification under distribution shift and to develop computationally efficient training objectives. As a final contribution, we draw parallels between our results and the classical signal processing problem of total-variation denoising (Mammen and van de Geer,1997; Tibshirani, 2014), uncovering novel optimal algorithms for such task.