Online Conformal Model Selection for Nonstationary Time Series

TL;DR

This paper presents MPS, an online conformal inference framework for adaptive model selection in nonstationary time series, effectively handling evolving data dynamics and outperforming traditional offline methods.

Contribution

The paper introduces MPS, a novel real-time model selection method combining conformal inference and confidence sets for nonstationary time series.

Findings

MPS reliably identifies optimal models in nonstationary environments.

MPS produces small, high-quality model sets that adapt over time.

Demonstrated effectiveness on simulations and real-world data.

Abstract

This paper introduces the MPS (Model Prediction Set), a novel framework for online model selection for nonstationary time series. Classical model selection methods, such as information criteria and cross-validation, rely heavily on the stationarity assumption and often fail in dynamic environments which undergo gradual or abrupt changes over time. Yet real-world data are rarely stationary, and model selection under nonstationarity remains a largely open problem. To tackle this challenge, we combine conformal inference with model confidence sets to develop a procedure that adaptively selects models best suited to the evolving dynamics at any given time. Concretely, the MPS updates in real time a confidence set of candidate models that covers the best model for the next time period with a specified long-run probability, while adapting to nonstationarity of unknown forms. Through simulations and real-world data analysis, we demonstrate that MPS reliably and efficiently identifies optimal models under nonstationarity, an essential capability lacking in offline methods. Moreover, MPS frequently produces high-quality sets with small cardinality, whose evolution offers deeper insights into changing dynamics. As a generic framework, MPS accommodates any data-generating process, data structure, model class, training method, and evaluation metric, making it broadly applicable across diverse problem settings.