Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift

TL;DR

This paper introduces a universal calibration method for time-series forecasting models to detect and adapt to context-driven distribution shifts, improving prediction accuracy in real-world scenarios.

Contribution

It proposes a novel CDS detector called Reconditionor and an adaptation framework named SOLID, which together enable model calibration against distribution shifts caused by temporal contexts.

Findings

Reconditionor effectively detects severe distribution shifts.

SOLID improves forecasting accuracy on real-world datasets.

The methods are model-agnostic and adaptable to various forecasting models.

Abstract

Recent years have witnessed the success of introducing deep learning models to time series forecasting. From a data generation perspective, we illustrate that existing models are susceptible to distribution shifts driven by temporal contexts, whether observed or unobserved. Such context-driven distribution shift (CDS) introduces biases in predictions within specific contexts and poses challenges for conventional training paradigms. In this paper, we introduce a universal calibration methodology for the detection and adaptation of CDS with a trained model. To this end, we propose a novel CDS detector, termed the "residual-based CDS detector" or "Reconditionor", which quantifies the model's vulnerability to CDS by evaluating the mutual information between prediction residuals and their corresponding contexts. A high Reconditionor score indicates a severe susceptibility, thereby necessitating model adaptation. In this circumstance, we put forth a straightforward yet potent adapter framework for model calibration, termed the "sample-level contextualized adapter" or "SOLID". This framework involves the curation of a contextually similar dataset to the provided test sample and the subsequent fine-tuning of the model's prediction layer with a limited number of steps. Our theoretical analysis demonstrates that this adaptation strategy can achieve an optimal bias-variance trade-off. Notably, our proposed Reconditionor and SOLID are model-agnostic and readily adaptable to a wide range of models. Extensive experiments show that SOLID consistently enhances the performance of current forecasting models on real-world datasets, especially on cases with substantial CDS detected by the proposed Reconditionor, thus validating the effectiveness of the calibration approach.