HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts

TL;DR

HyperTime is a hyperparameter optimization method designed to identify hyperparameters that ensure robust predictive performance under temporal distribution shifts, combining robust optimization principles with theoretical and empirical validation.

Contribution

It introduces HyperTime, a novel hyperparameter optimization approach that explicitly accounts for temporal shifts, leveraging worst-case validation loss to improve robustness.

Findings

HyperTime outperforms baseline methods on multiple tasks with temporal shifts.

Theoretical analysis shows bounds on expected test loss.

Empirical results confirm robustness against distribution shifts.

Abstract

In this work, we propose a hyperparameter optimization method named \emph{HyperTime} to find hyperparameters robust to potential temporal distribution shifts in the unseen test data. Our work is motivated by an important observation that it is, in many cases, possible to achieve temporally robust predictive performance via hyperparameter optimization. Based on this observation, we leverage the `worst-case-oriented' philosophy from the robust optimization literature to help find such robust hyperparameter configurations. HyperTime imposes a lexicographic priority order on average validation loss and worst-case validation loss over chronological validation sets. We perform a theoretical analysis on the upper bound of the expected test loss, which reveals the unique advantages of our approach. We also demonstrate the strong empirical performance of the proposed method on multiple machine learning tasks with temporal distribution shifts.