Time-Varying Propensity Score to Bridge the Gap between the Past and Present

TL;DR

This paper introduces a time-varying propensity score method to detect and adapt to gradual data shifts over time, improving model updates across various machine learning tasks.

Contribution

It proposes a novel time-varying propensity score that selectively samples past data based on gradual distribution changes, enhancing model robustness.

Findings

Effective in detecting gradual data shifts

Improves model updating in supervised learning

Enhances adaptation in reinforcement learning

Abstract

Real-world deployment of machine learning models is challenging because data evolves over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data which allows us to selectively sample past data to update the model -- not just similar data from the past like that of a standard propensity score but also data that evolved in a similar fashion in the past. The time-varying propensity score is quite general: we demonstrate different ways of implementing it and evaluate it on a variety of problems ranging from supervised learning (e.g., image classification problems) where data undergoes a sequence of gradual shifts, to reinforcement learning tasks (e.g., robotic manipulation and continuous control) where data shifts as the policy or the task changes.