Instance-Conditional Timescales of Decay for Non-Stationary Learning

TL;DR

This paper introduces a novel approach for non-stationary learning that models multiple timescales of decay to adaptively weight instances, leading to significant accuracy improvements on real-world datasets with concept drift.

Contribution

It proposes a mixture of timescales model with an auxiliary scorer learned via nested optimization to enhance learning under concept drift.

Findings

Achieved up to 15% relative accuracy gains on a large real-world dataset.

Demonstrated effectiveness on multiple non-stationary learning datasets.

Extended the approach to continual learning with state-of-the-art results.

Abstract

Slow concept drift is a ubiquitous, yet under-studied problem in practical machine learning systems. In such settings, although recent data is more indicative of future data, naively prioritizing recent instances runs the risk of losing valuable information from the past. We propose an optimization-driven approach towards balancing instance importance over large training windows. First, we model instance relevance using a mixture of multiple timescales of decay, allowing us to capture rich temporal trends. Second, we learn an auxiliary scorer model that recovers the appropriate mixture of timescales as a function of the instance itself. Finally, we propose a nested optimization objective for learning the scorer, by which it maximizes forward transfer for the learned model. Experiments on a large real-world dataset of 39M photos over a 9 year period show upto 15% relative gains in accuracy compared to other robust learning baselines. We replicate our gains on two collections of real-world datasets for non-stationary learning, and extend our work to continual learning settings where, too, we beat SOTA methods by large margins.