On the Change of Decision Boundaries and Loss in Learning with Concept Drift

TL;DR

This paper examines the theoretical justification for using interleaved test-train error to detect concept drift, relating it to actual distribution changes and model updates, supported by empirical evidence across various algorithms and datasets.

Contribution

It provides a mathematical analysis linking ITTE changes to true concept drift and model changes, enhancing understanding of drift detection methods.

Findings

ITTE change correlates with real distribution drift

Theoretical justification for ITTE-based drift detection

Empirical validation across multiple algorithms and datasets

Abstract

The notion of concept drift refers to the phenomenon that the distribution generating the observed data changes over time. If drift is present, machine learning models may become inaccurate and need adjustment. Many technologies for learning with drift rely on the interleaved test-train error (ITTE) as a quantity which approximates the model generalization error and triggers drift detection and model updates. In this work, we investigate in how far this procedure is mathematically justified. More precisely, we relate a change of the ITTE to the presence of real drift, i.e., a changed posterior, and to a change of the training result under the assumption of optimality. We support our theoretical findings by empirical evidence for several learning algorithms, models, and datasets.