Evaluating and Characterizing Incremental Learning from Non-Stationary Data

TL;DR

This paper introduces a standardized testbed using synthetic datasets to evaluate incremental learning algorithms on non-stationary data, addressing evaluation challenges and enabling consistent comparisons.

Contribution

It proposes a novel testing methodology for incremental non-stationary learning algorithms, facilitating better assessment and comparison of their behaviors.

Findings

Testbed effectively characterizes algorithm strengths and weaknesses.

Evaluation metrics need adaptation for non-stationary data.

Methodology provides a common basis for future research.

Abstract

Incremental learning from non-stationary data poses special challenges to the field of machine learning. Although new algorithms have been developed for this, assessment of results and comparison of behaviors are still open problems, mainly because evaluation metrics, adapted from more traditional tasks, can be ineffective in this context. Overall, there is a lack of common testing practices. This paper thus presents a testbed for incremental non-stationary learning algorithms, based on specially designed synthetic datasets. Also, test results are reported for some well-known algorithms to show that the proposed methodology is effective at characterizing their strengths and weaknesses. It is expected that this methodology will provide a common basis for evaluating future contributions in the field.