OEBench: Investigating Open Environment Challenges in Real-World Relational Data Streams

TL;DR

This paper introduces OEBench, a benchmark for evaluating incremental learning algorithms on real-world relational data streams, revealing significant challenges posed by open environment factors like drifts and anomalies.

Contribution

The study develops a comprehensive benchmark with real-world data, highlighting the widespread nature of open environment challenges and evaluating existing algorithms' performance.

Findings

Open environment scenarios are common in real-world data streams.

Increased data quantity does not always improve model accuracy.

Current techniques are insufficient to handle open environment challenges.

Abstract

How to get insights from relational data streams in a timely manner is a hot research topic. Data streams can present unique challenges, such as distribution drifts, outliers, emerging classes, and changing features, which have recently been described as open environment challenges for machine learning. While existing studies have been done on incremental learning for data streams, their evaluations are mostly conducted with synthetic datasets. Thus, a natural question is how those open environment challenges look like and how existing incremental learning algorithms perform on real-world relational data streams. To fill this gap, we develop an Open Environment Benchmark named OEBench to evaluate open environment challenges in real-world relational data streams. Specifically, we investigate 55 real-world relational data streams and establish that open environment scenarios are indeed widespread, which presents significant challenges for stream learning algorithms. Through benchmarks with existing incremental learning algorithms, we find that increased data quantity may not consistently enhance the model accuracy when applied in open environment scenarios, where machine learning models can be significantly compromised by missing values, distribution drifts, or anomalies in real-world data streams. The current techniques are insufficient in effectively mitigating these challenges brought by open environments. More researches are needed to address real-world open environment challenges. All datasets and code are open-sourced in https://github.com/sjtudyq/OEBench.