A Discussion on Generalization in Next-Activity Prediction

TL;DR

This paper critically examines current evaluation methods in next-activity prediction, revealing significant data leakage issues and emphasizing the need for more robust, generalization-focused evaluation scenarios.

Contribution

It uncovers data leakage problems in existing event logs and proposes diverse prediction scenarios to better assess generalization in next-activity prediction.

Findings

Common event logs contain substantial example leakage.

Trivial prediction methods perform nearly as well as deep learning models.

Robust evaluation requires new scenarios emphasizing generalization.

Abstract

Next activity prediction aims to forecast the future behavior of running process instances. Recent publications in this field predominantly employ deep learning techniques and evaluate their prediction performance using publicly available event logs. This paper presents empirical evidence that calls into question the effectiveness of these current evaluation approaches. We show that there is an enormous amount of example leakage in all of the commonly used event logs, so that rather trivial prediction approaches perform almost as well as ones that leverage deep learning. We further argue that designing robust evaluations requires a more profound conceptual engagement with the topic of next-activity prediction, and specifically with the notion of generalization to new data. To this end, we present various prediction scenarios that necessitate different types of generalization to guide future research.