Interpretable Early Warnings using Machine Learning in an Online Game-experiment

TL;DR

This paper presents a machine learning system that predicts critical transitions in an online social experiment, outperforming traditional indicators and providing insights into the dynamics of social regime shifts.

Contribution

It introduces a novel ML-based early warning system for social system transitions, demonstrating high accuracy and interpretability across different online contexts.

Findings

The system detects half of the transitions within 20 minutes at a 3.6% false positive rate.

It generalizes well across different years of the r/place experiment.

SHAP analysis reveals key drivers like critical slowing down and lack of coordination.

Abstract

Stemming from physics and later applied to other fields such as ecology, the theory of critical transitions suggests that some regime shifts are preceded by statistical early warning signals. Reddit's r/place experiment, a large-scale social game, provides a unique opportunity to test these signals consistently across thousands of subsystems undergoing critical transitions. In r/place, millions of users collaboratively created ''compositions'', or pixel-art drawings, in which transitions occur when one composition rapidly replaces another. We develop a machine-learning-based early warning system that combines the predictive power of multiple system-specific time series via gradient-boosted decision trees with memory-retaining features. Our method significantly outperforms standard early warning indicators. Trained on the 2022 r/place data, our algorithm detects half of the transitions occurring within 20 min at a false positive rate of just 3.6%. Its performance remains robust when tested on the 2023 r/place event, demonstrating generalizability across different contexts. Using SHapley Additive exPlanations (SHAP) for interpreting the predictions, we investigate the underlying drivers of warnings, which could be relevant to other complex systems, especially online social systems. We reveal an interplay of patterns preceding transitions, such as critical slowing down or speeding up, a lack of innovation or coordination, turbulent histories, and a lack of image complexity. These findings show the potential of machine learning indicators in socio-ecological systems for predicting regime shifts and understanding their dynamics.