Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?

TL;DR

This paper investigates how the Rashomon effect impacts variable importance in educational data mining, demonstrating that multiple models can provide more reliable insights into key demographic factors affecting student outcomes.

Contribution

It introduces the use of the Rashomon set with various machine learning models to improve variable importance analysis in educational research.

Findings

Rashomon set improves predictive accuracy by 2-6%

Variable importance varies across models and courses

Binary classification yields more consistent importance results

Abstract

This study explores how the Rashomon effect influences variable importance in the context of student demographics used for academic outcomes prediction. Our research follows the way machine learning algorithms are employed in Educational Data Mining, focusing on highlighting the so-called Rashomon effect. The study uses the Rashomon set of simple-yet-accurate models trained using decision trees, random forests, light GBM, and XGBoost algorithms with the Open University Learning Analytics Dataset. We found that the Rashomon set improves the predictive accuracy by 2-6%. Variable importance analysis revealed more consistent and reliable results for binary classification than multiclass classification, highlighting the complexity of predicting multiple outcomes. Key demographic variables imd_band and highest_education were identified as vital, but their importance varied across courses, especially in course DDD. These findings underscore the importance of model choice and the need for caution in generalizing results, as different models can lead to different variable importance rankings. The codes for reproducing the experiments are available in the repository: https://anonymous.4open.science/r/JEDM_paper-DE9D.