Learning Outcomes, Assessment, and Evaluation in Educational Recommender Systems: A Systematic Review

TL;DR

This systematic review analyzes how Educational Recommender Systems measure and optimize learning, revealing a focus on relevance metrics and a gap in outcome-based assessment of pedagogical effects across diverse educational contexts.

Contribution

It provides a comprehensive overview of evaluation methods in ERS research, highlighting the need for more outcome-based assessments in informal and large-scale settings.

Findings

Rating-based relevance is most common as a target metric.

Less than half of the papers optimize learning-based metrics.

Only a third use outcome-based assessment for pedagogical effects.

Abstract

In this paper, we analyse how learning is measured and optimized in Educational Recommender Systems (ERS). In particular, we examine the target metrics and evaluation methods used in the existing ERS research, with a particular focus on the pedagogical effect of recommendations. While conducting this systematic literature review (SLR), we identified 1395 potentially relevant papers, then filtered them through the inclusion and exclusion criteria, and finally selected and analyzed 28 relevant papers. Rating-based relevance is the most popular target metric, while less than a half of papers optimize learning-based metrics. Only a third of the papers used outcome-based assessment to measure the pedagogical effect of recommendations, mostly within a formal university course. This indicates a gap in ERS research with respect to assessing the pedagogical effect of recommendations at scale and in informal education settings.