Do intelligent tutoring systems benefit K-12 students? A meta-analysis and evaluation of heterogeneity of treatment effects in the U.S

TL;DR

This meta-analysis assesses the effectiveness of intelligent tutoring systems in U.S. K-12 education, finding overall positive effects and identifying key factors influencing their success across different school levels and student groups.

Contribution

It provides a comprehensive evaluation of heterogeneity in ITS effects and identifies critical moderators affecting their efficacy in K-12 settings.

Findings

Significant positive effect size of 0.271 on learning outcomes

Effectiveness consistent across elementary and middle schools

Lower effects observed in rural school studies

Abstract

To expand the use of intelligent tutoring systems (ITS) in K-12 schools, it is essential to understand the conditions under which their use is most beneficial. This meta-analysis evaluated the heterogeneity of ITS effects across studies focusing on elementary, middle, and high schools in the U.S. It included 18 studies with 77 effect sizes across 11 ITS. Overall, there was a significant positive effect size of ITS on U.S. K-12 students' learning outcomes (g=0.271, SE=0.011, p=0.001). Furthermore, effect sizes were similar across elementary and middle schools, and for low-achieving students, but were lower in studies including rural schools. A MetaForest analysis showed that providing worked-out examples, intervention duration, intervention condition, type of learning outcome, and immediate measurement were the most important moderators of treatment effects.