LLAMA LIMA: A Living Meta-Analysis on the Effects of Generative AI on Learning Mathematics

TL;DR

This paper introduces a living meta-analysis framework for continuously updating research on generative AI's impact on learning mathematics, using Bayesian methods to synthesize evolving evidence.

Contribution

It presents a novel, continuously updated meta-analytical approach for rapidly evolving AI research in mathematics education, addressing timeliness and evidence limitations.

Findings

Positive effect size of 0.42 on learning outcomes

No evidence of publication bias

Limited but growing evidence base

Abstract

The capabilities of generative AI in mathematics education are rapidly evolving, posing significant challenges for research to keep pace. Research syntheses remain scarce and risk being outdated by the time of publication. To address this issue, we present a Living Meta-Analysis (LIMA) on the effects of generative AI-based interventions for learning mathematics. Following PRISMA-LSR guidelines, we continuously update the literature base, apply a Bayesian multilevel meta-regression model to account for nested and cumulative data, and publish updated versions on a preprint server at regular intervals. This paper reports results from the second version, including 21 studies, 6 of which were newly included since the first version. The analyses indicate a positive effect (g = 0.42) with a wide credible interval [0.13, 0.72], reflecting the still limited evidence base. Results indicate no publication bias.