An experimental evaluation of an AI-powered interactive learning platform

TL;DR

This study shows that an AI-powered learning platform improves student engagement and learning outcomes compared to traditional digital textbooks.

Contribution

The novel contribution is demonstrating the effectiveness of an AI-driven platform with interactive representations in enhancing learning outcomes.

Findings

Students using Learn Your Way had a more positive learning experience.

Students using the platform achieved better learning outcomes than those using digital textbooks.

Abstract

Generative AI, which is capable of transforming static content into dynamic learning experiences, holds the potential to revolutionize student engagement in educational contexts. However, questions still remain around whether or not these tools are effective at facilitating student learning. In this research, we test the effectiveness of an AI-powered platform incorporating multiple representations and assessment through Learn Your Way, an experimental research platform that transforms textbook chapters into dynamic visual and audio representations. Through a between-subjects, mixed methods experiment with 60 US-based students, we demonstrate that students who used Learn Your Way had a more positive learning experience and had better learning outcomes compared to students learning the same content through a digital textbook. These findings indicate that AI-driven tools, capable of providing choice among interactive representations of content, constitute an effective and promising method for enhancing student learning.

Figures

Diagram of the Learn Your Way user interface for Immersive Text. Other learning modalities are featured at the top including Slides, Video, Audio Lesson, and Mindmap, as well as the original PDF.

Overview of the experimental set-up.

Violin and boxplots of immediate recall scores (left) and long-term recall scores (right) by tool condition. The immediate recall assessment was scored out of 12 points while the long-term recall assessment was scored out of 6 points.

Tables

Table 1. Duration of participant tool usage for Learn Your Way and Textbook.

Duration of tool use	Tool condition
	Learn Your Way	Textbook
Maximum time used (minutes and seconds)	40:50	32:49
Average time used (minutes and seconds)	30:49	22:05

Table 2. Participant demographics by educational tool condition, location, and grade level.

Tool condition	Location	Number of participants Grade level
		9th	10th	11th	12th	Grand total
Learn Your Way	Urban	1	3	4	2	10
	Suburban	0	2	7	2	11
	Rural	0	3	5	1	9
Total—Learn Your Way		1	8	16	5	30
Textbook	Urban	1	1	4	2	8
	Suburban	2	5	7	0	14
	Rural	0	4	3	1	8
Total—Textbook		3	10	14	3	30
Grand total		4	18	30	8	60

Table 3. Median (Mdn), interquartile range (IQR), and sample size (n) of immediate recall and long-term recall assessment scores by educational tool condition.

Tool condition	Immediate recall Mdn, IQR (n)	Long-term recall Mdn, IQR (n)
Learn Your Way	9.5, 3 (30)	5.0, 2 (29)
Textbook	8.0, 2 (30)	4.0, 2 (29)

Table 4. Summary of Wilcoxon-Mann-Whitney tests of immediate and long-term recall scores with directional hypothesis testing.

Directional hypothesis	Immediate recall Test statistics	Long-term recall Test statistics
Learn Your Way >Textbook	Z = 1.89	Z = 1.89
	p = 0.029*	p = 0.03*
	r = 0.24	r = 0.25

Table 5. Number of Learn Your Way participants who used one or more of the five core learning modalities provided in the tool.

Learn Your Way modality usage	Number of participants (Total n = 30)
Immersive Text + QuizMe	30
Slides	17
Mindmap	14
Audio lesson	6
Video	4

Table 6. Learn Your Way modality usage across participants and mean scores for the immediate recall assessment (n = 30; out of 12 points) and the long-term recall assessment (n = 29; out of 6 points).

Learn Your Way modality usage	Number of participants (total n = 30)	Immediate recall Mean (SD)	Long-term recall Mean (SD)
Immersive Text only	12	8.6 (1.8)	4.4 (1.3)
Immersive Text +1 modality	4	10.8 (1.9)	5.3 (0.5)
Immersive Text +2-4 modalities	14*	9.3 (2.4)	4.7 (1.3)

Table 7. Mean agreement ratings for learning experience statements shown by educational tool condition in middle two columns.

Learning experience statements	Agreement ratings		Test statistics
	Learn Your Way Mean (SD)	Textbook Mean (SD)	Z-score, p-value, r effect size
I felt like today's educational tool made me more comfortable taking the assessment.	4.60 (0.50)	3.83 (0.83)	Z = 3.77 padj<0.001***r = 0.49
The educational tool I used today would make me more effective at learning compared to other educational tools I currently use at home / school.	4.17 (0.79)	3.07 (0.98)	Z = 4.11 padj<0.001***r = 0.53
I would recommend this educational tool to other students to support their learning needs.	4.53 (0.78)	3.97 (0.89)	Z = 2.88 padj=0.004**r = 0.37
I found today's educational tool enjoyable to use.	4.27 (0.64)	3.67 (0.99)	Z = 2.44 padj=0.01*r = 0.31
I would like to use today's educational tool to support my learning needs in the future.	4.43 (0.73)	3.83 (1.05)	Z = 2.42 padj=0.01*r = 0.31
I felt like today's educational tool helped me gain a good understanding of the content.	4.50 (0.57)	4.07 (0.78)	Z = 2.28 padj=0.014*r = 0.29
I felt like I performed well on the assessment.	4.30 (0.70)	3.83 (0.95)	Z = 1.92 padj=0.027*r = 0.25

Full text

Introduction

1

Around the world, AI is being leveraged to transform lives and augment human capabilities in multiple industries, and education is no exception. Teachers, students, and other stakeholders in the industry have been known to leverage AI to support lesson planning and delivery, learning, administrative tasks, etc. (Łuczak et al., 2024).

With the continuous push toward digital equity and literacy over the years, there has been an increased interest in education technologies (particularly chat-based tools like ChatGPT1 and Gemini2) (Zhai et al., 2021; Zhang and Aslan, 2021; Adeshola and Adepoju, 2024; Memarian and Doleck, 2023; Li et al., 2025). Amongst other benefits, generative AI is uniquely poised to transform learning materials (including traditional textbooks) into more dynamic, engaging, and personalized content and to provide immediate feedback to improve understanding of learning materials (Łuczak et al., 2024).

As improvements are made in natural language processing and understanding, large language models (LLMs) show promise for generating effective learning resources at scale for diverse groups of learners. This goes beyond performance outcomes to other intrinsic factors such as interest and motivation in learning. For example, in a between-subjects study comparing pre-existing text to AI-generated output, Leong et al. (2024) show that while there was no difference in learning performance, AI-generated output positively affected learning motivation thereby facilitating positive downstream effects (including appreciating the relevance and utility of learning the material).

Research in education also shows that learners benefit from having information presented in multiple modalities or formats (Mayer, 2002). By presenting information in multimodal learning environments paired with guided activities, reflection, and feedback, learners benefit from numerous pedagogical advantages, including the promotion of independent learning and practice that is tailored to the needs of individual students (Moreno and Mayer, 2007).

The process of transforming traditional textbooks and learning materials into these more dynamic, engaging, and personalized sources can, however, be time consuming and labor intensive for teachers, thereby limiting the broader use of AI-powered education tools (Dennison et al., 2025). When considering direct and independent use by students for more personalized experiences at scale, several areas of concern emerge that demand careful consideration. These include the critical issues of accuracy, bias, academic integrity, factuality and reliability of model outputs, the potential for over-reliance on AI tools and its long-term impact on cognitive abilities and critical thinking skills, and students' inherent capacity to formulate effective prompts for facilitating their learning experience (Han et al., 2024; Park and Ahn, 2024; Tankelevitch et al., 2025; Chan and Lee, 2023; Denny et al., 2023). Furthermore, research has been mixed when it comes to demonstrating the efficacy of AI tools on performance outcomes (Do et al., 2025; Leong et al., 2024; Pallant et al., 2025).

In this paper, we build upon pedagogical best practices to experimentally test the impact of an AI-powered interactive learning platform on student understanding and recall of information. We investigate the following research questions:

RQ1 How does Learn Your Way, an experimental research platform that leverages generative AI to transform content (Wang et al., 2025), impact student comprehension and immediate recall of information?RQ2 How does Learn Your Way impact retention and recall of information over time?RQ3 Can Learn Your Way facilitate more engaging learning experiences for students?

To answer these questions, we conducted an experiment with 60 students aged 15-18 in the US, who were asked to learn a textbook chapter using a PDF reader or Learn Your Way. We then examined learning outcomes, in addition to quantitative metrics of user sentiment and qualitative feedback about their learning experience.

The results of the experiment suggest that AI-powered interactive learning platforms that incorporate content transformations and assessments, can facilitate positive learning outcomes and provide students with more engaging learning experiences compared to learning via traditional textbook chapters. We also demonstrate the positive role of timely feedback (as opposed to solutions) in equipping learners with knowledge and skills for self efficacy and academic independence.

Background and related work

2

HCI and education

2.1

The Human-Computer Interaction (HCI) community has a rich history of research exploring the design, development, and evaluation of educational technology (Stasko et al., 1993; Russell et al., 1993; Abowd et al., 1998; Scaife et al., 1997; Zuckerman et al., 2005; Saerbeck et al., 2010; Winkler et al., 2020). The discourse about generative AI in education has shifted from an initial period of denial and uncertainty to partial acceptance and explorations, ultimately leading to calls for further research (Pallant et al., 2025) and governance (UNICEF Innocenti, 2019). Recent HCI research has explored a wide range of important topics in this domain, including privacy and security considerations for technology use in schools (Kumar et al., 2019; Chanenson et al., 2023), extending participatory and co-design methods to the development of classroom-based technologies (Chang et al., 2024; Nicholson et al., 2022), and designing tactile-based learning experiences to foster accessibility in education (Melfi et al., 2020).

Additionally, research in HCI is beginning to explore how the increase in AI-powered capabilities can be incorporated into a variety of interactive experiences in order to positively impact learning. For example, Liu et al. (2024) created and evaluated an AI agent designed to interact with students in VR-based learning environments and promote classroom participation and more positive learning experiences, finding that these agents can provide pedagogical value through increased levels of engagement and the potential for learning gains. Chen et al. (2024) found that augmenting block-based programing software (Scratch) with an LLM-based chat experience allowed students to learn more independently, and create higher quality projects. This growing body of research suggests a strong desire from the HCI community to better understand how generative AI tools can help improve student outcomes.

AI assessment and feedback

2.2

One of the ways AI can help bring value to students is through AI-driven assessment and feedback. Pedagogical research shows that effective feedback is timely, actionable, grounded in standards or learning goals, and cites specific observations to facilitate learner understanding and support action (Wiggins, 2012). Providing feedback is an inherent part of a teacher's role, but as students partake in independent practice outside of classroom hours, feedback is not always readily available or accessible. However, the ability to access feedback is necessary to help students understand their performance and progress in relation to required standards and learning objectives (Hattie and Timperley, 2007).

From the early days of intelligent tutoring systems, AI-driven knowledge assessments—including personalized, adaptive feedback—has been a core area of research for education-focused AI (Anderson et al., 1985; Corbett et al., 1997; Graesser et al., 2004). With generative AI, students have the potential to access ongoing, targeted, timely, and individualized assessments and feedback at scale for self-directed learning and self-efficacy. Recent technological advances show improvements with the capabilities surrounding assessment and feedback (Choi and McClenen, 2020; Yang et al., 2022), but more improvements are needed to truly reach this vision (Tu et al., 2025).

In HCI, several qualitative studies with parents, teachers, and students have led to design recommendations that emphasize the value, importance, and opportunity for AI to provide students with high-quality assessments and feedback (Han et al., 2024; Prasad et al., 2025). A recent meta-analysis on emerging technologies in education also emphasizes the importance of HCI research that is grounded in pedagogical strategies and focuses on assessment and feedback (Van Mechelen et al., 2023). However, there is limited quantitative data and experimental studies evaluating how the design of AI-powered education tools can facilitate effective assessments and feedback. In this paper, we build upon prior work by incorporating design recommendations from past research into an experimental generative AI platform, Learn Your Way, and subsequently assessing the platform's impact on student learning experiences and performance outcomes.

AI-generated content transformations

2.3

By definition, content transformations transform static content into multiple representations to help facilitate learning, such as transforming text-based content into something that is audio- and/or visual-based (Ainsworth, 1999). Content transformations are an integrated process that isn't just changing the modality (i.e., text to audio), but also the structural reorganization of information that bridges the gap between raw information and a learner's cognitive architecture (Shulman, 1987; Koedinger et al., 2015). In contemporary educational technology and HCI research, there is a notable rise in the adoption of generative content transformation capabilities; these extend beyond simple modality shifts to include the structural transmutation of static information into interactive assessment tools and adaptive feedback mechanisms (Brusilovsky, 2001; Mitrovic, 2012). This trend reflects a broader shift toward dynamic, personalized learning resources that can be adapted to various student needs and learning preferences. The benefits of content transformations are guided by principles from Mayer's cognitive theory of multimedia learning, which states that people learn more deeply when information is received in multiple modalities as opposed to reading or hearing words alone (Mayer, 2002; Mayer and Moreno, 1998). For example, a study showed that children who listened to stories while also viewing relevant pictures recalled more information than those who only heard stories (Levin and Berry, 1980). This study (and others, e.g., Villegas-Ch et al., 2024) suggests that integrating materials across different modalities, such as audio paired with images or videos, can be particularly beneficial for student engagement and learning.

Generative AI capabilities, by their nature, create content transformations (text-to-text, text-to-audio, text-to-image, and text-to-video). By providing multiple modes of interaction and engagement with content, generative AI tools are well positioned to support self-directed learning, including theories that advocate for learners to play a central role in their own learning journeys, which has been shown to have positive impacts on learning outcomes (Deci and Ryan, 2000; Zimmerman and Moylan, 2009; Zimmerman, 2000). For example, this could include metacognitive strategies that allow for setting individual learning goals, choosing appropriate strategies to achieve these goals, and monitoring one's individual learning process and outcomes for adjustment and improvements (Panadero, 2017).

In HCI, the research community is beginning to explore the effects of AI-based content transformations on learning and education. In an experiment, Leong et al. found that a generative AI-based vocabulary app, that transformed standard vocabulary sentences into either personalized, generated sentences or personalized, generated text-based stories did not increase learning performance but did positively impact learning motivation (Leong et al., 2024). Similarly, in an experiment moving beyond text-to-text transformations, Do et al. found that transforming educational materials from text to podcast form led to a more enjoyable learning experience and also, significantly improved learning outcomes for certain subjects (Do et al., 2025).

Taken together, the latest research in HCI suggests that AI-based content transformations are poised to enhance learning experiences. However, more research is needed to understand the potential of AI in education by studying a broader range of the content transformation capabilities available through generative AI. Additionally, while a majority of prior HCI studies have explored model-driven interaction paradigms, there is also a notable gap in research that allows for user-driven experiences. This study addresses this gap through Learn Your Way, a system that extends beyond singular text and audio transformations, and provides learners with five different learning modalities that leverage multiple content transformations to create an interactive, engaging experience that enables user agency and self-direction, as well as real-time assessment and feedback.

Learn Your Way

3

Learn Your Way is an experimental platform that leverages generative AI to transform textbook PDFs into a variety of formats, or multiple representations (Wang et al., 2025). We used Learn Your Way to test our research hypotheses. Learn Your Way is not intended to be an educational product used in classrooms. Rather, it is a way for us to test how AI and pedagogical theories have the potential of impacting learning outcomes. These content transformations were grounded in pedagogical theories that demonstrate that multiple representations of content can be a powerful way to learn (Mayer and Gallini, 1990; Mayer and Moreno, 1998). Learn Your Way also incorporates learner control over format selection, a feature theoretically grounded in Self-Determination Theory (SDT) (Ryan and Deci, 2000). By enabling this choice, we aim to augment the students' perceived autonomy, a known precursor to enhancing intrinsic motivation, cognitive effort allocation, task performance, and self-efficacy (or perceived competence) (Reber et al., 2009; Schneider et al., 2018a; Patall et al., 2008).

Learn Your Way presents five learning modalities: Immersive Text, Slides, Video, Audio Lesson, and Mindmap (Figure 1). See Supplementary material for a visual example of each modality.

Immersive Text: Transforms the original content into smaller sections releveled to a selected grade, and provides opportunities for interactive learning and practice (e.g. memory aids, quizzes, embedded questions, and timelines). These transformations of chunking and scaffolding are not just “extra features,” but mechanisms by which AI generates “dynamic” content rather than “static”. Within Immersive Text, students had access to the following features:

• ○ Section-level quizzes (QuizMe): Interactively quizzes the learner to reinforce and assess what was just learned and uncover existing knowledge gaps.
• ○ Embedded questions: Short, in-line questions to check for understanding while the learner is reading.
• ○ AI-generated images and summary (Enimate): Briefly explains key concepts through visuals and short descriptions.
• ○ Timeline: AI-generated visualization of the sequence of key concepts in the original content.
• ○ Memory aids: AI-generated mnemonic that assists in memorization and recall of key concepts in the original content. Slides: Comprehensive presentations that cover the entire source material and include reflection prompts like fill-in-the-blanks.Video (slide deck with narration): Provides the option for a fully narrated slideshow mimicking a recorded lesson.Audio Lesson: Simulated conversations between an AI-powered teacher and student that models how a real learner might engage with the material. This virtual student asks questions and can even express common misconceptions which the teacher clarifies. The conversation is accompanied by visual diagrams to further illustrate the concepts.Mindmap: Organizes the knowledge hierarchically into an interactive diagram that allows the learner to zoom in and out from the big picture to the minute details.

Diagram of the Learn Your Way user interface for Immersive Text. Other learning modalities are featured at the top including Slides, Video, Audio Lesson, and Mindmap, as well as the original PDF.

When students open Learn Your Way, they start by selecting their grade level and interests (see Appendix 1). Learn Your Way then personalizes and adapts the content being generated by adjusting for grade level and incorporating the student's selected interests. As students use Learn Your Way, they engage with interactive quizzes designed to promote active learning and assess real-time understanding of the source material, enabling further elements of personalization. Students receive dynamic feedback on their answers, can track their overall progress, and are guided to review specific content areas where they demonstrated gaps in understanding. Students are also able to view the original content, in PDF form, at any time.

Learn your way design and evaluation

3.1

Throughout the design of Learn Your Way, each modality was evaluated by pedagogical experts to ensure the fidelity and quality of each content transformation while preserving the integrity of the source content. A unified rubric was developed to assess key pedagogical criteria, including accuracy (the extent to which the AI faithfully represented the source material), coverage (the thoroughness with which all key concepts were addressed), and learning science principles. The results indicated a high level of quality, with all modalities and features achieving an average expert rating of 0.9 or higher on a scale of 0 to 1 across all pedagogical criteria. Detailed information about the rubric and results of the evaluation can be found in Supplementary material.

Experimental study

4

We conducted an experimental study to compare performance outcomes and learning experiences between the AI-powered learning platform (Learn Your Way) and a digital reading of an original textbook chapter (Textbook) as a baseline comparison, consistent with established baseline comparisons employed in the literature (Do et al., 2025).

The source content used in the experiment was a neuroscience textbook chapter titled, “Brain Development During Adolescence” (Learning, 2019). The source material was chosen by a panel of pedagogy experts using a set of criteria based on education psychology (see Supplementary material).

Research hypotheses

4.1

We present the following research hypotheses:

H1: We hypothesize that participants will perform better on both an immediate and long-term recall assessment in the Learn Your Way condition compared to the Textbook condition, drawing on prior research demonstrating that transforming static content into different modalities and self-directed learning can have a positive effect on learning outcomes (Ainsworth, 1999; Do et al., 2025; Deci and Ryan, 2000).H2: Consistent with research that demonstrates that students experience enjoyment from AI-generated tools (Abdaljaleel et al., 2024; Do et al., 2022, 2025; Leong et al., 2024), we hypothesize that participants in the Learn Your Way condition will report more positive learning experiences compared to those in the Textbook condition.

Dependent variables

4.2

All assessments and questionnaires were programmed and administered through Qualtrics.3 Full question banks can be found in the Supplementary material.

Learning outcomes

4.2.1

To assess immediate recall and comprehension, participants completed a ~10 min, in-session assessment that consisted of short answer questions (SAQs), single-answer multiple choice questions (SMCQs), multi-answer multiple choice questions (MMCQs), and matching questions. The immediate recall assessment was scored out of 12 points.

To assess long-term recall, participants were sent a brief follow-up assessment three days post-session that consisted of one short answer question, one single-answer multiple choice question, and one matching question designed to reassess the learning outcomes targeted by the initial, immediate assessment. The long-term recall assessment was scored out of 6 points.

Both assessments were developed by pedagogical experts and targeted specific levels of Bloom's Taxonomy in order to ensure cognitive complexity (Bloom, 1971). The assessments were therefore designed to be a good measure of content comprehension and long-term knowledge consolidation (see Supplementary material for all assessment questions).

In order to establish consistency of SAQ scoring across both assessments, one of our pedagogy experts pre-defined rubrics with scoring criteria and examples for each SAQ on a 3-point scale. All SAQ responses were evaluated solely on students' demonstrated knowledge of the source material (ranked from 0-Fragmentary to 3-Demonstrating), excluding any consideration of students' grammar skills or writing proficiency in this evaluation (see Supplementary material for complete rubrics). Our pedagogy expert and a member of the research team independently scored and compared results for one third of the data to ensure alignment and adherence to the rubric guidelines. Both were blind to the experimental condition. The remainder of SAQ scoring was completed independently by the study's pedagogy expert.

Learning experience

4.2.2

Following the immediate recall assessment, participants responded to a series of survey questions that assessed their experiences recalling the content and using the educational tool. These questions included ratings of participants' familiarity and interest in the topic, perceived ease/difficulty of the assessment, and the utility the tool provided in helping the participant learn the content. We also asked a series of agreement statements that evaluated the tool's effectiveness, desire to use in the future, and enjoyment.

Recruitment

4.3

Participants were recruited through a research vendor (Bold Insight4). During recruitment, participants completed a screener survey that collected information on their age, grade level, GPA, standardized test scores, and relevant academic courses taken.

In order to evaluate potential biasing effects of prior familiarity with the learning material on participant feedback and assessment scores, we had participants rate their familiarity across 15 age-appropriate topics, including the topic of the chapter featured in the experiment—adolescent brain development. Across participants, baseline familiarity with the selected learning topic was highly clustered around the mid-point of the scale (Mdn = 3, M = 2.7, SD = 1.2) suggesting partial but not high familiarity with the learning material. Furthermore, after random assignment, baseline content familiarity was approximately matched across experimental conditions.

To ensure students participating in the study had similar levels of reading comprehension, they read a short passage and completed a reading comprehension task during the recruitment process. The average score on the assessment was 6.4 out of 10, with a standard deviation of 2.3. We included students who scored one standard deviation above or below the mean, or those who scored a 4–9 out of 10 on the reading assessment.

Procedure

4.4

The experiment consisted of a single 90 min in-person lab session and a follow-up assessment shared remotely with participants three days post-session. See Figure 2 for an overview of the experimental setup. The study was conducted in adherence to Google and Bold Insight's ethical, legal, and privacy standards for human subjects research, and our participants completed an informed consent process.

Overview of the experimental set-up.

At the beginning of the study, participants were randomly assigned to one of the two experimental conditions (Learn Your Way or Textbook) and were given a brief overview (three slides) that introduced the educational tool they would be using during the task (see Supplementary material). They were then asked to imagine they were sitting in one of their classrooms at school where their teacher asked them to learn the educational content using the provided tool and complete an assessment based on what they learned. Participants in the Learn Your Way condition selected a preloaded PDF of the learning material and indicated the grade level they wanted to learn the content in, as well as their interests (e.g., music, video games, biking, etc). All participants in the Learn Your Way condition started in the Immersive Text modality, and therefore interacted with the QuizMe feature. Participants in the Textbook condition viewed the chapter in a web-based PDF reader. To ensure uniformity across conditions, all participants were given up to 40 min (but required to spend at least 20 min) using the tool and learning the content before taking a short assessment. We report the maximum and average learning duration by educational tool condition in Table 1.

After the allotted 40 min (or when the participant indicated they were ready to move on), participants completed the immediate assessment (~10 min) and the learning experience questionnaire (~5 min) via a digital survey. Finally, they spent 30 min in a qualitative, debrief interview with a skilled moderator. In these in-depth interviews, participants were asked questions about their experience using the educational tool, utility of the tool and corresponding capabilities, experience with the assessment, recommendations for improving the tool, and how the tools compare to current tools they use for learning.

Three days post-session, participants were emailed a follow-up assessment that measured long-term recall of the content they had learned during the experiment. 58 out of 60 participants completed the long-term recall assessment within the required 3–7 day window. Participants were compensated between $115–$160, accounting for study time, travel time, and completion of the follow-up survey.

Participants

4.5

We recruited 60 high school students from the Chicago area in the United States. We intentionally recruited students from rural (n = 17), urban (n = 18), and suburban (n = 25) areas to ensure diversity in our participant pool. 52% of participants were male, 47% female, and 1% non-binary. All participants were pre-screened to ensure they had similar levels of reading comprehension and did not require additional test-taking accommodations.

Participants were randomly assigned to one of two educational tool conditions: Learn Your Way (treatment condition) or Textbook (control condition). Table 2 shows a breakdown of participants' demographics across conditions.

Data analysis and approach

4.6

To address our research questions, we analyzed the collected data at multiple levels. All data and statistical analyses were conducted in R and RStudio including the statistical package coin for our non-parametric tests (Hothorn et al., 2008) and package mgcv for the generalized additive models (Wood, 2017).

First, to inform the statistical method for immediate recall and long-term recall, we conducted a Shapiro-Wilk test on each dependent variable to check for the parametric assumption of a normal distribution (Shapiro and Wilk, 1965). We addressed any violations of this assumption by conducting a non-parametric alternative to the independent t-test, the Wilcoxon-Mann-Whitney test. Specifically, we conducted one-sided Wilcoxon-Mann-Whitney tests on both immediate recall assessment scores and long-term recall assessment scores, in line with our first research hypothesis (H1). To address our second hypothesis (H2), stating that Learn Your Way participants will report more positive learning experiences than Textbook participants, we conducted directional Wilcoxon-Mann-Whitney tests for each of our learning experience metrics. Additionally, to control for the inflation of Type I error rate due to multiple comparisons, p-values were adjusted for the learning experience metrics using the Benjamini-Hochberg procedure to maintain a false discovery rate (FDR) of 0.05 (Benjamini and Hochberg, 1995). For all statistical tests, we report the Z-statistic, p-value, adjusted p-value (where applicable), and subsequent effect size r.

Second, as an exploratory analysis, we investigated immediate recall and long-term recall performance for Learn Your Way participants as a function of learning modality usage. We report mean values and directional usage patterns for three levels of modality usage: Immersive Text only, Immersive Text plus one additional modality, and Immersive Text plus 2 or more learning modalities utilized by Learn Your Way participants. To increase sample and our subsequent power to detect differences, we combined the latter two usage types for our statistical analyses including two-sided Wilcoxon-Mann-Whitney tests and Pearson's correlation tests on both immediate recall and long-term recall scores.

In addition, we conducted several bias checks to assess whether extraneous factors may be driving the primary effects we observed. To explore whether students' prior reading comprehension levels influenced our learning outcomes, we compared screener assessment scores for Learn Your Way participants to those for Textbook participants through a two-sided Wilcoxon-Mann-Whitney test on a reading comprehension task (eligible scores for participation ranging from 4–9 points). Next, to ensure the learning material struck a balance of being relatively unfamiliar to participants (as to not bias their recall scores), but still interesting enough to encourage participant engagement in the task, we had them rate their level of interest and familiarity with the content. Participants' baseline familiarity was evaluated in two ways: first, through a screener questionnaire and again, during the in-session survey. Both content familiarity and interest ratings were analyzed using two-sided Wilcoxon-Mann-Whitney tests, appropriately suited for ordinal data. For our final bias check, we conducted a two-sided Wilcoxon-Mann-Whitney test on students' perceived assessment difficulty, rated on a standard five-point scale from “very difficult” to “very easy”.

To further ensure that the observed learning outcomes were not driven by procedural confounds, we modeled the influence of learning duration (transformed into seconds) and delay duration (within the 3–7 day window) on assessment scores. Specifically, we employed semi-parametric generalized additive models (GAMs) to (1) allow for potential non-linear dependencies between learning duration and recall performance, and (2) realistically model the non-linear memory decay typically associated with the forgetting curve (Ebbinghaus, 1913). Within these models, smoothing terms were applied to both continuous predictors—learning duration and delay duration—nested by experimental condition, to allow the models to optimally fit both linear and non-linear trends. The complexity of these smooth terms was evaluated using effective degrees of freedom (edf), where edf = 1 corresponds to a linear relationship and edf>1 indicates increasing non-linearity (Wood, 2017).

Finally, for our qualitative interviews, we applied Braun and Clarke's thematic analysis method (Braun and Clarke, 2021) to generate themes that helped us understand the key patterns observed in the quantitative data in more depth. We chose this approach because it is widely used in other HCI interview-based studies (Johnson et al., 2022; Wong et al., 2025). A data collection tool was created in Google Sheets to capture participant behavior and interview data. During each session, the moderator used a paper guide to facilitate the interview and take notes while a notetaker entered participant responses in the data collection tool. Post-session, the moderator and notetaker reviewed the study data for accuracy and completeness. As needed, session videos and transcripts were also reviewed to confirm study data accuracy. The study team debriefed daily to review, discuss, and refine ongoing themes from the interviews. Each week, the team also met to discuss the study data and further align on trends and themes observed. The team met to refine the analysis until data saturation was achieved (Hennink et al., 2017) resulting in a final set of themes focused on differences in students' behavior, attitudes, and reflections on their learning experiences.

Results

5

Bias checks

5.1

A Shapiro-Wilk test confirmed a significant departure from normality for reading comprehension scores (N = 60, W = 0.93, p = 0.002). We therefore conducted a non-parametric Wilcoxon-Mann-Whitney test and found no significant difference in scores between Learn Your Way (Mdn = 7;M = 6.5, SD = 1.4) and Textbook students (Mdn = 7;M = 6.7, SD = 1.4) on the reading comprehension task (Z = 0.54, p = 0.588, r = 0.07).

In addition, we explored whether prior familiarity with the learning material differed by experimental condition for either the screener ratings or in-session ratings of familiarity. We found no significant difference in baseline familiarity with the learning material between Learn Your Way (Mdn = 2.5; M = 2.6, SD = 1.1) and Textbook conditions (Mdn = 3; M = 2.9, SD = 1.2) for screener ratings of familiarity (Z = 0.98, p = 0.337, r = 0.13). Additionally, no significant difference was detected between the Learn Your Way condition (Mdn = 3; M = 2.6, SD = 0.9) or the Textbook condition (Mdn = 2.5; M = 2.4, SD = 0.9) for content familiarity assessed during the in-session questionnaire (Z = 0.52, p = 0.616, r = 0.07). We also found no significant difference between the Learn Your Way (Mdn = 3; M = 3.3, SD = 0.6) and Textbook conditions (Mdn = 3; M = 3.2, SD = 0.8) in terms of students' reported interest in the chapter topic (Z = 0.11, p = 0.934, r = 0.01). In fact, the majority of participants, regardless of experimental condition, rated the learning content as “somewhat interesting” or “very interesting” on a 4-point Likert scale.

Lastly, we evaluated students' perceived assessment difficulty, as it was important for the immediate recall assessment to feel like a reasonable task for high school students - in that it was not too difficult nor too easy. We found no significant difference in student ratings of assessment ease or difficulty between the Learn Your Way condition (Mdn = 4; M = 3.6, SD = 0.6) and the Textbook condition (Mdn = 3; M = 3.4, SD = 0.8) after conducting a two-sided Wilcoxon-Mann-Whitney test (Z = 1.15, p = 0.268, r = 0.15).

These bias checks confirm that students' reading comprehension levels, familiarity and interest in the topic, and perceptions of assessment difficulty did not meaningfully differ between experimental conditions, thereby providing no differential benefit to learning outcomes or learning experiences.

Learning outcomes

5.2

In support of our first hypothesis (H1), we found that students in the Learn Your Way condition (Mdn = 9.5; M = 9.2, SD = 2.2) performed significantly better on the immediate recall assessment compared to students in the Textbook condition (Mdn = 8; M = 8.2, SD = 2).

In addition, analysis of long-term recall scores provides further support for H1, with Learn Your Way participants outperforming Textbook participants on the 3-7 day post-session assessment. Students in the Learn Your Way condition (Mdn = 5; M = 4.7, SD = 1.2) scored significantly higher on the long-term recall assessment than students in the Textbook condition (Mdn = 4; M = 4, SD = 1.4).

In Table 3, we further breakdown immediate and long-term recall performance, including median scores and interquartile range (IQR), and in Table 4, we show the results of statistical tests performed. Finally, we use violin and box plots to represent the overall score distributions for Learn Your Way and Textbook in Figure 3.

Violin and boxplots of immediate recall scores (left) and long-term recall scores (right) by tool condition. The immediate recall assessment was scored out of 12 points while the long-term recall assessment was scored out of 6 points.

To ensure that performance outcomes were not driven by the duration of time students spent learning the content, we modeled the relationship between educational tool condition and learning duration on recall scores for both immediate and long-term recall assessments. Specifically, we employed a generalized additive model (GAM) to account for potential non-linear relationships between learning duration and assessment performance.

For immediate recall scores, the model explained 13.4% of the total deviance. The non-parametric smooth terms for learning duration were not significant for either Learn Your Way (edf = 1, F = 0.48, p = 0.49) or Textbook (edf = 1.82, F = 1.43, p = 0.283). Similar results were observed for long-term recall scores, where the model explained 9.1% of the total deviance. Again, the smooth terms for learning duration were not significant for Learn Your Way (edf = 1, F = 1.53, p = 0.222) or the Textbook condition (edf = 1, F = 0.15, p = 0.703). Collectively, these findings suggest that learning duration did not meaningfully influence performance outcomes on either the immediate or long-term recall assessment, regardless of the experimental condition.

Finally, to account for variations in the retention interval across students, we employed another GAM to model the relationship between length of delay (3–7 days) and long-term recall performance. The model explained 8.2% of the total deviance. The non-parametric smooth terms for delay duration were not significant for Learn Your Way (edf = 1, F = 1.09, p = 0.3) or Textbook (edf = 1, F = 0.02, p = 0.903), demonstrating that the specific length of delay did not significantly influence long-term recall within our required follow-up window of 3–7 days.

Regardless of experimental condition, assessment performance remained remarkably stable across time. Specifically, the Learn Your Way participants achieved nearly identical percentage scores on the immediate recall assessment (M = 9.2/12; 77%) and the long-term recall assessment (M = 4.7/6; 78%). Similarly, Textbook participants demonstrated a slight decrease of only one percentage point from immediate recall (M = 8.2/12; 68%) to long-term recall (M = 4/6; 67%). These results suggest that students retained the vast majority of acquired information, with negligible decay occurring across the 3–7 day retention interval. Consequently, the performance advantage observed for Learn Your Way during the immediate assessment was maintained through the long-term recall phase.

As an exploratory analysis, we also investigated whether or not the use of more learning modalities, and thus, more content transformation capabilities, in the Learn Your Way condition had an effect on students' immediate and long-term recall performance. Table 5 shows the different learning modalities that Learn Your Way participants used during the learning phase. As a reminder, all participants landed in Immersive Text by default and 100% of students utilized the QuizMe feature during their learning session.

We further split Learn Your Way participants into students that used Immersive Text only, students that used Immersive Text plus one additional learning modality and students that used Immersive Text plus two or more learning modalities (Table 6). For the following analysis, we collapsed between the latter two groups when comparing immediate assessment scores between Immersive Text + QuizMe only (n = 12; Mdn = 8.5; M = 8.6, SD = 1.8) vs. Immersive Text + any additional modality(ies) (n = 18; Mdn = 10; M = 9.6, SD = 2.3). A two-sided Wilcoxon-Mann-Whitney test revealed no significant difference between groups (Z = 1.48, p = 0.144, r = 0.27). Furthermore, while we see, directionally, that scores are higher for students that used more modalities in Learn Your Way, we did not find a significant correlatio n between learning modality usage and performance on the immediate recall assessment (Pearson's r = 0.24, p = 0.206). In other words, we did not find evidence that students who used more content transformation modalities had a better performance on the immediate assessment. This suggests that the pedagogical value of the system may reside in the quality of engagement with a chosen modality rather than the breadth of modalities accessed.

Similarly, we did not find a significant difference between Immersive Text + QuizMe only (n = 12; Mdn = 4.5; M = 4.4, SD = 1.3) vs. Immersive Text + any additional modality(ies) (n = 17; Mdn = 5; M = 4.8, SD = 1.1) for long-term recall scores (Z = 0.87, p = 0.396, r = 0.16) nor did we find a significant correlation (Pearson's r = 0.17, p = 0.380).

For both exploratory analyses reported above, we acknowledge that statistical power to assess between-group differences and correlations was lower than our primary analyses, due to splitting our sample size for Learn Your Way into two groups (see Section 7 for more details).

Learning Experience

5.3

In support of our second hypothesis (H2), we observed meaningful differences when we analyzed students' reported experience with the educational tools. Students in the Learn Your Way condition (Mdn = 4; M = 3.6, SD = 0.6) found the tool to be more useful for learning the educational content compared to Textbook students (Mdn = 3; M = 3.1, SD = 0.7). Specifically, a one-sided Wilcoxon-Mann-Whitney test revealed that Learn Your Way participants rated the educational tool significantly higher on a 4-point Likert scale than Textbook participants, before and after correcting for multiple comparisons with other experience metrics (Z = 3.08, p = 0.001, padj = 0.003, r = 0.40).

Furthermore, through a collection of agreement statements about the educational tool, we can see how the Learn Your Way platform positively impacted students' perceived learning experience as compared to students in the Textbook condition (as shown in Table 7). Specifically, students in the Learn Your Way condition reported higher levels of agreement compared to students in the Textbook condition across all of our experiential measures. Directional Wilcoxon-Mann-Whitney tests were conducted on each statement's agreement ratings, followed by FDR corrections for multiple comparisons. Table 7 provides a full breakdown of the included statements and mean values by educational tool condition, as well as the associated test statistics with the adjusted p-values reported. The most significant differences revealed between the Learn Your Way platform and the Textbook condition suggest Learn Your Way has the strongest influence on students' comfort in test-taking, and their perception of the tool's effectiveness for learning compared to other educational tools already being used at home or in their classroom.

Qualitative findings

6

Data from the qualitative interviews shed light on the trends we observed in the quantitative data.

RQ1 and RQ2: learning outcomes

6.1

Dynamic representations of content can effectively help students learn

6.1.1

One of the unique attributes of Learn Your Way is that it provides multiple interactive transformations in one learning experience. This approach is based on established learning principles, such as Mayer and Moreno's “Cognitive Theory of Multimedia Learning” (Mayer and Moreno, 1998) and Paivio's “Mental representations: A dual coding approach,” (Paivio, 1990) which posit that encoding information in multiple formats can make learning more effective by engaging multiple senses and cognitive channels, and by simplifying complex ideas. Students in the present study expressed value in the ability to access multiple learning modalities. These were perceived as useful for clarifying concepts and improving comprehension and retention of the learning material:

“Reading it multiple times but in different ways made it stick in my mind. In the assessment, some topics I remembered from the slide, some I remembered from reading. So, reading it multiple times but in different ways made it stick in my mind.”- P86, Urban, Learn Your Way“If I didn't feel I was getting good information from one [modality], I could go to the next one… Immersive Text and the slides are really good for a study tool especially. Mindmap helped me put everything together and really see where everything fits in here… I think because it's the same information, you just kind of can play around with it and go to different things like Slides and stuff. I think that, in itself, helped me to remember [the material] and see what's really going on.”- P20, Suburban, Learn Your Way

Conversely, our qualitative feedback revealed that participants from the Textbook condition desired the ability to view content in multiple formats. Despite being unaware of the Learn Your Way condition, several Textbook participants described features already included in Learn Your Way when asked what their ideal learning tool would look like:

“I'm more of a visual learner, so if I see something visual, I better understand it than words or text.”- P09, Urban, Textbook

Students in both conditions highly valued and preferred interactive and multimodal learning materials, as evidenced by their feedback. This approach felt like an important way to create effective and engaging learning experiences. Instead of reading passively through lengthy passages, students using Learn Your Way absorbed content in diverse ways that aligned with their preferences, which was perceived to facilitate a deeper understanding and better retention of the content.

Summaries and smaller chunks of text via Immersive Text in Learn Your Way improved students' comprehension of the content

6.1.2

A key feature of the Immersive Text modality in Learn Your Way was that it broke down the chapter into manageable chunks of information, and generated interactive practice and quizzes to test the level of comprehension of each section. Given that participants in the experimental condition spent most of their time utilizing this feature, its effectiveness likely played a role in the learning outcomes we observed. Many students spoke about the ways Immersive Text helped them digest and learn the content:

“I do feel the way that they structured it—having only a few paragraphs or split up text definitely helps my brain not get overwhelmed by the whole thing. So not just putting a whole chunk of text that is relating to the whole topic and then just being like, ‘okay here.”'- P40, Suburban, Learn Your Way“Like anything in my life, if I see something I have to tackle…and it's one big thing—I struggle breaking it up into smaller parts. If it's reading a really long article for a class or cleaning my room, I look at it and I'm like, ‘whoa, that's a lot of stuff. I can't do it.' If it's broken up for me, I feel less overwhelmed by it.”- P13, Suburban, Learn Your Way

In contrast, some Textbook students felt overwhelmed by having only the chapter's content, which made it harder to focus and retain information. We also observed that some students in this condition attempted to create summaries and break the content up on their own by using the annotation tools within the digital textbook.

“Because even though the paragraphs aren't that big, there's still enough paragraphs where you'll forget your information.”- P54, Rural, Textbook“Maybe if it stated the key points a little better because only about half of the article has like an actual head headliner, you know, forgot what they're called…so yeah, I mean maybe just like breaking down the big chunks of text a little better.”-P94, Suburban, Textbook

In summary, the qualitative feedback from students in both conditions reveal a preference for content presented in more manageable segments. In the absence of these summarized chunks of information, students expressed difficulty retaining the information from the textbook chapter, which likely impacted their ability to recall the information. This highlights the need to consider the impact of visual presentation and organization in the design of content transformation tools.

The value of assessments via QuizMe was a key driver in performance outcomes

6.1.3

One of the helpful ways students effectively learned the material in Learn Your Way was through the QuizMe feature. All participants in Learn Your Way acknowledged the role of quizzes and feedback on their performance. This is consistent with previous research that demonstrates that quizzes can significantly enhance learning outcomes by offering additional practice exercises in areas where a student is struggling (Chang and Chien, 2024). This approach helps learners attain mastery at their own pace and speeds up the learning process (Al-Badi et al., 2022; Kulik and Fletcher, 2016). Quizzes were a favored way to test comprehension of the material, and feedback instilled confidence in learning by increasing perceptions of knowledge retention for students in the Learn Your Way condition.

“The quizzes for sure helped me get the right answers. It helped me even if I needed help, it gave a hint if I didn't understand it. I feel like it also made it really easy to explain or understand.”- P20, Urban, Learn Your Way“I liked how it gave me feedback and told me why I got it right versus just bringing me to the next question right away. It was helpful and made me realize how my thought process was and how I got to the answer.”- P11, Suburban, Learn Your Way

In contrast, students in the Textbook condition reported missing the ability to check their understanding and receive feedback through guided activities and tools such as quizzes, flashcards and look up tools:

“I would probably make a tool where everything is together, like the test taking with the text so I can select what I want to learn more about in the text and put it in a test where I'm able to answer the questions if I know it goes away, if I don't I have to keep studying it and learn more.”- P12, Suburban, Textbook“Like the ideal learning tool, like if I were to learn this article a different way, I mean, yeah, like I said, I mean flashcards I find useful. I definitely think once you read it, once you like read a certain amount of it or whatever, like certain checkpoints I guess it could quiz you on what you've done so far to review to make sure that you got all the key points.”- P94, Suburban, Textbook

Feedback from participants aligns with educational theories about active engagement in learning and emphasizes the need to provide active checkpoints in learning experiences (Moreno and Mayer, 2000). This approach allows students to leverage metacognitive strategies to reflect on their understanding of the learning material and increase the likelihood of retaining information (Zimmerman and Moylan, 2009).

RQ3: learning experiences

6.2

The variety of capabilities and self-directed nature of Learn Your Way played a critical role in creating a positive learning experience

6.2.1

Access to multiple transformations and the freedom to choose which ones to learn with resulted in an engaging learning experience for students in the Learn Your Way condition. This is also consistent with research that advocates for learners to play a central role in their own learning journeys (Deci and Ryan, 2000; Zimmerman and Moylan, 2009; Zimmerman, 2000).

“I personally like going in my own structured way… It's just a learning preference… I like it more of me leading it because I feel more in control of my learning…it allows me to go my own pace and let me learn how I want to learn.”- P08, Urban, Learn Your Way“I like how many options it has for different ways of taking in the information because I definitely like learning it one way and then being able to pick another way to review…”- P13, Suburban, Learn Your Way

Conversely, students in the Textbook condition who did not have access to transformations, noted the challenges of this learning experience and desired more interactions.

“I guess I liked how it was kind of just all there and I could read it. What I didn't like…it was just like kind of boring reading it… I was just like sitting there trying to memorize it. Reading it…I usually learn best like videos and stuff. I can't like read stuff and store in memory… I guess I'm like more of like a visual learner.”- P19, Suburban, Textbook

Feedback from students in both conditions demonstrates the opportunity to facilitate student-led learning through the provision of multiple content transformations within a single learning experience.

Auditory transformations were perceived as useful for combining learning with other activities

6.2.2

While a small percentage (n = 6, 20%) of Learn Your Way participants used Audio or Video modalities, several of them recognized it as a way to multitask and incorporate learning into their daily routines. Because audio content does not require much physical interaction, students liked the idea of completing less cognitively demanding tasks while listening to educational content. They explained that it would help to use their time more efficiently and help them to feel more productive while completing routine tasks.

“I do listen to like a lot of podcasts in general, but I always like to have like, stuff playing while I'm doing stuff. So like, like I was saying before, like, oh, if you're like cooking or cleaning your room or just doing anything really like, I don't know, like even like hobbies, like you're like drawing or something. Like just having, I always like to have something playing in the background. So like when I have the option to have that for like school stuff, I just like have, I feel it feels more productive. Like I can get it done while I'm doing other stuff, while like retaining the information as well.”- P86, Urban, Learn Your Way“I mean it's the same with the Video I guess. It's like, like people talking rather than somebody explaining they're, I mean, in my mind pretty similar in use case. Like if I'm driving or like, you know, cleaning or doing something where I can't like read and I can just throw this on and learn about it. That would be the use case.”- P01, Urban, Learn Your Way

The Textbook condition, on the other hand, required more visual focus to process the content in the textbook chapter. Without access to text-to-speech tools or additional support, multitasking within the digital textbook would therefore be challenging. It is worth noting however that there are open questions about the impact of multitasking on the comprehension and recall of information and which tasks are suitable for this (Coens et al., 2011). It is therefore crucial to consider approaches to support metacognitive strategies (e.g., reflection) with auditory modalities to ensure effective learning.

Discussion

7

Our results are in support of our hypotheses, which demonstrate that Learn Your Way can lead to both improved learning outcomes and experiences. AI-powered learning platforms can be a powerful driver of pedagogical innovation. In particular, our experiment reveals that these tools are not merely supplementary aids but can play a critical role in reshaping how students interact with and comprehend academic content. While many studies exploring the impact of AI tools on educational outcomes have been mixed (e.g., Do et al., 2025; Sun et al., 2024), our research demonstrates that Learn Your Way can enhance students' immediate and long-term recall of information and create a more effective learning environment compared to a traditional textbook.

The experimental results suggest that this can be attributed to several factors. First, learning is a complex and multifaceted process and students exhibit diverse preferences in how they process information and acquire knowledge, including visual, kinesthetic, and reading/writing approaches (MCcall, 2024). Traditional instructional methods often fail to address these unique needs at scale, leading to suboptimal educational experiences (Villegas-Ch et al., 2024). Instead, engaging with information through multiple representational formats, such as slide decks, audio lessons, or visual mind-maps, facilitates the construction of a more robust and comprehensive mental model of the content (Ainsworth, 1999). This is because diverse representations can clarify complex ideas and highlight distinct facets of a concept. Learn Your Way addresses this need by offering students the ability to select their preferred learning modalities, enabling them to explore concepts from a range of engaging and diverse formats. Second, simplified and multimodal presentations can reinforce learning, leading to greater retention when information is seen, heard, and actively engaged with (Clark and Paivio, 1991). By breaking down content into manageable chunks and allowing students to personalize the presentation of the content in dynamic formats based on their learning preferences, we believe students forged better mental connections between different representations, resulting in better comprehension of the content compared to students who only engaged with a digital textbook.

Drawing from established literature on the benefits of adaptive quizzing and personalized feedback (Frank, 2024), this study's quantitative and qualitative findings underscore their critical role in the learning experience. All students who used Learn Your Way experienced quizzing and feedback. Participants' qualitative feedback further highlighted the value of these capabilities, with many emphasizing how quizzes aided in testing comprehension and retention, while the associated feedback was instrumental in helping them understand the rationale behind their answers.

Given the power of content transformation capabilities on student learning, one might wonder why there was no direct correlation between engagement with usage of more content transformation modalities and better learning outcomes among students who used Learn Your Way. This analysis was exploratory and the pattern of data is in the direction we expected (i.e., more use of Learn Your Way modalities was correlated with better learning outcomes) but not statistically significant. We expect this may have occurred for two reasons. First, this analysis required us to segment our Learn Your Way students into two different groups, which significantly reduced our sample size and therefore power to detect statistical significance. Second, our qualitative feedback suggests that the usage of additional modalities, such as Slides and Video were often treated as supplementary resources for content review, following primary engagement with the Immersive Text and QuizMe features.

“I feel like [Slides] would be good for review after I already read the Immersive Text if I wanted to review the main points.” - P72, Rural, Learn Your Way

Our findings therefore suggest two potential explanations for the observed phenomena. First, a larger sample size could provide a more robust basis for investigating whether increased use of content transformation capabilities positively impacts learning outcomes. Second, it may be the case that a greater quantity of engagement with content transformation features does not, in itself, lead to improved learning outcomes. Rather, our results corroborate previous research demonstrating that enhanced learning is contingent upon the strategic alignment of an individual's learning preferences with the specific demands of the task (Cordova and Lepper, 1996; Schneider et al., 2018b). In other words, the observed benefits are likely driven by the availability of multimodal options and the resulting learner agency, rather than a cumulative effect of exposure to multiple formats. As our current design did not manipulate modality exposure experimentally, we refrain from implying that using a higher number of transformations leads to superior outcomes. Instead, the results suggest that providing a diverse “menu” of representations allows students to select the single most effective format for their individual needs. While these tools offer significant value, their effectiveness is maximized when learners have the autonomy to select the specific transformations that align with their personal learning goals and contextual needs. Crucially, our data indicates that effectiveness is not predicated on the number of modalities utilized, but rather on the student's ability to find a representation that aligns with their specific learning needs.

Furthermore, it is important to note that Learn Your Way is not predicated on the efficacy of a single content representation, but rather on the theoretical synergy between learner agency and the benefits of multiple representations (as grounded in Dual Coding Theory and Multimedia Learning principles). Therefore, attempting to isolate the performance contribution of any individual content transformation from the element of student choice would fundamentally detach the intervention from its core theoretical foundation, misrepresenting the mechanism by which we hypothesize the system promotes positive performance outcomes.

Our results also demonstrate that students who engaged with Learn Your Way reported a more positive learning experience compared to those who used a traditional textbook. The present study's findings are supported by a body of literature suggesting that transforming educational content into multiple representations can make the learning process more enjoyable, engaging, and motivating (Ainsworth, 1999). This is particularly powerful when students are able to select representations that best match their individual learning preferences (Surjono, 2015; Popescu, 2010). Learn Your Way, which is capable of dynamically transforming content, represents a critical evolution from standard digital textbooks by enabling more tailored and engaging educational experiences for students.

Finally, our qualitative feedback indicated a distinct preference for content transformation tools over other generative AI applications, citing their substantial contribution to the learning experience. This finding underscores a potential differentiation in the pedagogical impact of various AI tools, where those that enable students to explore concepts from multiple perspectives may offer a more direct and perceived benefit to student learning.

Limitations and future work

7.1

While our findings provide evidence that AI-powered content transformation tools can positively impact learning outcomes and experiences, our study is not without limitations. First, our study took place in an experimental lab setting. Future research should explore best practices for how and when to utilize these content transformation tools within a classroom environment, and collect longitudinal outcomes on learning and retention. Second, a limitation of the current study design is the lack of a feature-matched control group. While the Textbook condition represents the current standard for digital learning, it lacks the interactive scaffolds (e.g., quizzes and feedback) present in Learn Your Way. Consequently, the observed gains may be attributed to the combined effect of content transformation and these interactive elements. Future research should employ a control condition that includes non-AI-generated quizzes to isolate the specific impact of the generative content transformations. Third, while the observed effect sizes for immediate (r = 0.24) and long-term recall (r = 0.25) are characterized as small to medium by traditional conventions, their practical significance in a classroom setting is noteworthy.

In educational contexts, small improvements in recall can lead to significant cumulative gains in student mastery over a full academic year. Furthermore, these effects were achieved with a relatively brief intervention (up to 40 min). For a teacher, moving a student from an average score of 8.2/12 to 9.2/12—as seen in our immediate recall results—represents the difference between a “D” and a “C” grade. Given that Learn Your Way is an automated, AI-driven tool, these gains suggest a scalable method for providing personalized support that would otherwise require intensive one-on-one tutoring. Thus, the tool demonstrates a “low-effort, high-yield” potential for augmenting standard classroom instruction.

Finally, while this study provides a foundational proof-of-concept, several factors present potential threats to its external validity. Specifically, the controlled lab environment and the focus on a single age group (high school students) and a single subject area (neuroscience) limit the generalizability of our findings to more ecologically valid settings. To address this, future research should transition from the lab to a longitudinal field study within diverse classroom environments. This will allow researchers to observe how the efficacy of AI-powered learning persists across various disciplines—such as mathematics or literature—where the nature of “transformation” may differ structurally. Furthermore, by expanding the participant pool to include middle school and university students, researchers can identify the specific developmental stages at which these interactive representations provide the most significant pedagogical benefit.

Conclusion

8

Our experiment results demonstrate the potential of using generative AI to transform traditional textbook chapters into more engaging formats. We found that US high school students from the Chicago area preferred Learn Your Way over a traditional digital textbook, appreciating the dynamic representations of the content, quizzes, and feedback. The burgeoning field of content transformation tools offers a compelling opportunity to not only enhance learning outcomes but also to enrich the overall student experience.