SlideItRight: Using AI to Find Relevant Slides and Provide Feedback for Open-Ended Questions

TL;DR

This study explores AI-generated multimodal feedback using slides and text to enhance student learning, finding positive perceptions but no significant difference in learning gains compared to traditional methods.

Contribution

It introduces a novel approach combining Large Language Models with relevant lecture slides for multimodal feedback in education.

Findings

Students perceived slide feedback as helpful but hard to understand.

AI feedback was seen as personalized but less trustworthy than human feedback.

All conditions showed significant learning gains, with no significant differences between them.

Abstract

Feedback is important in supporting student learning. While various automated feedback systems have been implemented to make the feedback scalable, many existing solutions only focus on generating text-based feedback. As is indicated in the multimedia learning principle, learning with more modalities could help utilize more separate channels, reduce the cognitive load and facilitate students' learning. Hence, it is important to explore the potential of Artificial Intelligence (AI) in feedback generation from and to different modalities. Our study leverages Large Language Models (LLMs) for textual feedback with the supplementary guidance from other modality - relevant lecture slide retrieved from the slides hub. Through an online crowdsourcing study (N=91), this study investigates learning gains and student perceptions using a 2x2 design (i.e., human feedback vs. AI feedback and with vs. without relevant slide), evaluating the clarity, engagement, perceived effectiveness, and reliability) of AI-facilitated multimodal feedback. We observed significant pre-to-post learning gains across all conditions. However, the differences in these gains were not statistically significant between conditions. The post-survey revealed that students found the slide feedback helpful in their learning process, though they reported difficulty in understanding it. Regarding the AI-generated open-ended feedback, students considered it personalized and relevant to their responses, but they expressed lower trust in the AI feedback compared to human-generated feedback.