RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices

TL;DR

RetAssist is an interactive vocabulary learning system that uses generative images to enhance story retelling practices, significantly improving learners' fluency and perceived usefulness compared to baseline methods.

Contribution

This paper introduces a novel workflow and system leveraging text-to-image generation to support vocabulary learning through story retelling, validated by empirical user study.

Findings

RetAssist significantly improves learners' fluency in target words.

Participants find RetAssist easier and more useful for learning.

Generative images aid in understanding and recalling story contexts.

Abstract

Reading and repeatedly retelling a short story is a common and effective approach to learning the meanings and usages of target words. However, learners often struggle with comprehending, recalling, and retelling the story contexts of these target words. Inspired by the Cognitive Theory of Multimedia Learning, we propose a computational workflow to generate relevant images paired with stories. Based on the workflow, we work with learners and teachers to iteratively design an interactive vocabulary learning system named RetAssist. It can generate sentence-level images of a story to facilitate the understanding and recall of the target words in the story retelling practices. Our within-subjects study (N=24) shows that compared to a baseline system without generative images, RetAssist significantly improves learners' fluency in expressing with target words. Participants also feel that RetAssist eases their learning workload and is more useful. We discuss insights into leveraging text-to-image generative models to support learning tasks.