Useful but Distracting: Keyword Highlights and Time-Synchronization in Captions for Language Learning

TL;DR

This study explores how enhanced captions with keyword highlights and time-synchronization affect language learners' experience, finding that while preferred for learning, they can be distracting in everyday viewing, highlighting the need for better design integration.

Contribution

The paper investigates learner preferences for enhanced captions, introduces a processing pipeline for keyword highlights, and evaluates their effectiveness and distraction levels in real-world scenarios.

Findings

Time-synchronized keyword highlights are preferred for learning.

Enhanced captions can be distracting in everyday viewing.

Design optimization is needed for seamless integration.

Abstract

Captions provide language learners with a scaffold for comprehension and vocabulary acquisition. Past work has proposed several enhancements such as keyword highlights for increased learning gains. However, little is known about learners' experience with enhanced captions, although this is critical for adoption in everyday life. We conducted a survey and focus group to elicit learner preferences and requirements and implemented a processing pipeline for enhanced captions with keyword highlights, time-synchronized keyword highlights, and keyword captions. A subsequent online study (n = 49) showed that time-synchronized keyword highlights were the preferred design for learning but were perceived as too distracting to replace standard captions in everyday viewing scenarios. We conclude that keyword highlights and time-synchronization are suitable for integrating learning into an entertaining everyday-life activity, but the design should be optimized to provide a more seamless experience.