RhythmTA: A Visual-Aided Interactive System for ESL Rhythm Training via Dubbing Practice

TL;DR

RhythmTA is an interactive system that helps ESL learners independently improve their speech rhythm through visual-aided dubbing practice, addressing limitations of traditional instructor-dependent training.

Contribution

It introduces a novel visual-aided dubbing system for ESL rhythm training that enables independent practice and incorporates three interactive stages based on formative study insights.

Findings

Improves ESL learners' rhythm perception.

Significantly enhances rhythm production.

Supports independent practice with visual aids.

Abstract

English speech rhythm, the temporal patterns of stressed syllables, is essential for English as a second language (ESL) learners to produce natural-sounding and comprehensible speech. Rhythm training is generally based on imitation of native speech. However, it relies heavily on external instructor feedback, preventing ESL learners from independent practice. To address this gap, we present RhythmTA, an interactive system for ESL learners to practice speech rhythm independently via dubbing, an imitation-based approach. The system automatically extracts rhythm from any English speech and introduces novel visual designs to support three stages of dubbing practice: (1) Synchronized listening with visual aids to enhance perception, (2) Guided repeating by visual cues for self-adjustment, and (3) Comparative reflection from a parallel view for self-monitoring. Our design is informed by a formative study with nine spoken English instructors, which identified current practices and challenges. A user study with twelve ESL learners demonstrates that RhythmTA effectively enhances learners' rhythm perception and shows significant potential for improving rhythm production.