Speech-Synchronized Whiteboard Generation via VLM-Driven Structured Drawing Representations

TL;DR

This paper introduces a new dataset and a vision-language model fine-tuned to generate speech-synchronized whiteboard drawings, advancing automated educational video creation.

Contribution

It provides the first paired dataset of timed whiteboard demonstrations and shows that a fine-tuned VL model can predict synchronized drawing sequences from limited data.

Findings

Timestamp conditioning improves temporal alignment.

Model generalizes across unseen STEM topics.

Dataset and code are publicly released.

Abstract

Creating whiteboard-style educational videos demands precise coordination between freehand illustrations and spoken narration, yet no existing method addresses this multimodal synchronization problem with structured, reproducible drawing representations. We present the first dataset of 24 paired Excalidraw demonstrations with narrated audio, where every drawing element carries millisecond-precision creation timestamps spanning 8 STEM domains. Using this data, we study whether a vision-language model (Qwen2-VL-7B), fine-tuned via LoRA, can predict full stroke sequences synchronized to speech from only 24 demonstrations. Our topic-stratified five-fold evaluation reveals that timestamp conditioning significantly improves temporal alignment over ablated baselines, while the model generalizes across unseen STEM topics. We discuss transferability to real classroom settings and release our dataset and code to support future research in automated educational content generation.