AutoLV: Automatic Lecture Video Generator
Wenbin Wang, Yang Song, Sanjay Jha

TL;DR
AutoLV is an end-to-end system that automatically generates realistic lecture videos from slides, voice, and portrait, reducing instructor workload and enabling multilingual dissemination.
Contribution
It introduces a novel integrated system combining speech synthesis with speaker adaptation and talking-head generation for lecture videos.
Findings
Outperforms existing methods in authenticity and naturalness
Capable of multilingual and accent variations
Reduces instructor workload significantly
Abstract
We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor's reference voice and instructor's reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors' workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experimental results show that the proposed model outperforms other current approaches in terms of authenticity, naturalness and accuracy. Here is a video demonstration of how our system works, and the outcomes of the evaluation and comparison: https://youtu.be/cY6TYkI0cog.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Analysis and Summarization · Speech and Audio Processing · Multimedia Communication and Technology
