TL;DR
This paper introduces a new approach to frame interpolation in speech videos, emphasizing linguistically-informed metrics and datasets to improve speech understanding in video generation models.
Contribution
It proposes linguistically-informed metrics and datasets specifically designed for speech video frame interpolation, addressing limitations of conventional non-linguistic evaluation methods.
Findings
Computer vision models often fail to produce faithful speech video interpolation.
Linguistically-informed metrics better capture speech content fidelity.
New datasets enable testing of speech understanding in video generation.
Abstract
In this work, we explore a new problem of frame interpolation for speech videos. Such content today forms the major form of online communication. We try to solve this problem by using several deep learning video generation algorithms to generate the missing frames. We also provide examples where computer vision models despite showing high performance on conventional non-linguistic metrics fail to accurately produce faithful interpolation of speech. With this motivation, we provide a new set of linguistically-informed metrics specifically targeted to the problem of speech videos interpolation. We also release several datasets to test computer vision video generation models of their speech understanding.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
