Detection of Audio-Video Synchronization Errors Via Event Detection
Joshua P. Ebenezer, Yongjun Wu, Hai Wei, Sriram Sethuraman, Zongyi Liu

TL;DR
This paper introduces a deep learning-based method and a large-scale database to detect audio-video synchronization errors in tennis videos by identifying specific event signatures in both streams.
Contribution
It presents a novel approach combining deep networks and a large database to accurately detect A/V sync errors in sports videos, specifically tennis.
Findings
High accuracy in detecting A/V sync errors
Effective detection of tennis ball hit events in audio and video
Large-scale database with over 500,000 frames
Abstract
We present a new method and a large-scale database to detect audio-video synchronization(A/V sync) errors in tennis videos. A deep network is trained to detect the visual signature of the tennis ball being hit by the racquet in the video stream. Another deep network is trained to detect the auditory signature of the same event in the audio stream. During evaluation, the audio stream is searched by the audio network for the audio event of the ball being hit. If the event is found in audio, the neighboring interval in video is searched for the corresponding visual signature. If the event is not found in the video stream but is found in the audio stream, A/V sync error is flagged. We developed a large-scaled database of 504,300 frames from 6 hours of videos of tennis events, simulated A/V sync errors, and found our method achieves high accuracy on the task.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Analysis and Summarization · Music and Audio Processing · Human Pose and Action Recognition
