Character-aware audio-visual subtitling in context

TL;DR

This paper introduces an advanced character-aware audio-visual subtitling framework for TV shows that combines speech recognition, speaker diarisation, and character recognition using audio-visual cues, improving accuracy and speaker identification especially in short segments.

Contribution

The novel integration of audio-visual synchronisation with local voice embeddings and large language models for improved speaker diarisation and character recognition in TV show subtitling.

Findings

Enhanced speaker diarisation accuracy on TV show dataset

Improved character recognition accuracy over existing methods

Effective identification of speakers in short dialogue segments

Abstract

This paper presents an improved framework for character-aware audio-visual subtitling in TV shows. Our approach integrates speech recognition, speaker diarisation, and character recognition, utilising both audio and visual cues. This holistic solution addresses what is said, when it's said, and who is speaking, providing a more comprehensive and accurate character-aware subtitling for TV shows. Our approach brings improvements on two fronts: first, we show that audio-visual synchronisation can be used to pick out the talking face amongst others present in a video clip, and assign an identity to the corresponding speech segment. This audio-visual approach improves recognition accuracy and yield over current methods. Second, we show that the speaker of short segments can be determined by using the temporal context of the dialogue within a scene. We propose an approach using local voice embeddings of the audio, and large language model reasoning on the text transcription. This overcomes a limitation of existing methods that they are unable to accurately assign speakers to short temporal segments. We validate the method on a dataset with 12 TV shows, demonstrating superior performance in speaker diarisation and character recognition accuracy compared to existing approaches. Project page : https://www.robots.ox.ac.uk/~vgg/research/llr-context/