SBAAM! Eliminating Transcript Dependency in Automatic Subtitling

TL;DR

This paper introduces a novel model that generates subtitles directly from audio without relying on transcripts, improving accessibility and performance across languages.

Contribution

The first direct subtitle generation model that eliminates the need for intermediate transcripts for translation, segmentation, and timestamp prediction.

Findings

Achieves state-of-the-art performance across multiple languages

Outperforms transcript-dependent methods in accuracy

Validated through manual evaluation

Abstract

Subtitling plays a crucial role in enhancing the accessibility of audiovisual content and encompasses three primary subtasks: translating spoken dialogue, segmenting translations into concise textual units, and estimating timestamps that govern their on-screen duration. Past attempts to automate this process rely, to varying degrees, on automatic transcripts, employed diversely for the three subtasks. In response to the acknowledged limitations associated with this reliance on transcripts, recent research has shifted towards transcription-free solutions for translation and segmentation, leaving the direct generation of timestamps as uncharted territory. To fill this gap, we introduce the first direct model capable of producing automatic subtitles, entirely eliminating any dependence on intermediate transcripts also for timestamp prediction. Experimental results, backed by manual evaluation, showcase our solution's new state-of-the-art performance across multiple language pairs and diverse conditions.