WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speech

TL;DR

This paper introduces WhisperAlign and WhisperX-anchored diarization techniques tailored for long-form Bengali speech, improving transcription accuracy and speaker boundary detection in challenging multi-speaker scenarios.

Contribution

It presents a novel chunking strategy using whisper timestamps and domain-specific fine-tuning of diarization models for Bengali, enhancing performance in low-resource, long-form speech tasks.

Findings

Reduced Word Error Rate (WER) in Bengali ASR

Improved Diarization Error Rate (DER) for overlapping speakers

Effective handling of long-form, multi-speaker Bengali audio

Abstract

This paper presents our solution for the DL Sprint 4.0, addressing the dual challenges of Bengali Long-Form Speech Recognition (Task 1) and Speaker Diarization (Task 2). Processing long-form, multi-speaker Bengali audio introduces significant hurdles in voice activity detection, overlapping speech, and context preservation. To solve the long-form transcription challenge, we implemented a robust audio chunking strategy utilizing whisper-timestamped, allowing us to feed precise, context-aware segments into our fine-tuned acoustic model for high-accuracy transcription. For the diarization task, we developed an integrated pipeline leveraging pyannote.audio and WhisperX. A key contribution of our approach is the domain-specific fine-tuning of the Pyannote segmentation model on the competition dataset. This adaptation allowed the model to better capture the nuances of Bengali conversational dynamics and accurately resolve complex, overlapping speaker boundaries. Our methodology demonstrates that applying intelligent timestamped chunking to ASR and targeted segmentation fine-tuning to diarization significantly drives down Word Error Rate (WER) and Diarization Error Rate (DER), in low-resource settings.