SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Models

TL;DR

This paper introduces SEAL, an acoustic-conditioned large language model approach that significantly improves speaker error correction in diarization tasks by leveraging acoustic cues and constrained decoding, reducing errors by up to 43%.

Contribution

The paper presents a novel acoustic conditioning method for LLMs in speaker diarization, enhancing correction accuracy without complex post-processing.

Findings

Reduces speaker error rates by 24-43% across multiple datasets.

Uses acoustic cues to improve LLM-based correction.

Employs constrained decoding to minimize hallucinations.

Abstract

Speaker Diarization (SD) is a crucial component of modern end-to-end ASR pipelines. Traditional SD systems, which are typically audio-based and operate independently of ASR, often introduce speaker errors, particularly during speaker transitions and overlapping speech. Recently, language models including fine-tuned large language models (LLMs) have shown to be effective as a second-pass speaker error corrector by leveraging lexical context in the transcribed output. In this work, we introduce a novel acoustic conditioning approach to provide more fine-grained information from the acoustic diarizer to the LLM. We also show that a simpler constrained decoding strategy reduces LLM hallucinations, while avoiding complicated post-processing. Our approach significantly reduces the speaker error rates by 24-43% across Fisher, Callhome, and RT03-CTS datasets, compared to the first-pass Acoustic SD.