BER: Balanced Error Rate For Speaker Diarization

TL;DR

This paper introduces the Balanced Error Rate (BER), a new comprehensive metric for speaker diarization that addresses limitations of existing metrics by considering segment duration, speaker importance, and semantic content.

Contribution

The paper proposes a novel BER metric incorporating segment-level errors, speaker-specific weighting, and adaptive matching, providing a more balanced evaluation of diarization performance.

Findings

BER effectively balances errors across short and long segments.

BER highlights errors in less-talked speakers overlooked by DER.

The metric is validated on real datasets using EEND and multi-modal systems.

Abstract

DER is the primary metric to evaluate diarization performance while facing a dilemma: the errors in short utterances or segments tend to be overwhelmed by longer ones. Short segments, e.g., `yes' or `no,' still have semantic information. Besides, DER overlooks errors in less-talked speakers. Although JER balances speaker errors, it still suffers from the same dilemma. Considering all those aspects, duration error, segment error, and speaker-weighted error constituting a complete diarization evaluation, we propose a Balanced Error Rate (BER) to evaluate speaker diarization. First, we propose a segment-level error rate (SER) via connected sub-graphs and adaptive IoU threshold to get accurate segment matching. Second, to evaluate diarization in a unified way, we adopt a speaker-specific harmonic mean between duration and segment, followed by a speaker-weighted average. Third, we analyze our metric via the modularized system, EEND, and the multi-modal method on real datasets. SER and BER are publicly available at https://github.com/X-LANCE/BER.