Benchmarking Speech Systems for Frontline Health Conversations: The DISPLACE-M Challenge

TL;DR

This paper presents the DISPLACE-M challenge, a benchmark for evaluating speech processing systems in real-world medical conversations involving multiple speakers, spontaneous speech, and noise, with datasets and baseline systems.

Contribution

It introduces a new medical conversational dataset, defines four benchmark tasks, and provides baseline systems for evaluating speech processing in medical dialogue scenarios.

Findings

Baseline systems achieved measurable performance on all tasks.

Evaluation results highlight challenges in diarization and ASR in medical conversations.

The dataset and benchmarks facilitate future research in medical conversational AI.

Abstract

The DIarization and Speech Processing for LAnguage understanding in Conversational Environments - Medical (DISPLACE-M) challenge introduces a conversational AI benchmark for understanding goal-oriented, real-world medical dialogues. The challenge addresses multi-speaker interactions between frontline health workers and care seekers, characterized by spontaneous, noisy and overlapping speech. As part of the challenge, medical conversational dataset comprising 40 hours of development and 15 hours of blind evaluation recordings was released. We provided baseline systems across 4 tasks - speaker diarization, automatic speech recognition, topic identification and dialogue summarization - to enable consistent benchmarking. System performance is evaluated using diarization error rate (DER), time-constrained minimum-permutation word error rate (tcpWER) and ROUGE-L. This paper describes the Phase-I evaluation - data, tasks and baseline systems - along with the summary of the evaluation results.