SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation

TL;DR

SpeechMedAssist introduces a novel two-stage training paradigm for SpeechLMs, enabling effective speech-based medical consultations with limited real speech data, outperforming baselines in simulated interactions.

Contribution

The paper proposes a two-stage training approach for SpeechLMs that reduces data requirements and enhances performance in medical speech interactions.

Findings

Outperforms baselines in effectiveness and robustness.

Requires only 10k synthesized speech samples for training.

Excels in both single-turn and multi-turn medical interactions.

Abstract

Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and patient-unfriendly. Recent advances in speech language models (SpeechLMs) have enabled more natural speech-based interaction, yet the scarcity of medical speech data and the inefficiency of directly fine-tuning on speech data jointly hinder the adoption of SpeechLMs in medical consultation. In this paper, we propose SpeechMedAssist, a SpeechLM natively capable of conducting speech-based multi-turn interactions with patients. By exploiting the architectural properties of SpeechLMs, we decouple the conventional one-stage training into a two-stage paradigm consisting of (1) Knowledge & Capability Injection via Text and (2) Modality Re-alignment with Limited Speech Data, thereby reducing the requirement for medical speech data to only 10k synthesized samples. To evaluate SpeechLMs for medical consultation scenarios, we design a benchmark comprising both single-turn question answering and multi-turn simulated interactions. Experimental results show that our model outperforms all baselines in both effectiveness and robustness in most evaluation settings.