Aligning Spoken Dialogue Models from User Interactions

TL;DR

This paper introduces a new framework for aligning spoken dialogue models using user interaction data, enhancing real-time speech conversations by incorporating rich dynamics and feedback.

Contribution

It presents a large-scale dataset and offline alignment method to fine-tune speech-to-speech models, improving factuality, safety, and contextual relevance in dialogue systems.

Findings

Feedback improves dialogue model performance

Large-scale annotated speech dataset created

Enhanced real-time speech interaction quality

Abstract

We propose a novel preference alignment framework for improving spoken dialogue models on real-time conversations from user interactions. Current preference learning methods primarily focus on text-based language models, and are not directly suited to the complexities of real-time speech interactions, with richer dynamics (e.g. interruption, interjection) and no explicit segmentation between speaker turns.We create a large-scale dataset of more than 150,000 preference pairs from raw multi-turn speech conversations, annotated with AI feedback, to cover preferences over both linguistic content and temporal context variations. We leverage offline alignment methods to finetune a full-duplex autoregressive speech-to-speech model. Extensive experiments demonstrate that feedback on generic conversations can be consistently effective in improving spoken dialogue models to produce more factual, safer and more contextually aligned interactions. We deploy the finetuned model and conduct holistic human evaluations to assess the impact beyond single-turn conversations. Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.