Spoken Conversational Agents with Large Language Models

TL;DR

This paper reviews the development of spoken conversational agents powered by large language models, discussing system architectures, adaptation techniques, datasets, and open challenges in privacy and safety.

Contribution

It provides a comprehensive overview of system designs, adaptation methods, and open problems for integrating large language models into spoken conversational agents.

Findings

Comparison of cascaded vs. end-to-end systems

Analysis of robustness across accents

Reproducible baselines and practical recipes

Abstract

Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.