Sink or SWIM: Tackling Real-Time ASR at Scale

TL;DR

This paper introduces SWIM, a scalable, real-time multilingual ASR system built on Whisper, capable of supporting multiple concurrent users with low latency and high accuracy.

Contribution

SWIM enables true model-level parallelization for scalable, multilingual transcription without modifying the underlying Whisper model, supporting multiple clients efficiently.

Findings

SWIM supports up to 20 concurrent users with maintained accuracy.

SWIM achieves around 2.4 seconds delay with 5 clients, lower than single-client settings.

SWIM maintains high throughput and transcription quality in multi-user environments.

Abstract

Real-time automatic speech recognition systems are increasingly integrated into interactive applications, from voice assistants to live transcription services. However, scaling these systems to support multiple concurrent clients while maintaining low latency and high accuracy remains a major challenge. In this work, we present SWIM, a novel real-time ASR system built on top of OpenAI's Whisper model that enables true model-level parallelization for scalable, multilingual transcription. SWIM supports multiple concurrent audio streams without modifying the underlying model. It introduces a buffer merging strategy that maintains transcription fidelity while ensuring efficient resource usage. We evaluate SWIM in multi-client settings -- scaling up to 20 concurrent users -- and show that it delivers accurate real-time transcriptions in English, Italian, and Spanish, while maintaining low latency and high throughput. While Whisper-Streaming achieves a word error rate of approximately 8.2% with an average delay of approximately 3.4 s in a single-client, English-only setting, SWIM extends this capability to multilingual, multi-client environments. It maintains comparable accuracy with significantly lower delay -- around 2.4 s with 5 clients -- and continues to scale effectively up to 20 concurrent clients without degrading transcription quality and increasing overall throughput. Our approach advances scalable ASR by improving robustness and efficiency in dynamic, multi-user environments.