WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition

TL;DR

WhisperPipe is a streaming ASR architecture that reduces memory and latency while maintaining high transcription accuracy, enabling efficient real-time speech recognition on resource-constrained devices.

Contribution

It introduces a hybrid VAD, dynamic buffering, and adaptive processing to achieve bounded memory use and low latency without sacrificing accuracy.

Findings

Median latency of 89ms with 48% less GPU memory usage.

Maintains stable memory over 150-minute continuous operation.

Achieves near-offline Whisper accuracy with 3-5x lower latency.

Abstract

Real-time automatic speech recognition (ASR) systems face a fundamental trade-off between transcription accuracy and computational efficiency, particularly when deploying large-scale transformer models like Whisper. Existing streaming approaches either sacrifice accuracy through aggressive chunking or incur prohibitive memory costs through unbounded context accumulation. We present WhisperPipe, a novel streaming architecture that achieves bounded memory consumption while maintaining transcription quality through three key innovations a hybrid Voice Activity Detection (VAD) pipeline combining Silero VAD with energy-based filtering to reduce false activations by 34%, a dynamic buffering mechanism with overlapping context windows that prevents information loss at segment boundaries, and an adaptive processing strategy that balances latency and accuracy based on speech characteristics. Evaluated on 2.5 hours of diverse audio data, WhisperPipe demonstrates a median end-to-end latency of 89ms (90th percentile: 142ms) while consuming 48% less peak GPU memory and 80.9% lower average GPU utilization compared to baseline Whisper implementations. The system maintains stable memory usage over extended sessions, with zero growth rate across 150-minute continuous operation. Comparative analysis against related work shows that WhisperPipe achieves competitive accuracy (WER within 2% of offline Whisper) while operating at 3-5x lower latency than existing streaming solutions. The architecture's modular design enables deployment across resource-constrained environments, from edge devices to cloud infrastructure. Our results demonstrate that careful architectural design can reconcile the competing demands of real-time responsiveness and model sophistication in production ASR systems.