Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognition

TL;DR

Uni-ASR is a unified LLM-based speech recognition framework that seamlessly integrates non-streaming and streaming modes, improving accuracy and latency performance without architectural changes.

Contribution

It introduces a joint training paradigm, context-aware training, and a fallback decoding strategy for unified speech recognition with LLMs.

Findings

Achieves competitive non-streaming recognition performance.

Demonstrates strong streaming recognition effectiveness under various latency constraints.

Enhances streaming accuracy without additional latency.

Abstract

Although the deep integration of the Automatic Speech Recognition (ASR) system with Large Language Models (LLMs) has significantly improved accuracy, the deployment of such systems in low-latency streaming scenarios remains challenging. In this paper, we propose Uni-ASR, a unified framework based on LLMs that integrates both non-streaming and streaming speech recognition capabilities. We propose a joint training paradigm that enables the system to seamlessly transition between two recognition modes without any architectural modifications. Furthermore, we introduce a context-aware training paradigm and a co-designed fallback decoding strategy, which can enhance streaming recognition accuracy without introducing additional latency. The experimental results demonstrate that Uni-ASR not only achieves competitive performance within non-streaming mode, but also demonstrates strong effectiveness in streaming scenarios under diverse latency constraints.