Attention2Probability: Attention-Driven Terminology Probability Estimation for Robust Speech-to-Text System

TL;DR

Attention2Probability is a novel, lightweight method that leverages attention weights to accurately estimate the presence of domain-specific terms in speech-to-text systems, improving recognition of neologisms and terminology.

Contribution

It introduces an attention-driven probability estimation approach combined with curriculum learning and provides a new dataset for terminology in speech recognition.

Findings

Outperforms VectorDB with up to 92.57% recall in Chinese

Achieves low latency of 8.71ms per query

Improves terminology accuracy by 6-17% in recognition tasks

Abstract

Recent advances in speech large language models (SLMs) have improved speech recognition and translation in general domains, but accurately generating domain-specific terms or neologisms remains challenging. To address this, we propose Attention2Probability: attention-driven terminology probability estimation for robust speech-to-text system, which is lightweight, flexible, and accurate. Attention2Probability converts cross-attention weights between speech and terminology into presence probabilities, and it further employs curriculum learning to enhance retrieval accuracy. Furthermore, to tackle the lack of data for speech-to-text tasks with terminology intervention, we create and release a new speech dataset with terminology to support future research in this area. Experimental results show that Attention2Probability significantly outperforms the VectorDB method on our test set. Specifically, its maximum recall rates reach 92.57% for Chinese and 86.83% for English. This high recall is achieved with a latency of only 8.71ms per query. Intervening in SLMs' recognition and translation tasks using Attention2Probability-retrieved terms improves terminology accuracy by 6-17%, while revealing that the current utilization of terminology by SLMs has limitations.