Training end-to-end speech-to-text models on mobile phones

TL;DR

This paper presents a framework for training end-to-end speech-to-text models directly on mobile phones, demonstrating the trade-offs between accuracy, training time, and resource consumption across various devices.

Contribution

The authors develop a detailed on-device training framework for speech-to-text models, evaluated on multiple mobile phones, highlighting practical considerations for resource-limited environments.

Findings

7.6% reduction in word error rate (WER) with on-device training

Trade-off analysis between WER, training time, and memory usage

Real-time computational cost measurements on mobile devices

Abstract

Training the state-of-the-art speech-to-text (STT) models in mobile devices is challenging due to its limited resources relative to a server environment. In addition, these models are trained on generic datasets that are not exhaustive in capturing user-specific characteristics. Recently, on-device personalization techniques have been making strides in mitigating the problem. Although many current works have already explored the effectiveness of on-device personalization, the majority of their findings are limited to simulation settings or a specific smartphone. In this paper, we develop and provide a detailed explanation of our framework to train end-to-end models in mobile phones. To make it simple, we considered a model based on connectionist temporal classification (CTC) loss. We evaluated the framework on various mobile phones from different brands and reported the results. We provide enough evidence that fine-tuning the models and choosing the right hyperparameter values is a trade-off between the lowest WER achievable, training time on-device, and memory consumption. Hence, this is vital for a successful deployment of on-device training onto a resource-limited environment like mobile phones. We use training sets from speakers with different accents and record a 7.6% decrease in average word error rate (WER). We also report the associated computational cost measurements with respect to time, memory usage, and cpu utilization in mobile phones in real-time.