VoiceMoji: A Novel On-Device Pipeline for Seamless Emoji Insertion in Dictation

TL;DR

This paper introduces VoiceMoji, an on-device pipeline that intelligently inserts emojis into transcribed speech to enhance emotional expression, using a novel architecture that is efficient and suitable for deployment on resource-constrained devices.

Contribution

It presents the first on-device system for emoji insertion in speech transcription, utilizing a novel Attention-based Char Aware LSTM architecture that handles OOV words and reduces model size.

Findings

Achieves comparable emoji prediction accuracy to neural approaches with 80% fewer parameters.

Operates entirely on-device with a small 4MB footprint.

Demonstrates effective semantic analysis for emoji placement.

Abstract

Most of the speech recognition systems recover only words in the speech and fail to capture emotions. Users have to manually add emoji(s) in text for adding tone and making communication fun. Though there is much work done on punctuation addition on transcribed speech, the area of emotion addition is untouched. In this paper, we propose a novel on-device pipeline to enrich the voice input experience. It involves, given a blob of transcribed text, intelligently processing and identifying structure where emoji insertion makes sense. Moreover, it includes semantic text analysis to predict emoji for each of the sub-parts for which we propose a novel architecture Attention-based Char Aware (ACA) LSTM which handles Out-Of-Vocabulary (OOV) words as well. All these tasks are executed completely on-device and hence can aid on-device dictation systems. To the best of our knowledge, this is the first work that shows how to add emoji(s) in the transcribed text. We demonstrate that our components achieve comparable results to previous neural approaches for punctuation addition and emoji prediction with 80% fewer parameters. Overall, our proposed model has a very small memory footprint of a mere 4MB to suit on-device deployment.