Language Detection Engine for Multilingual Texting on Mobile Devices

TL;DR

This paper introduces a fast, accurate, and lightweight language detection engine for multilingual mobile typing, significantly improving accuracy and speed over existing solutions, and effectively handling code-switching and language ambiguity.

Contribution

The paper presents a novel fusion of character N-gram and logistic regression models for real-time language detection on mobile devices, with parameter reduction for faster inference.

Findings

Achieves 94.5% accuracy for Indian languages in Latin script.

Attains 98% accuracy for European languages on code-switched data.

Outperforms fastText and ML-Kit in F1 score, with faster inference time.

Abstract

More than 2 billion mobile users worldwide type in multiple languages in the soft keyboard. On a monolingual keyboard, 38% of falsely auto-corrected words are valid in another language. This can be easily avoided by detecting the language of typed words and then validating it in its respective language. Language detection is a well-known problem in natural language processing. In this paper, we present a fast, light-weight and accurate Language Detection Engine (LDE) for multilingual typing that dynamically adapts to user intended language in real-time. We propose a novel approach where the fusion of character N-gram model and logistic regression based selector model is used to identify the language. Additionally, we present a unique method of reducing the inference time significantly by parameter reduction technique. We also discuss various optimizations fabricated across LDE to resolve ambiguity in input text among the languages with the same character pattern. Our method demonstrates an average accuracy of 94.5% for Indian languages in Latin script and that of 98% for European languages on the code-switched data. This model outperforms fastText by 60.39% and ML-Kit by 23.67% in F1 score for European languages. LDE is faster on mobile device with an average inference time of 25.91 microseconds.