Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance

TL;DR

This paper introduces a novel seq2seq-based architecture for normalizing phonetic variations in code-mixed social media data, also capable of back-transliteration and word identification, achieving over 90% accuracy.

Contribution

The work presents a new model that effectively normalizes transliterated words in code-mixed data, addressing phonetic spelling variations and enhancing NLP tools for social media analysis.

Findings

Achieved 90.27% accuracy on test data.

Model effectively normalizes phonetic spelling variations.

Supports back-transliteration and word identification.

Abstract

Building tools for code-mixed data is rapidly gaining popularity in the NLP research community as such data is exponentially rising on social media. Working with code-mixed data contains several challenges, especially due to grammatical inconsistencies and spelling variations in addition to all the previous known challenges for social media scenarios. In this article, we present a novel architecture focusing on normalizing phonetic typing variations, which is commonly seen in code-mixed data. One of the main features of our architecture is that in addition to normalizing, it can also be utilized for back-transliteration and word identification in some cases. Our model achieved an accuracy of 90.27% on the test data.