Unsupervised Separation of Transliterable and Native Words for Malayalam

TL;DR

This paper introduces an unsupervised method called DTIM for distinguishing native Malayalam words from transliterable ones, improving text processing accuracy by leveraging character n-gram distributions.

Contribution

The paper presents a novel unsupervised optimization approach that effectively scores words based on nativeness, specifically tailored for Malayalam language processing.

Findings

DTIM significantly outperforms existing methods in nativeness scoring.

The approach effectively captures character diversity beyond word stems.

Empirical results validate DTIM as the preferred method for Malayalam word separation.

Abstract

Differentiating intrinsic language words from transliterable words is a key step aiding text processing tasks involving different natural languages. We consider the problem of unsupervised separation of transliterable words from native words for text in Malayalam language. Outlining a key observation on the diversity of characters beyond the word stem, we develop an optimization method to score words based on their nativeness. Our method relies on the usage of probability distributions over character n-grams that are refined in step with the nativeness scorings in an iterative optimization formulation. Using an empirical evaluation, we illustrate that our method, DTIM, provides significant improvements in nativeness scoring for Malayalam, establishing DTIM as the preferred method for the task.