Contextual Analysis for Middle Eastern Languages with Hidden Markov Models

TL;DR

This paper introduces a machine learning approach using Hidden Markov Models for contextual analysis of Middle Eastern languages, demonstrated with Farsi, achieving high accuracy and adaptable to other similar languages.

Contribution

The paper presents a novel application of first-order Hidden Markov Models for language-specific contextual analysis, reducing the need for complex rule coding across multiple languages.

Findings

Farsi model achieves 94% accuracy.

Approach can be extended to Arabic, Urdu, Sindhi.

Software can perform language analysis without complex rules.

Abstract

Displaying a document in Middle Eastern languages requires contextual analysis due to different presentational forms for each character of the alphabet. The words of the document will be formed by the joining of the correct positional glyphs representing corresponding presentational forms of the characters. A set of rules defines the joining of the glyphs. As usual, these rules vary from language to language and are subject to interpretation by the software developers. In this paper, we propose a machine learning approach for contextual analysis based on the first order Hidden Markov Model. We will design and build a model for the Farsi language to exhibit this technology. The Farsi model achieves 94 \% accuracy with the training based on a short list of 89 Farsi vocabularies consisting of 2780 Farsi characters. The experiment can be easily extended to many languages including Arabic, Urdu, and Sindhi. Furthermore, the advantage of this approach is that the same software can be used to perform contextual analysis without coding complex rules for each specific language. Of particular interest is that the languages with fewer speakers can have greater representation on the web, since they are typically ignored by software developers due to lack of financial incentives.