Language Lexicons for Hindi-English Multilingual Text Processing

TL;DR

This paper introduces language lexicons for Hindi-English multilingual text processing, addressing the challenge of identifying multiple languages in a single document by creating lexical databases from transliterated vocabularies.

Contribution

It proposes a novel lexical database for Hindi-English multilingual tasks, built using classifiers over transliterated vocabularies, filling a gap due to lack of large corpora.

Findings

Lexicons have richer quantitative features than original vocabularies

Visualization reveals the enhanced characteristics of the lexicons

Supports multiple multilingual language processing tasks

Abstract

Language Identification in textual documents is the process of automatically detecting the language contained in a document based on its content. The present Language Identification techniques presume that a document contains text in one of the fixed set of languages, however, this presumption is incorrect when dealing with multilingual document which includes content in more than one possible language. Due to the unavailability of large standard corpora for Hindi-English mixed lingual language processing tasks we propose the language lexicons, a novel kind of lexical database that supports several multilingual language processing tasks. These lexicons are built by learning classifiers over transliterated Hindi and English vocabulary. The designed lexicons possess richer quantitative characteristic than its primary source of collection which is revealed using the visualization techniques.