Part of speech and gramset tagging algorithms for unknown words based on morphological dictionaries of the Veps and Karelian languages

TL;DR

This paper presents algorithms for part of speech and grammatical tagging of unknown words in low-resource Veps and Karelian languages, utilizing morphological dictionaries and suffix-based analogy hypotheses, achieving over 86% accuracy.

Contribution

The study introduces novel suffix-based algorithms for tagging unknown words in low-resource languages using morphological dictionaries, improving accuracy in part of speech and gramset assignment.

Findings

92.4% Vepsian words correctly tagged with part of speech

86.8% Karelian words correctly tagged with part of speech

Over 95% of Vepsian words correctly assigned gramset

Abstract

This research devoted to the low-resource Veps and Karelian languages. Algorithms for assigning part of speech tags to words and grammatical properties to words are presented in the article. These algorithms use our morphological dictionaries, where the lemma, part of speech and a set of grammatical features (gramset) are known for each word form. The algorithms are based on the analogy hypothesis that words with the same suffixes are likely to have the same inflectional models, the same part of speech and gramset. The accuracy of these algorithms were evaluated and compared. 313 thousand Vepsian and 66 thousand Karelian words were used to verify the accuracy of these algorithms. The special functions were designed to assess the quality of results of the developed algorithms. 92.4% of Vepsian words and 86.8% of Karelian words were assigned a correct part of speech by the developed algorithm. 95.3% of Vepsian words and 90.7% of Karelian words were assigned a correct gramset by our algorithm. Morphological and semantic tagging of texts, which are closely related and inseparable in our corpus processes, are described in the paper.