Development of a rule-based lemmatization algorithm through Finite State Machine for Uzbek language

TL;DR

This paper presents a rule-based lemmatization algorithm for Uzbek using a finite state machine, leveraging affix databases and part of speech data to accurately identify root words.

Contribution

It introduces a novel finite state machine approach for Uzbek lemmatization, incorporating affix classification and part of speech knowledge.

Findings

Effective affix removal using FSM

Accurate lemma identification for Uzbek words

Utilizes affix database and POS data

Abstract

Lemmatization is one of the core concepts in natural language processing, thus creating a lemmatization tool is an important task. This paper discusses the construction of a lemmatization algorithm for the Uzbek language. The main purpose of the work is to remove affixes of words in the Uzbek language by means of the finite state machine and to identify a lemma (a word that can be found in the dictionary) of the word. The process of removing affixes uses a database of affixes and part of speech knowledge. This lemmatization consists of the general rules and a part of speech data of the Uzbek language, affixes, classification of affixes, removing affixes on the basis of the finite state machine for each class, as well as a definition of this word lemma.