Accuracy of the Uzbek stop words detection: a case study on "School corpus"

TL;DR

This study evaluates the accuracy of automatically generated Uzbek stop words lists within the 'School corpus', highlighting the challenges and potential methods for assessing stop words in agglutinative languages like Uzbek.

Contribution

It introduces a method to evaluate stop words quality in Uzbek, demonstrating its applicability and analyzing the distribution of stop words in texts.

Findings

Stop words lists achieved acceptable accuracy.

The method can be adapted for similar agglutinative languages.

Numerical analysis helps identify key parts of sentences with stop words.

Abstract

Stop words are very important for information retrieval and text analysis investigation tasks of natural language processing. Current work presents a method to evaluate the quality of a list of stop words aimed at automatically creating techniques. Although the method proposed in this paper was tested on an automatically-generated list of stop words for the Uzbek language, it can be, with some modifications, applied to similar languages either from the same family or the ones that have an agglutinative nature. Since the Uzbek language belongs to the family of agglutinative languages, it can be explained that the automatic detection of stop words in the language is a more complex process than in inflected languages. Moreover, we integrated our previous work on stop words detection in the example of the "School corpus" by investigating how to automatically analyse the detection of stop words in Uzbek texts. This work is devoted to answering whether there is a good way of evaluating available stop words for Uzbek texts, or whether it is possible to determine what part of the Uzbek sentence contains the majority of the stop words by studying the numerical characteristics of the probability of unique words. The results show acceptable accuracy of the stop words lists.