Method for Determining the Similarity of Text Documents for the Kazakh language, Taking Into Account Synonyms: Extension to TF-IDF

TL;DR

This paper introduces an extension to the TF-IDF method that incorporates synonyms to improve text document similarity measurement for the Kazakh language, validated through experiments using various similarity functions.

Contribution

The paper presents a novel TF-IDF extension that accounts for synonyms, enhancing document similarity analysis specifically for Kazakh language texts.

Findings

Improved accuracy in document similarity measurement with synonym-aware TF-IDF.

Effective validation using Cosine, Dice, and Jaccard similarity functions.

Demonstrated applicability to Kazakh language processing tasks.

Abstract

The task of determining the similarity of text documents has received considerable attention in many areas such as Information Retrieval, Text Mining, Natural Language Processing (NLP) and Computational Linguistics. Transferring data to numeric vectors is a complex task where algorithms such as tokenization, stopword filtering, stemming, and weighting of terms are used. The term frequency - inverse document frequency (TF-IDF) is the most widely used term weighting method to facilitate the search for relevant documents. To improve the weighting of terms, a large number of TF-IDF extensions are made. In this paper, another extension of the TF-IDF method is proposed where synonyms are taken into account. The effectiveness of the method is confirmed by experiments on functions such as Cosine, Dice and Jaccard to measure the similarity of text documents for the Kazakh language.