Word Similarity Datasets for Thai: Construction and Evaluation

TL;DR

This paper introduces three Thai word similarity datasets derived from popular English datasets, enabling better evaluation of Thai word embeddings and highlighting challenges like out-of-vocabulary issues.

Contribution

The creation and release of three Thai word similarity datasets based on translation and re-rating of established English datasets, with baseline evaluations and a tool for model assessment.

Findings

High out-of-vocabulary rate in Thai embeddings

Datasets cover various difficulty levels and domains

Baseline evaluations demonstrate current model limitations

Abstract

Distributional semantics in the form of word embeddings are an essential ingredient to many modern natural language processing systems. The quantification of semantic similarity between words can be used to evaluate the ability of a system to perform semantic interpretation. To this end, a number of word similarity datasets have been created for the English language over the last decades. For Thai language few such resources are available. In this work, we create three Thai word similarity datasets by translating and re-rating the popular WordSim-353, SimLex-999 and SemEval-2017-Task-2 datasets. The three datasets contain 1852 word pairs in total and have different characteristics in terms of difficulty, domain coverage, and notion of similarity (relatedness vs.~similarity). These features help to gain a broader picture of the properties of an evaluated word embedding model. We include baseline evaluations with existing Thai embedding models, and identify the high ratio of out-of-vocabulary words as one of the biggest challenges. All datasets, evaluation results, and a tool for easy evaluation of new Thai embedding models are available to the NLP community online.