Associative Measures and Multi-word Unit Extraction in Turkish

TL;DR

This study evaluates 12 associative measures on a large Turkish corpus to identify those that produce linguistically relevant rankings of word associations, focusing on corpus optimization and overall measure performance.

Contribution

It introduces a systematic evaluation of associative measures on Turkish data, emphasizing corpus preparation and aggregate ranking quality over individual n-gram relevance.

Findings

Identifies linguistically relevant associative measures for Turkish.

Highlights the importance of corpus optimization before applying measures.

Provides a comparative analysis of measure performance on Turkish text.

Abstract

Associative measures are "mathematical formulas determining the strength of association between two or more words based on their occurrences and cooccurrences in a text corpus" (Pecina, 2010, p. 138). The purpose of this paper is to test the 12 associative measures that Text-NSP (Banerjee & Pedersen, 2003) contains on a 10-million-word subcorpus of Turkish National Corpus (TNC) (Aksan et.al., 2012). A statistical comparison of those measures is out of the scope of the study, and the measures will be evaluated according to the linguistic relevance of the rankings they provide. The focus of the study is basically on optimizing the corpus data, before applying the measures and then, evaluating the rankings produced by these measures as a whole, not on the linguistic relevance of individual n-grams. The findings include intra-linguistically relevant associative measures for a comma delimited, sentence splitted, lower-cased, well-balanced, representative, 10-million-word corpus of Turkish.