A Trie-Structured Bayesian Model for Unsupervised Morphological Segmentation

TL;DR

This paper presents a trie-structured Bayesian model for unsupervised morphological segmentation that integrates neural embeddings and letter successor information, improving segmentation accuracy across multiple languages.

Contribution

It introduces a novel Bayesian model combining neural embeddings and trie-based priors for unsupervised morphological segmentation.

Findings

Outperforms existing models on Turkish morphology segmentation

Achieves promising results on English and German with limited resources

Utilizes neural embeddings and letter successor counts to enhance segmentation accuracy

Abstract

In this paper, we introduce a trie-structured Bayesian model for unsupervised morphological segmentation. We adopt prior information from different sources in the model. We use neural word embeddings to discover words that are morphologically derived from each other and thereby that are semantically similar. We use letter successor variety counts obtained from tries that are built by neural word embeddings. Our results show that using different information sources such as neural word embeddings and letter successor variety as prior information improves morphological segmentation in a Bayesian model. Our model outperforms other unsupervised morphological segmentation models on Turkish and gives promising results on English and German for scarce resources.

Figures

Tables

Table 1. Table 1: Size of the datasets used in the experiments. m1 denotes the train set built by the first method (Section  3.1 ) and m2 denotes the train set built by the second method (Section  3.2 )

Language	Train-m1	Train-m2	Train word2vec	Test
Turkish	43884 types	67292 types	725K types	1760 types
			361M tokens
English	2560 types	34594 types	218K types	1686 types
			129M tokens
German	13747 types	23875 types	608K types	1779 types
			651M tokens

Table 2. Table 2: Results obtained from the tries based on semantic relatedness (see Section  3.2 ). TST denotes our trie-structured model.

	TURKISH
	Precision (%)	Recall (%)	F-measure (%)
TST	58.27	35.55	44.16
M-CatMAP	77.78	22.91	35.40
M-Baseline	84.39	19.27	31.38
MorphoChain	69.45	18.29	28.95
	ENGLISH
M-Baseline	64.82	64.07	64.44
TST	56.40	47.90	51.81
MorphoChain	86.26	25.95	39.90
M-CatMAP	76.37	19.23	30.72
	GERMAN
M-Baseline	64.74	30.10	41.09
TST	38.66	38.57	38.61
M-CatMAP	62.32	15.68	25.06
MorphoChain	56.39	13.72	22.07

Table 3. Table 3: Results obtained from the tries structured from the same stem (see Section  3.1 ). TST denotes our trie-structured model.

	TURKISH
	Precision (%)	Recall (%)	F-measure (%)
M-CatMAP	59.44	33.41	42.78
TST	58.85	30.17	39.89
M-Baseline	74.09	20.52	32.14
Morpho-Chain	72.28	25.77	38.00
	ENGLISH
M-Baseline	75.28	61.05	67.42
TST	58.69	51.22	54.70
MorphoChain	91.74	30.39	45.66
M-CatMAP	90.20	5.86	11.00
	GERMAN
M-Baseline	59.65	29.47	39.45
TST	39.62	35.28	37.33
MorphoChain	79.06	16.36	27.11
M-CatMAP	55.96	16.41	25.38

Equations

$$p(Model|Corpus)\propto p(Corpus|Model)p(Model)$$

$$p(Corpus|Model)$$

$$m_{ij}$$

$$H(m_{ij})$$

$$p(m_{ij}=k|Model)=\frac{n_k+\alpha H(k)}{N+\alpha }$$

$$p(Model)$$

$$p(b_{ij})$$

$$p({b_{ij}}_{branch})$$

$$=$$

$$p({b_{ij}}_{semantics})$$

$$p({b_{ij}}_{presence})$$

$$p(w_i=m_{i1}+m_{i2}|Corpu{s}^{-w_i},Mode{l}^{-w_i},\alpha ,\lambda ,\gamma )$$

$$\propto p(m_{i1}|Mode{l}^{-w_i},\alpha ,\gamma )p(m_{i2}|Mode{l}^{-w_i},\alpha ,\gamma )p(b_{i1})$$

$$\arg \underset{m_{i1},\cdots ,m_{i{t}_i}}{\max }p(w_i=m_{i1}+\cdots +m_{i{t}_i}|Model,\alpha ,\gamma )$$

Full text

1 Introduction

Morphological segmentation is the task of segmenting words into their meaningful units called morphemes. For example, the word transformations is split into trans, form, ation, and s. This process serves mainly as a preprocessing task in many natural language processing (NLP) applications such as information retrieval, machine translation, question answering, etc. This process is essential because sparsity becomes crucial in those NLP applications due to morphological generation that produces various word forms from a single root. It is infeasible to build a dictionary that involves all possible word forms in a language in order to use in an NLP application. Hankamer [14] suggests that the number of possible word forms in an agglutinative language such as Turkish is infinite. Therefore, instead of building a model based on word forms, morphological segmentation is applied to reduce the sparsity principally in any NLP application.

Various features have been used for morphological segmentation. Many approaches use orthographic features. However, morphology is tightly connected with syntax and semantics. Syntactic and semantic features have also been used for the segmentation task.

Features are normally used in Bayesian models in the form of a prior distribution. For example, [7] utilize frequency and length information of morphemes as prior information, which provide some orthographic features.

In this paper, we aggregate prior information from different sources in morphological segmentation within a Bayesian framework. We use orthographic features such as letter successor variety (LSV) counts obtained from tries, semantic information obtained from the neural word embeddings [17] to measure the semantic relatedness between substrings of a word, and we use the presence information of a stem in a dataset after its suffixes are stripped off assuming a concatenative morphology. Our results show that combining prior information from different sources give promising results in unsupervised morphological segmentation.

In this study, we learn tries based on semantic and orthographic features. Therefore, the output of our model is not only segmentation, but also tries that are composed of semantically and morphologically related words.

The paper is organized as follows: Section 2 presents the previous work on unsupervised morphological segmentation, section 3 defines the mathematical model, section 5 describes the inference algorithm to learn the mathematical model, section 7 presents the experimental results, and finally section 8 concludes the paper with a discussion and potential future work.

2 Related Work

Morphological segmentation, as one of the oldest fields in NLP, has been excessively studied. Deterministic methods are the oldest ones used in morphological segmentation. Harris [15] defines the distributional characteristics of letters in a word for the first time for unsupervised morphological segmentation. LSV model is named after Harris, which defines the morpheme boundaries based on letter successor counts. If words are inserted into a trie, branches correspond to potential morpheme boundaries. An example is given in Figure 1. In the example, re- is a potential prefix, and -s, -ed and -ing are potential suffixes in the trie due to branching that emerges before those morphemes. LSV model has been applied in various works [13, 11, 1, 2, 3]. In our study, we also use a LSV-inspired prior information, but this time in a Bayesian framework.

Stochastic methods have also been extensively used in unsupervised morphological segmentation. Morfessor is the name of the family of a group of unsupervised morphological segmentation systems which are all stochastic [8, 10, 9]. Non-parametric Bayesian models have also been applied in morphological segmentation [12, 19, 5].

Neural-inspired features are used in the recent studies. Narasimhan et al. [18] use semantic similarity obtained from neural word embeddings by word2vec [17]. Narasimhan et al. [18] adopt the semantic similarity as a feature in a log-linear model. Soricut and Och [20] use word embeddings to learn morphological rules in an unsupervised setting.

In this work, we are both inspired by the oldest works and the recent works in terms of various features used. Thus our model has inspirations from the LSV methods, stochastic methods, and neural-based models in a combined framework.

3 Building Neural Word Embedding-based Tries

Our model is based on neural word embedding-based tries that are built by using two different methods:

3.1 Tries Structured from the Same Stem

These tries contain semantically related (morphologically derived or inflected from each other) words having the same stem. In order to find the stem of a given word in the training set, we used the algorithm which is introduced in [21]. In the algorithm, all potential prefixes of a word are extracted. For example, fe, fea, fear, fearf, fearfu, fearful, fearfull, fearfully are the prefixes of fearfully. The rightmost segmentation point where the cosine similarity between the word and the first prefix (from the right of the word; i.e. fearful) is higher than a manually set threshold111We assign 0.25 as the threshold following [21]. gives the first valid prefix, which refers to the first segmentation point.

Other segmentation points are found by repeating the process towards the head of the word by checking the cosine similarity between the just detected valid prefix and the subsequent prefix to the left of the word. The final detected prefix with the leftmost segmentation point in the word becomes the stem of the word.

Among the nearest $50$ neighbors of the stem which are obtained from word2vec [17], the ones that begin with the same stem are inserted to the same trie. This process is repeated for each word that is inserted on the trie recursively until all the words that are semantically similar which share the same stem (detected by using the same algorithm described above) are covered. An example trie that is built with the words having the same stem is given in Figure 2.

3.2 Tries Based on Semantic Relatedness

Semantically related 50 words are retrieved for each word in the training set by using word2vec [17]. For each word, a trie is built and 50 similar words are inserted on the word’s trie. Eventually, a trie that consists of 51 words is created for each word in the training set. A portion of a trie that involves semantically related words is given in Figure 3.

4 Bayesian Model Definition

We define a Bayesian model in order to find the morpheme boundaries on the tries:

[TABLE]

where $Corpus$ is a list of raw words and $Model$ denotes the segmentation of the corpus. The $Model$ that maximizes the given posterior probability will be searched for the segmentation task. We apply a unigram model for the likelihood:

[TABLE]

where $w_{i}$ is the ith word in $Corpus=\{w_{1},\cdots,w_{|W|}\}$, $m_{ij}$ is the jth morpheme in $w_{i}$, $t_{i}$ is the number of morphemes in word $w_{i}$, and $|W|$ is the number of words in the corpus. Here, morphemes are generated by a Dirichlet Process (DP) as follows:

[TABLE]

with the concentration parameter $\alpha$ and the base distribution $H$ that is formed with a geometric distribution:

[TABLE]

Here, $|m_{ij}|$ is the length of $m_{ij}$ and $\gamma$ is the parameter of the geometric distribution. We assume that each letter is uniformly distributed. Therefore, we assign $\gamma=1/L$ where $L$ denotes the size of the alphabet in the language. Shorter morphemes will be favored with the usage of length-inspired base distribution in the DP. From the Chinese Restaurant Process (CRP) perspective, each morpheme is generated proportionally to the number of morphemes of the same type that have already been generated (i.e. customers having the same dish):

[TABLE]

This computes the probability of $m_{ij}$ being of type $k$ where $k$ refers to a distinct morpheme (i.e. morpheme type). Here, $n_{k}$ is the number of morphemes of type $k$ and $N$ is the total number of morpheme tokens in the model. We generate each morpheme regardless of its type, such as stem, prefix, or suffix.

As for the prior information, we model the morpheme boundaries:

[TABLE]

Here, $b_{ij}$ refers to the jth morpheme boundary in $w_{i}=m_{i1}+m_{i2}+\cdots+m_{it_{i}}$ where $w_{i}=\{b_{i1},b_{i2},\cdots,b_{it_{i}}\}$.

The probability of each $b_{ij}$ is decomposed in terms of the number of branches leaving that node (when inserted on the trie), semantic similarity that is introduced between the two word forms that is split with $b_{ij}$, and the presence of the word form once the suffix is stripped off from the word:

[TABLE]

where $p({b_{ij}}_{branch})$ denotes the probability of $b_{ij}$ being a morpheme boundary based on the branches leaving the trie node, $p({b_{ij}}_{semantics})$ is based on the semantic similarity of the two word forms where $b_{ij}$ separates the two forms, and $p({b_{ij}}_{presence})$ is estimated based on the word form whether it exists in the corpus once the suffix after $b_{ij}$ is stripped off.

Based on the LSV, the branching on the tries corresponds to the potential morpheme boundaries. We model the branching with a Poisson distribution:

[TABLE]

where $z_{ij}=l$ denotes the number of branches leaving the node below $b_{ij}$ and $\lambda$ is the parameter of the Poisson distribution222In the experiments, we assign $\lambda=4$..

We use the cosine similarity (which is always between 0 and 1) between the neural word embeddings of the two word forms that are separated by $b_{ij}$ for the semantic distribution:

[TABLE]

Here, $x_{m_{i1}+\cdots+m_{ij}}$ corresponds to the word vector of the word form $m_{i1}+\cdots+m_{ij}$ obtained from word2vec. It is the full word vector and not the compositional vector obtained from morpheme vectors.

As for the presence of the word form in the word list, we compute the likelihood of the word form $m_{i1}+\cdots+m_{ij}$:

[TABLE]

where $f(m_{i1}+\cdots+m_{ij})$ denotes the frequency of the word form in the corpus.

5 Inference

We use Gibbs sampling [6] for the inference. In each iteration, a word is uniformly selected from any trie and removed from the corpus. A binary segmentation of the word is sampled from the given posterior distribution:

[TABLE]

Once a binary segmentation is sampled, another binary segmentation is sampled for $m_{i1}$. Therefore, a left-recursion is applied for the left part of the word. This is because of the cosine similarity that is computed between neural word embeddings of word forms and not suffixes by the original word2vec.

This process is repeated recursively until having at least 4 letters in the stem or having sampled the word itself from the posterior distribution (i.e. when the word is not segmented). An illustration is given in Figure 4.

6 Segmentation

Once the model is learned, any unseen word can be segmented by using the learned model. Each word is split based on the maximum likelihood in the learned model:

[TABLE]

For the segmentation, we apply two different strategies. In both methods, we select the segmentation with the maximum likelihood, however the set of possible segmentations for the given word differs. In the first method, we only consider the segmentations learned by the model. Since the same word can exist in multiple tries, a word may have more than one different segmentation. In the second method, we consider all possible segmentations of a word and choose the one with the maximum likelihood.

7 Experiments and Results

We did experiments on Turkish, English and German. For each language, we built two sets of tries based on the methods described in Section 3.1 and Section 3.2. We aggregated the publicly available training and development sets provided by Morpho Challenge 2010 [16] for English, Turkish and German for training. Although gold segmentations are provided in the datasets, we only used the raw words in training. Gold segmentations were only used for evaluation purposes.

We began with 1686 English words, 1760 Turkish words, and 1779 German words obtained from the aggregated sets. Once the tries have been built by recursively augmenting the tries by using word2vec[17], eventually we obtained 2560 English word types, 43884 Turkish word types, and 13747 German word types in the tries structured from similar stems (see Section 3.1). Additionally, we obtained 34594 English word types, 67292 Turkish word types, and 23875 German types in the tries that were built based on the semantic relatedness (see Section 3.2).

We used 200-dimensional word embeddings that were obtained by training word2vec [17] on 361 million word tokens and 725.000 word types in Turkish, 129 million word tokens and 218.000 word types in English, and 651 million word tokens and 608.000 word types in German. The size of all datasets used in the experiments are given in Table 1.

We compared our model with Morfessor Baseline [8] (M-Baseline), Morfessor CatMap [9] (M-CatMAP) and MorphoChain System [18]. For that purpose, we trained these models on the same training sets. We obtained the frequency information from the full word lists provided by Morpho Challenge which was need by other systems. The evaluation was performed on the aggregated training and development sets of Morpho Challenge 2010 using the Morpho Challenge evaluation method [16]. All word pairs that have a common morpheme are extracted from the results and checked whether they really share a common morpheme in the gold standard data. One point is given for each correct pair. The Precision is the proportion of the collected points to the total number of words. Recall is computed analogously. This time all word pairs that share a common morpheme are extracted from the gold standard data and checked whether they have a common morpheme in the results. For each correct pair, one point is given. Finally, the Recall is the proportion of the collected points is to the total number of words.

The results are given in Table 2 and Table 3 for tries that are composed of words structured from the same stem (see Section 3.1) and for tries that are based on semantic relatedness (see Section 3.2). According to the results, tries that contain semantically similar words achieve a better performance on morphological segmentation proving that semantically similar words also manifest similar syntactic and thus similar morphological features.

Our trie-structured model (TST) performs better than Morfessor Baseline [8], Morfessor CatMAP [9] and Morphological Chain [18] on Turkish with a F-measure of %44.16 on the tries based on semantic relatednesss. We obtained a F-measure of %39.89 for Turkish from the tries structured from the same stem, which is poorer than the other method. This shows that for morphologically rich languages, semantic relatedness plays a more important role in segmentation. That is because of the sparseness of the word forms in morphologically rich languages. Here we overcome the sparsity problem with semantic information that is used in semantically built tries.

Our TST model performs better on the tries structured from the same stem on English with a F-measure of %54.70 compared to the tries based on semantic relatedness, which has a F-measure of %51.81. Since English is not a morphologically rich language, obtaining the correct stem plays an important role in segmenting the word. Words usually do not have more than one suffix and therefore finding the stem is normally sufficient to do morphological segmentation in morphologically poor languages such as English.

Our German results are close to each other obtained from two types of tries. We obtain a F-measure of %38.61 from the tries based on semantic relatedness and it performs better than Morfessor CatMAP and Morphological Chain. The F-measure is %37.33 on German, which is obtained from the tries structured from the same stem.

The results also show that Morfessor CatMAP suffers from sparsity in small datasets (especially in English), whereas our trie-structured model learns also well in small datasets.

8 Conclusion and Future Work

We propose a Bayesian model that utilizes semantically built trie structures that are built by using neural word embeddings (i.e. obtained from word2vec [17]) for morphological segmentation in an unsupervised setting. The current study constitutes the first part of the on-going project which in the end aims to learn part-of-speech tags and morphological segmentation jointly. To this end, the fact that the tries having semantically related words achieves the best performance paves the way of using semantically similar words in learning syntactic features.

Moreover, considering the resource-scarce languages like Turkish, our trie-structured model shows a good performance on comparably smaller datasets. In comparison to other available systems, our model outperforms them in spite of the limited training data. This shows that the small size of data can be compensated to a certain extent with structured data, that is the main contribution of this paper.

Acknowledgments

This research is supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with the project number EEEAG-115E464.