Building a Semantic Role Labelling System for Vietnamese

TL;DR

This paper introduces the first semantic role labeling system for Vietnamese, achieving an F1 score of 73.53%, by developing new algorithms and features tailored to Vietnamese language processing.

Contribution

The paper presents a novel SRL system for Vietnamese, including a new candidate extraction algorithm and language-specific features, establishing a baseline for future research.

Findings

Achieved 73.53% F1 score on Vietnamese PropBank corpus.

Developed a more accurate candidate extraction algorithm.

Proposed language-specific features for Vietnamese SRL.

Abstract

Semantic role labelling (SRL) is a task in natural language processing which detects and classifies the semantic arguments associated with the predicates of a sentence. It is an important step towards understanding the meaning of a natural language. There exists SRL systems for well-studied languages like English, Chinese or Japanese but there is not any such system for the Vietnamese language. In this paper, we present the first SRL system for Vietnamese with encouraging accuracy. We first demonstrate that a simple application of SRL techniques developed for English could not give a good accuracy for Vietnamese. We then introduce a new algorithm for extracting candidate syntactic constituents, which is much more accurate than the common node-mapping algorithm usually used in the identification step. Finally, in the classification step, in addition to the common linguistic features, we propose novel and useful features for use in SRL. Our SRL system achieves an $F_1$ score of 73.53\% on the Vietnamese PropBank corpus. This system, including software and corpus, is available as an open source project and we believe that it is a good baseline for the development of future Vietnamese SRL systems.

Tables

Table 1. TABLE I : Adjunct arguments in Vietnamese

Role Name	Description	Role Name	Description
ArgM-ADV	general-purpose	ArgM-CAU	cause
ArgM-DIS	discourse marker	ArgM-DIR	direction
ArgM-NEG	negation marker	ArgM-MNR	manner
ArgM-PRD	predication	ArgM-PRP	purpose
ArgM-MOD	modal verb	ArgM-TMP	temporal
ArgM-REC	reciprocal	ArgM-GOL	goal
ArgM-LVB	light verb	ArgM-EXT	extent
ArgM-COM	comitative	ArgM-I	interjection
ArgM-Partice	partice	ArgM-PNC	purpose
ArgM-ADJ	unknown	ArgM-RES	unknown

Table 2. TABLE II : Accuracy of two extraction algorithms

	1-1 Node Mapping	Our Extraction Alg.
Precision	29.53%	81.00%
Recall	45.60%	86.43%
F1	35.84%	83.63%

Table 3. TABLE III : Accuracy of baseline system

	1-1 node mapping	Our Extraction Alg.
Precision	52.80%	53.79%
Recall	3.30%	47.51%
F1	6.20%	50.45%

Table 4. TABLE IV : Accuracy of two labelling strategies

	ME	SVM
1-step strategy	50.45%	68.91%
2-step strategy	49.76%	68.55%

Table 5. TABLE V : Feature sets

Feature Set	Description
${\Phi }_1$	${\Phi }_0\cup${Function Tag}
${\Phi }_2$	${\Phi }_0\cup${Predicate Type}
${\Phi }_3$	${\Phi }_0\cup${Distance}

Table 6. TABLE VI : Accuracy of feature sets in Table  V

Feature Set	Precision	Recall	F1
${\Phi }_0$	72.27%	65.84%	68.91%
${\Phi }_1$	76.49%	69.65%	72.91%
${\Phi }_2$	72.26%	65.87%	68.92%
${\Phi }_3$	72.35%	65.86%	68.95%

Table 7. TABLE VII : Feature sets (continued)

Feature Set	Description
${\Phi }_5$	${\Phi }_4\setminus${Predicate} $\cup$ {Predicate Cluster}
${\Phi }_6$	${\Phi }_4\setminus${Head Word} $\cup$ {Head Word Cluster}
${\Phi }_7$	${\Phi }_4\setminus${Full Path} $\cup$ {Partial Path}

Table 8. TABLE VIII : Accuracy of feature sets in Table  VII

Feature Set	Precision	Recall	F1
${\Phi }_4$	76.60%	69.72%	73.00%
${\Phi }_5$	76.86%	70.36%	73.47%
${\Phi }_6$	72.50%	66.59%	69.41%
${\Phi }_7$	76.29%	69.58%	72.78%

Table 9. TABLE IX : Feature sets (continued)

Feature Set	Description
${\Phi }_9$	${\Phi }_8\setminus${Function Tag}
${\Phi }_{10}$	${\Phi }_8\setminus${Predicate Cluster}
${\Phi }_{11}$	${\Phi }_8\setminus${Head Word}
${\Phi }_{12}$	${\Phi }_8\setminus${Path}
${\Phi }_{13}$	${\Phi }_8\setminus${Position}
${\Phi }_{14}$	${\Phi }_8\setminus${Voice}
${\Phi }_{15}$	${\Phi }_8\setminus${Subcategorization}
${\Phi }_{16}$	${\Phi }_{10}\cap {\Phi }_{15}$

Table 10. TABLE X : Accuracy of feature sets in Table  IX

Feature Set	Precision	Recall	F1
${\Phi }_8$	76.86%	70.36%	73.47%
${\Phi }_9$	72.27%	66.12%	69.06%
${\Phi }_{10}$	76.87%	70.41%	73.50%
${\Phi }_{11}$	72.91%	67.05%	69.86%
${\Phi }_{12}$	76.81%	70.36%	73.44%
${\Phi }_{13}$	76.41%	70.21%	73.18%
${\Phi }_{14}$	76.85%	70.36%	73.46%
${\Phi }_{15}$	76.83%	70.51%	73.53%
${\Phi }_{16}$	76.70%	70.31%	73.36%

Equations

$$\underset{Who}{\underbracket{\text{Nam}}}gi\acute{u}p\underset{Whom}{\underbracket{\text{Huy}}}\underset{What}{\underbracket{\text{h@text@daccentoc baˋi}}}\underset{When}{\underbracket{\text{vaˋo hoˆm qua}}}$$

$$\underset{Arg0}{\underbracket{\text{Nam}}}gi\acute{u}p\underset{Arg1}{\underbracket{\text{Huy}}}\underset{Arg2}{\underbracket{\text{h@text@daccentoc baˋi}}}\underset{ArgM-TMP}{\underbracket{\text{vaˋo hoˆm qua}}}$$

$$\underbracket{\text{Nam}}gi\acute{u}p\underbracket{\text{Huy}}\underbracket{\text{h@text@daccentoc baˋi}}\underbracket{\text{vaˋo hoˆm qua}}$$

$$\underset{Cook}{\underbracket{\text{The boy}}}grills\underset{Food}{\underbracket{\text{their catches}}}\underset{Heating-instrument}{\underbracket{\text{on an open fire}}}$$

$$\underset{Arg0}{\underbracket{\text{The boy}}}grills\underset{Arg1}{\underbracket{\text{their catches}}}\underset{Arg2}{\underbracket{\text{on an open fire}}}$$

$$\text{Do}\underset{Arg1}{\underbracket{\text{ h@text@daccentoc cha˘m}}},\text{ Nam dja˜ dj@text@daccentat thaˋnh tıˊch cao.}$$

$$\text{Do h@text@daccentoc}\underset{Arg1}{\underbracket{\text{cha˘m, Nam}}}\text{ dja˜ dj@text@daccentat thaˋnh tıˊch cao.}$$

$$\text{Do}\underset{Arg1}{\underbracket{\text{h@text@daccentoc}}}\text{cha˘m},\underset{Arg1}{\underbracket{\text{Nam}}}\text{dja˜ dj@text@daccentat thaˋnh tıˊch cao.}$$

$$\underset{Arg0}{\underbracket{\text{Do h@text@daccentoc cha˘m, Nam}}}\text{dja˜ dj@text@daccentat}\underset{Arg0}{\underbracket{\text{thaˋnh tıˊch cao}}}.$$

$${\Phi }_4={\Phi }_0\cup \{\text{Function Tag}\}\cup \{\text{Distance}\}.$$

Full text

I Introduction

SRL is the task of identifying semantic roles of predicates in the sentence. In particular, it answers a question Who did What to Whom, When, Where, Why?. A simple Vietnamese sentence Nam giúp Huy học bài vào hôm qua (Nam helped Huy to do homework yesterday) is given in Figure 1.

To assign semantic roles for the sentence above, we must analyse and label the propositions concerning the predicate giúp (helped) of the sentence. Figure 2 shows a result of the SRL for this example, where meaning of the labels will be described in detail in Section IV.

SRL has been used in many natural language processing (NLP) applications such as question answering [1], machine translation [2], document summarization [3] and information extraction [4]. Therefore, SRL is an important task in NLP.

The first SRL system was developed by Gildea and Jurafsky [5]. This system was performed on the FrameNet corpus and was used for English. After that, SRL task has been widely researched by the NLP community. In particular, there have been two shared-tasks, CoNLL-2004 [6] and CoNLL-2005 [7], focusing on SRL task for English. Most of the systems participating in these share-tasks treated this problem as a classification problem and applied some supervised machine learning techniques. In addition, there were some systems developed for other languages such as Chinese [8] or Japanese [9].

In this paper, we present the first SRL system for Vietnamese with encouraging accuracy. We first demonstrate that a simple application of SRL techniques developed for English or other languages could not give a good accuracy for Vietnamese. In particular, in the constituent identification step, the widely used 1-1 node-mapping algorithm for extracting argument candidates performs poorly on the Vietnamese dataset, having $F_{1}$ score of 35.84%. We thus introduce a new algorithm for extracting candidates, which is much more accurate, achieving an $F_{1}$ score of 83.63%.

In the classification step, in addition to the common linguistic features, we propose novel and useful features for use in SRL, including function tags and word clusters obtained by performing a Gaussian mixture analysis on the distributed representations of Vietnamese words. These features are employed in two statistical classification models, Maximum Entropy and Support Vector Machines, which are proved to be good at many classification problems.

Our SRL system achieves an $F_{1}$ score of 73.53% on the Vietnamese PropBank corpus. This system, including software and corpus, is available as an open source project and we believe that it is a good baseline for the development of future Vietnamese SRL systems.

The paper is structured as follows. Section II introduces briefly the SRL task and two well-known corpora for English. Section III describes the methodologies of some existing systems and of our system. Some difficulties of SRL for Vietnamese are also discussed. Section IV presents the evaluation results and discussion. Finally, Section V concludes the paper and suggests some directions for future work.

II Background

II-A SRL Task Description

The SRL task is usually divided into two steps. The first step is argument identification. The goal of this step is to identify the syntactic constituents of a sentence which are the most likely to be semantic arguments of its predicates. This is a difficult problem since the number of constituent candidates is exponentially large, especially for long sentences.

The second step is argument classification which decides the exact semantic role for each constituent candidate identified in the first task. For example, the identification step of the sentence in the previous example Nam giúp Huy học bài vào hôm qua is described in Figure 3 and in the classification task, semantic roles are labelled as shown Figure 2.

II-B Existing Corpora for SRL

II-B1 FrameNet

The FrameNet project is a lexical database of English. It was built by annotating examples of how words are used in actual texts. It consists of more than 10,000 word senses, most of them with annotated examples that show the meaning and usage and more than 170,000 manually annotated sentences [10]. This is the most widely used dataset upon which SRL systems for English have been developed and tested.

FrameNet is based on the Frame Semantics theory [11]. The basic idea is that the meanings of most words can be best understood on the basis of a semantic frame: a description of a type of event, relation, or entity and the participants in it. All members in semantic frames are called frame elements. For example, a sentence in FrameNet is annotated in cooking concept as shown in Figure 4.

II-B2 PropBank

PropBank is a corpus that is annotated with verbal propositions and their arguments [12]. PropBank tries to supply a general purpose labelling of semantic roles for a large corpus to support the training of automatic semantic role labelling systems. However, defining such a universal set of semantic roles for all types of predicates is a difficult task; therefore, only Arg0 and Arg1 semantic roles can be generalized. In addition to the core roles, PropBank defines several adjunct roles that can apply to any verb. It is called Argument Modifier. The semantic roles covered by the PropBank are the following:

• •

Core Arguments (Arg0-Arg5, ArgA): Arguments define predicate specific roles. Their semantics depend on predicates in the sentence.

• •

Adjunct Arguments (ArgM-): General arguments that can belong to any predicate. There are 13 types of adjuncts.

• •

Reference Arguments (R-): Arguments represent arguments realized in other parts of the sentence.

• •

Predicate (V): Participant realizing the verb of the proposition.

For example, the sentence of Figure 4 can be annotated in the PropBank role schema as shown in Figure 5.

III Methodology

III-A Existing Approaches

This section summarizes existing approaches used by typical SRL systems for well-studied languages. We describe these systems by investigating two aspects, namely data type that the systems use and their strategies for labelling semantic roles, including model types, labelling strategies and degrees of granularity.

III-A1 Data Types

There are some kinds of data used in the training of SRL systems. Some systems use bracketed trees as the input data. A bracketed tree of a sentence is the tree of nested constituents representing its constituency structure. Some systems use dependency trees of a sentence, which represents dependencies between individual words of a sentence. The syntactic dependency represents the fact that the presence of a word is licensed by another word which is its governor. In a typed dependency analysis, grammatical labels are added to the dependencies to mark their grammatical relations, for example nominal subject (nsubj) or direct object (dobj). Figure III-A1 shows the bracketed tree and the dependency tree of an example sentence.