A weakly supervised sequence tagging and grammar induction approach to semantic frame slot filling

TL;DR

This paper presents a weakly supervised method for semantic frame slot filling that improves accuracy and induces interpretable grammars using hierarchical models and inductive techniques.

Contribution

It introduces a novel combination of hierarchical hidden Markov models with discriminative and grammar induction approaches for slot filling.

Findings

Significant F-score improvements without extra data

Automatic induction of domain-specific, interpretable grammars

Effective weakly supervised learning for semantic tasks

Abstract

This paper describes continuing work on semantic frame slot filling for a command and control task using a weakly-supervised approach. We investigate the advantages of using retraining techniques that take the output of a hierarchical hidden markov model as input to two inductive approaches: (1) discriminative sequence labelers based on conditional random fields and memory-based learning and (2) probabilistic context-free grammar induction. Experimental results show that this setup can significantly improve F-scores without the need for additional information sources. Furthermore, qualitative analysis shows that the weakly supervised technique is able to automatically induce an easily interpretable and syntactically appropriate grammar for the domain and task at hand.

Full text

1 Introduction

The basic task of a command and control (C&C) interface is to convert a language user’s unstructured input into some structured format that can be unambiguously interpreted by a controller. Except for the simplest of devices, this requires at some point in the processing chain some kind of syntactic component that is aware of such issues as compositionality and word order. For example, in the home automation C&C phrase turn on the lights, the system needs to at least be aware that the input language is SVO to trigger the difference between action and patient. In many cases, this is done through the use of a context-free grammar in the decoding process [Jurafsky et al., 1995], as illustrated in Figure 1 for the task of controlling a television using natural language.

While this approach has proved to be quite effective for many applications, such devices are limited in the sense that they require the user to adhere to the grammar and lexicon as predefined by the designers of the device. This paper describes research conducted in the context of the ALADIN project, which investigates a command and control interface that is not bound by predefined linguistic constraints, but rather adapts to the user in a language and domain independent way. The aim of the project is to find techniques that minimize the training phase for such a self-learning system, so that it can be easily deployed as a home automation C&C system for people with a physical impairment, who are often challenged by (progressively aggravating) speech impediments. A self-learning C&C interface that is able to automatically learn and adapt to an individual user’s linguistic characteristics, can provide an important means for a physically impaired person to regain some independence in a domestic setting.

In this paper, we will investigate the syntactic component of the ALADIN-system, where we define “syntax” loosely as a means to interface between the surface utterance and its underlying semantic representation. Our semantic representation of choice is that of a semantic frame with slots that need to be filled with values (Section 2). Previous research efforts show that this can be done with a fair degree of accuracy using a hierarchical hidden markov model architecture (Section 3) under a minimal amount of supervision. The research described in this paper improves on this accuracy, through means of retraining using two different techniques: a shallow approach using sequence labeling (Section 4.1) and a more traditional parsing approach employing stochastic context-free grammar induction (Section 4.2). We will quantitatively and qualitatively evaluate these techniques on the basis of comparative experiments (Section 5) and finish with a discussion of the results and pointers to future work (Section 6).

2 The patcor Corpus

The experiments described in this paper were conducted on the patcor corpus111https://github.com/clips/patcor. patcor is a collection of command and control utterances for the single-player card game Patience. This control task was chosen because the disambiguation of these types of sentences requires awareness of compositionality and word order, while at the same time the domain and vocabulary are fairly restricted. The corpus contains audio recordings of nine players, totaling around 3000 spoken commands (Belgian Dutch) and their associated action in the game (represented as a semantic frame; cf. infra). All of the spoken commands were manually transcribed.

patcor consists of a core data set of eight speakers, for which around 250 utterances were collected. We will refer to this core set as the p1-8 set. Additional data was gathered for one player to study the effect of increasing training sizes. This data set, referred to as the p9 set contains over 1,000 utterances. In the experiments below, p1-8 is used for development purposes, while p9 is used for evaluation in learning curve experiments. More details about the data set and the command structures that were used by the speakers are described in ?).

The semantic representation of the controls in patcor takes the form of semantic frames. This is illustrated in Figure 2: the semantic frame describes the card that is moved (as a suit/value pair), the card that it is moved onto, as well as the respective columns that are involved in the move. Additional slots are available for moves from the hand and the foundation (respectively at the bottom and top of Figure 2). An action on the playing field is thus described as a collection of slot-value pairs. There are two types of semantic frames: the movecard frame exemplified in Figure 2 and an additional dealcard frame that denotes the action of asking for a new hand of cards. The latter frame has no slots or values to be filled.

Note that the automatically generated semantic frame is overspecified with respect to the command: in the command in Figure 2, columns are not mentioned, although this information is included in the automatically generated frame. We will use the utterances and their associated frames as training material to trigger the mapping from words onto slot-value pairs. The difficulty for the induction task therefore lies in identifying which words can be linked to which slots and associated values. Furthermore, a word might underspecify its slot-value in the semantic frame: for instance in Figure 2, the word “rood” (red) can refer to suits hearts and diamonds alike, while only the former is represented in the associated semantic frame.

3 Related Research

In this section we will provide some pointers to related research on mapping tokens to semantic frames in C&C interfaces. We will also discuss previous research efforts on the patcor data using two different techniques.

3.1 Semantic frame induction

There are many studies in spoken language understanding that use semantic frames or an equivalent to encode the meaning of utterances. Mapping new utterances to these semantic representations has been attempted using a variety of techniques, such as generatively associating word sequences with concept sequences [Pieraccini et al., 1991, Wang et al., 2011], using a discriminative technique to perform concept tagging [Hahn et al., 2011], or applying alignment techniques from the field of statistical machine translation to relate words with semantic concepts [Epstein et al., 1996, Macherey et al., 2001].

The research described in this paper differs from these previous approaches in that our semantic representation is overspecified with respect to the utterance and that we do not assume the ready availability of gold-standard alignments between tokens and concepts. With respect to the latter issue, our research is more akin to the unsupervised alignment of utterances and concepts in the atis corpus, as described in ?). The patcor task itself on the other hand, is more similar to that of the ROBOCUP data [Chen and Mooney, 2008], for which a variety of approaches have been coined, ranging from learning the alignment between utterances and frames [Liang et al., 2009] to learning semantic parsers [Chen et al., 2010].

3.2 Supervised concept tagging

Previous work on patcor data is described in ?) and ?). These papers approach the problem of inducing semantic frame slots from (transcribed) utterances as a concept/sequence labeling task that can be learned by a machine learning classifier. To this end, the transcriptions of patcor were manually annotated with IOB-concept tags, where each word is associated with a concept tag that denotes its association to a specific type of frame slot and its relative position in the text chunk (Inside, Outside or at the Beginning). This is illustrated in the following command (English: put the hearts five on the clubs six) which denotes a move from a $<$From_Suit$>$ $<$From_Value$>$ card to a $<$Target_Suit$>$ $<$Target_Value$>$ card.

{exe}\ex

legO deO hartenI_FS vijfI_FV opO deO klaverenI_TS zesI_TS

The eight data sets of the p1-8 set were each used individually as training data for a memory-based concept tagger and learning curves were plotted using the last 50 utterances in the respective data sets. Almost all of the data sets achieve an I-chunk F-score between 95% and 100% after around 130 training utterances, underlining the learnability of the task. However, this approach relies on the existence of a gold-standard mapping between the observed tokens and their relevant frame slots and values. This type of supervision is not available in a C&C interface that is to be trained through the association of an utterance in its entirety with an (overspecified) semantic frame. Furthermore, [van de Loo et al., 2012a, van de Loo et al., 2012b] only report tag and chunk accuracy scores, which only gives us an indirect indication of how well semantic frame slots and values can be filled by this technique. In the experiments below, we consider this supervised sequence labeler as the upperbound of the level of performance that can be expected for this data set.

3.3 FramEngine

Research on the acoustic part of patcor focuses on finding descriptive word units in the utterances, using non-negative matrix factorization. To map observed units to frame slots (frame decoding), early research [Gemmeke et al., 2012] employed a hand-crafted grammar, which encoded expected word order patterns. This approach obtained slot-value recall scores between 37% and 85% on the p1-8 set. One of the major bottlenecks in achieving higher accuracies was the method’s limited ability to build a robust acoustic representation on the basis of a limited amount of training data.

To streamline frame decoding, a semantic frame induction engine was developed (FramEngine) that uses a combination of non-negative matrix factorization and a hierarchical hidden markov model (HHMM) to simultaneously detect recurring patterns in commands (words, morphemes, acoustic units) and relate them to their relevant slots and values in semantic frames. This is a weakly supervised induction task, as there is only supervision at the utterance level and no relations between the components of the utterances and the semantic frame slots are specified in advance.

Previous work [Ons et al., 2013] demonstrated the performance of an early implementation of this system on pathological speech input for a simple home automation command & control task and non-pathological patcor speech data. The results show that the system has a promising learning potential even with small amounts of training data, but that enhancements are needed in order to produce practically usable accuracies for more complex utterances, such as the ones found in patcor. Work described in ?) focused on improvements with respect to the system architecture. Extensions regarding the HMM structure and a technique called “expression sharing” were added to FramEngine’s workings and were shown to significantly improve on the frame-slot filling abilities on transcribed patcor data.

?) describes parameter selection experiments on p1-8, while a final evaluation was conducted on the basis of learning curve experiments on the p9 set. FramEngine was able to score over 95% frame-slot F-score using around 200 utterances as training data. While these results are encouraging, there are reasons to believe we can still increase accuracy by means of retraining. FramEngine’s slot-value allocation technique is limited in the context that it can consider during disambiguation. And while word order is indirectly modeled through the structural setup of the hierarchical hidden markov model, a further explicitation of the syntactic properties of the observed utterances may provide additional insights into the induction process proper.

In the context of this paper, we refer to retraining as the process of using an automatically labeled data set as input data for another inductive processing step. Typically, retraining techniques attempt to label large volumes of unlabeled data to increase the volume of available training data (self-training [McClosky et al., 2006]) or adapt a classifier to a new domain (co-training [Blum and Mitchell, 1998]). The work in this paper investigates retraining as the process of using successive classifiers on the same training data to improve results. This approach has been successfully applied to a variety of problems, including morphological analysis [De Pauw and de Schryver, 2008] and spoken language understanding [Wu et al., 2006].

4 Retraining

This paper explores two different retraining techniques: inductive sequence labeling (Section 4.1) and stochastic context-free grammar induction (Section 4.2). As described in ?), FramEngine can be shown to exhibit a more than decent performance on the task of frame-slot filling for the patcor data. For retraining purposes, we can use FramEngine to decode its own training set, linking each word in the training set with a frame slot/value label through Viterbi decoding of the utterance. This means we can now also associate each word with an automatically assigned IOB-tag, expressing its frame-slot and in addition also its value:

{exe}\ex

legO deO hartenI_FS=h vijfI_FV=5 opO deO klaverenI_TS=c zesI_TS=6

This information is consequently used as source data for retraining purposes.

4.1 Sequence Labeling

The first type of retraining technique recalls the research described in Section 3.2: words annotated with IOB-tags are considered as training material for a discriminative sequence tagger. Fully supervised (i.e. with manually attributed IOB-tags), this approach can be observed to obtain near perfect accuracy scores on the patcor data. Using training data labeled by a weakly supervised technique (i.e. FramEngine), a sequence labeler can be expected to perform less accurately than the fully supervised approach, but more accurately than FramEngine itself, since it can take into account more contextual information. We consider two different machine learning techniques driving the sequence labeler: we compare a memory-based tagger (mbt [Daelemans et al., 2010]) with one based on conditional random fields (wapiti [Lavergne et al., 2010]).

Figure 3 displays learning curves for exploratory experiments, in which we compared fully supervised sequence labeling (using gold-standard labels), weakly supervised decoding using FramEngine, and retrained sequence labelers. As expected, the accuracy of the retrained sequence labeler eventually ends up somewhere between that of the supervised labeler and that of FramEngine. But this result is not obtained until rather late in the learning curve, and even the fully supervised labeler trails compared to the weakly supervised FramEngine until training data hits around 400 utterances.

This is due to the fact that the sequence labelers consider the IOB-tags holistically. FramEngine as described in ?), uses the method of expression sharing, in which the hidden markov model is aware of the fact that certain frame slots (such as the suit or value of a card) may be expressed using the same tokens, regardless of whether it occurs in a from-clause or a target-clause. This means that if a certain value has only occurred in one particular context in the training set (e.g. vijfI_FV=5), the tagger would not be able to predict the correct tag when it occurs in the other context in held-out data (e.g. vijfI_TV=5), as it is simply unaware this tag even exists.

We therefore implement a two-step tagging approach that implicitly introduces expression sharing in the workings of the sequence labelers. The process is illustrated in Figure 4. The tags are decomposed into two parts: first a tagger attributes to the words slot-value tags that generalize over slots of the same type (in this case: ignoring ‘from’ and ‘target’ information). In the second step, a tagger is trained using the slot-portion of the tag from step 1 as the token to be tagged and the remainder of the original tag as the target tag. The original word in the utterance is used as an extra information layer towards disambiguation. The tags of both steps are then combined into the conglomerate tag. In case of incompatible tag-parts, the word is attributed the O-tag. We will use the two-step tagging approach in the experiments described in Section 5.

4.2 PCFG Induction

The utterances tagged automatically by FramEngine (cf. Example 4) can also be transformed into tree-structures, given information about which expressions can be shared. Slot-values (=? in Example 4) can be considered as terminals, while suit/value and from/target indicators can function as nonterminals. Two consecutive clauses headed by the same nonterminal are joined. As such, the tagged sentence in Example 4 can be algorithmically transformed into the phrase structure in Figure 5. These phrase structures can then be transformed into a (probabilistic) context free grammar (Figure 6), by observing the productions used to construct the tree structures for the utterances in the training set. Finally, a Viterbi parser is used to assign tree structures to new utterances and a reversal of the tree construction process outlined above, can be used to assign IOB-labels that refer to frame slot/value pairs.

5 Experimental Results

5.1 Experimental Setup

We compare four different approaches on the basis of their ability to fill slots with the correct values in semantic frames:

Fully supervised sequence labeling (mbt and wapiti) 2. 2.

FramEngine 3. 3.

Retraining using sequence labeling (mbt and wapiti) 4. 4.

Retraining using PCFG-induction

FramEngine fills frame slots by choosing slot-value tags with the highest summed probability observed during decoding, exceeding a certain threshold (established during development experiments). If no slot value tag with sufficient probability mass occurs in the utterance, the slot remains empty. Approaches 1, 3 and 4 fill frames in a similar vein, except that these techniques predict for each token in the utterance a single slot-value tag without a probability score (i.e. probability = 1). To solve ties, we add a small bias proportional to the time index to each single tag probability (t * 10-5).

5.2 Evaluation

An important requirement for the ALADIN system is that the amount of effort involved for the user in training the system be kept to a minimum. To evaluate our techniques in this vein, we perform learning curve experiments to see which techniques learn the required task the fastest with the least amount of training data. The experiments were performed on the p9 set, which consists of 1142 commands and corresponding action frames of one single speaker. The last third of the data set is used as evaluation data, while the remaining training data is divided into partitions of 50 utterances. We preserve the order of the training utterances as they have been recorded, to mimic how the actual system would be trained in practice.

The training partitions in the p9 set are used incrementally as training data for FramEngine. Optimal parameters and information sources for FramEngine were obtained during expansive experiments on the p1-8 [van de Loo et al., 2019]. The trained hierarchical hidden markov model is then used to transcode the training utterances into slot-value tag sequences. These automatically tagged utterances, derived through a weakly supervised induction process, are then used to train the memory-based and CRF-based sequence taggers and to induce a PCFG respectively. Finally, the retrained techniques are evaluated on the held-out data, creating a data point on the learning curve.

We evaluate against gold-standard semantic frames in which only the slots mentioned in the utterance have been specified (in contrast to the overspecified frame in Figure 2) and which slot values can be ambiguous: e.g. in the case of ’aas opzij leggen’ (‘put ace aside’), the reference frame includes all four possible foundation values (target_foundation=1,2,3,4), because no specific value is specified in the command. We use frame slot F-score as our metric of choice, which is calculated as follows:

• •

slot precision: #correctly filled slots / #total filled slots in predicted frame

• •

slot recall: #correctly filled slots / #total filled slots in reference frame

• •

slot Fβ=1-score: 2* slot precision * slot recall /(slot precision + slot recall)

Finally, as FramEngine’s performance is affected by random initializations of the system, the experiments with FramEngine and the retrained systems are carried out ten times, with different random initializations. Scores are averaged across the ten runs and 95% confidence levels for the scores are reported as well.

5.3 Results

Figure 7 displays the learning curves for the p9 data. We consider the fully supervised sequence labelers as the upper bound, since they are trained on gold-standard data. We notice that they achieve optimal F-score at around 250 utterances. Before that, their accuracy scores are marred by unseen words in the evaluation set, something that the weakly supervised FramEngine approach is able to overcome through its looser generalization properties. Retraining using the sequence labelers (Figure 7(a)), trained on the automatically tagged data, improves F-scores for nearly all training sizes over FramEngine. For this task, retraining using conditional random fields generally yields more favorable learning curves than using memory-based learning.

The large leaps in the curves in Figure 7 are to a great extent related to a shift in the user’s lexicon over time. In the test set (featuring recordings during the end of the data collection process), the card value ‘king’ is always referred to as ’heer’ (English: Sir), whereas initially, this card value is always referred to as ’koning’ (English: ‘king’). The word ’heer’ starts to appear in the training set after the 200th utterance, thereby causing a significant leap for all of the approaches.

Retraining through PCFG induction (Figure 7(b)) similarly suffers from out-of-vocabulary words and, additionally, unseen grammatical productions in the evaluation set on smaller training sizes. Its continuing performance is fairly constant, although it seldom outperforms FramEngine in a convincing manner. Figure 7(b) shows two PCFG-curves: one that expresses performance of 10 individual PCFGs for each FramEngine initialization (single run) and one that joins the data from all the runs and induces one single PCFG (all runs). Joining data over initializations dramatically improves the number of structures in the evaluation set that are covered by the grammars, while precision does not seem to suffer, thereby yielding higher F-scores for most training set sizes over 250.

These experiments show that, although FramEngine achieves workable F-scores using a weakly supervised approach, we can automatically increase F-scores through retraining. For this task, sequence labeling is the recommended retraining technique, although PCFGs can also be observed to increase F-scores for this task. The latter result is encouraging, since it allows for the weakly supervised induction of a classic command & control interface that restricts the acoustic search space by means of a context-free grammar.

5.4 PCFG induction: a qualitative analysis

While the F-scores of the PCFG-retraining approach trail compared to those of the retrained sequence labelers, the former presents an interesting opportunity for interpretability. Looking at the induced PCFGs can provide us with further insight into how the structural properties of the language are being modeled.

The grammar in Figure 8 corresponds rather well with our intuitions for this task. Apart from an artifact (production T$\rightarrow$TF cannot reach terminal nodes) and a needlessly ambiguous handling of the tokens nieuwe (new), kaarten (cards) and omdraaien (turn over), the induced PCFG presents a clean description of the grammar used in the utterances of the training set. The probabilities for the different card suits are evenly distributed, as are their card values. The two different frame types are triggered by three ROOT rules, two for the dealcard frame, one for the movecard frame. Interestingly, while the grammar in Figure 8 excludes infrequent rules (such as ROOT $\rightarrow$ F T O T O F O T), applying this kind of filtering in practice yields significantly lower F-scores during the experiments, as these productions provide coverage for idiosyncratic utterances containing repairs and the like.

6 Conclusion and Future work

This paper described research on data collected for a command & control interface for the card game Patience. The task of associating components of a command to slots in semantic frame representations of the intended control was already established with a fair degree of accuracy using a semantic frame induction technique based on non-negative matrix factorization and a hierarchical hidden markov model (FramEngine). This paper improves on this result by applying retraining using a discriminative sequence labeler as a post-processing step. Additional retraining experiments using probabilistic context-free grammar induction likewise show improvements when used as a post-processing step.

Next steps in this research should investigate whether these retraining techniques can yield similar accuracy increases when applied on the acoustic portion of the patcor data. It will be particularly interesting to see whether the PCFG induction technique can yield a similarly transparent collection of productions as displayed in Figure 8, when confronted with a noisy set of observations.

We will also investigate the advantage of the retraining techniques on other data sets, such as the grammatically less complex home automation data, collected in the context of the ALADIN project [Ons et al., 2013], but also the ROBOCUP data [Chen and Mooney, 2008], which in recent years has become an interesting benchmark to gauge advances in this field. Finally, further research efforts should investigate whether the retraining techniques, now used as a post-processing step, can be compiled into FramEngine so that decoding can be achieved in a single processing sweep.

7 Acknowledgments*

This research described in this paper was funded by IWT-SBO grant 100049 (ALADIN). The patcor dataset is available from https://github.com/clips/patcor. We would like to thank Jort Gemmeke, Hugo Van hamme and Bart Ons for their valued input on the research described in this paper.