Multiple topic identification in human/human conversations

X. Bost (LIA); G. Senay (LIA); M. El-B\`eze (LIA); R. De Mori (LIA)

arXiv:1812.07207·cs.CL·January 1, 2019

Multiple topic identification in human/human conversations

X. Bost (LIA), G. Senay (LIA), M. El-B\`eze (LIA), R. De Mori (LIA)

PDF

TL;DR

This paper presents a method for automatically identifying multiple themes in real-life telephone conversations between agents and customers, despite challenges like speech recognition errors and conversation unpredictability, to improve customer service analysis.

Contribution

It introduces a sequential decision strategy for sampling and analyzing conversations to accurately estimate theme proportions, handling high word error rates and conversation variability.

Findings

01

Effective theme detection despite ASR errors

02

Successful sampling strategy for large conversation datasets

03

Improved accuracy in customer problem statistics

Abstract

The paper deals with the automatic analysis of real-life telephone conversations between an agent and a customer of a customer care service (ccs). The application domain is the public transportation system in Paris and the purpose is to collect statistics about customer problems in order to monitor the service and decide priorities on the intervention for improving user satisfaction. Of primary importance for the analysis is the detection of themes that are the object of customer problems. Themes are defined in the application requirements and are part of the application ontology that is implicit in the ccs documentation. Due to variety of customer population, the structure of conversations with an agent is unpredictable. A conversation may be about one or more themes. Theme mentions can be interleaved with mentions of facts that are irrelevant for the application purpose. Furthermore,…

Tables10

Table 1. Table 1: Decoda corpus - Statistical data

Corpus	Train	Dev	Test	Total
#	880	196	576	1,654
#mono-label	744	146	424	1,314
#multi-label	136	50	154	340
label card.	1.16	1.29	1.29	1.22
%multi	15.45	25.51	26.64	20.56

Table 2. Table 2: Results obtained for the manual transcriptions of the dev set with the svm , the four systems and their linear combination (confidence interval = 0.03)

dev	man
	$a c c_{1}$	$a c c_{2}$	prec.	rec.	F-sc.
svm.	0.62	0.78	0.87	0.87	0.87
cos.	0.72	0.85	0.95	0.88	0.92
dens.	0.74	0.85	0.94	0.89	0.91
hmm	0.74	0.85	0.94	0.88	0.91
poiss.	0.71	0.84	0.92	0.88	0.90
comb.	0.77	0.87	0.97	0.89	0.93

Table 3. Table 3: Results obtained for the automatic transcriptions of the dev set with the svm , the four systems and their linear combination (confidence interval = 0.03)

dev	asr
	$a c c_{1}$	$a c c_{2}$	prec.	rec.	F-sc.
svm.	0.57	0.75	0.85	0.85	0.85
cos.	0.68	0.81	0.91	0.86	0.89
dens.	0.68	0.81	0.90	0.85	0.87
hmm	0.65	0.78	0.86	0.84	0.85
poiss.	0.66	0.80	0.90	0.85	0.87
comb.	0.69	0.82	0.92	0.86	0.89

Table 4. Table 4: Results obtained for the manual transcriptions of the test set with the svm , the four systems and their linear combination (confidence interval = 0.03)

test	man
	$a c c_{1}$	$a c c_{2}$	prec.	recall	F-sc.
svm.	0.60	0.75	0.85	0.83	0.84
cos.	0.67	0.81	0.90	0.85	0.88
dens.	0.64	0.78	0.87	0.84	0.85
hmm	0.66	0.80	0.89	0.87	0.88
poiss.	0.62	0.77	0.85	0.85	0.85
comb.	0.66	0.81	0.89	0.87	0.88

Table 5. Table 5: Results obtained for the automatic transcriptions of the test set with the svm , the four systems and their linear combination (confidence interval = 0.03)

test	asr
	$a c c_{1}$	$a c c_{2}$	prec.	recall	F-sc.
svm.	0.52	0.71	0.80	0.83	0.81
cos.	0.63	0.76	0.86	0.80	0.83
dens.	0.55	0.71	0.81	0.78	0.79
hmm	0.55	0.71	0.79	0.78	0.79
poiss.	0.59	0.74	0.83	0.79	0.81
comb.	0.57	0.72	0.81	0.78	0.80

Table 6. Table 6: Results of the simple consensus strategy on the automatic transcriptions of the dev set (confidence interval = 0.03)

dev	asr
	cov.	$a c c_{1}$	$a c c_{2}$	prec.	rec.	F-sc.
$set ({𝐌𝐀𝐉}_{𝟒})$	0.64	0.92	0.96	0.99	0.62	0.76
$set ({𝐌𝐀𝐉}_{𝟑})$	0.86	0.87	0.93	0.98	0.80	0.88
$set ({𝐌𝐀𝐉}_{𝟐})$	1	0.81	0.88	0.94	0.89	0.91

Table 7. Table 7: Theme related concepts expressed with selected words, bigrams and coexistence of them in a dialogue turn.

Traffic state (ts)	Time schedule	Itinerary
- ts inquiry/response	- schedule mention	- itinerary reference
- ts causal facts	- start hours minutes	- itinerary start
- tf situation	- period reference	- itinerary end
- tf state timing		- itinerary connections
Cards	Fares	Lost and found (lf)
- specific card mention	- products	- lf inquiry/response
nvgo and vgc	- suburban extension	- lf objects
- card status	- price	- lf actions
- card related actions	- payment modality	- lf addresses
- required documents	- restrictions
- accounting
Contravention	Special offers (so)	Complementary
		information
- contravention mention	- special offer mention	- transportation mean
- status	- so product	- generic problem
- payment	- so price	mention
- procedure	- conditions	- call transfer to
- complain		specific services
- declare usurpation		- specific addresses and
- request indulgence		telephone numbers
- motivation for request		- qualitative reference
- legal aspects		to time
		- relation with other
		city entities

Table 8. Table 8: Results of the simple consensus strategy on the automatic transcriptions of the test set (confidence interval = 0.03)

test	asr
	cov.	$a c c_{1}$	$a c c_{2}$	prec.	rec.	F-sc.
$set ({𝐌𝐀𝐉}_{𝟒})$	0.54	0.88	0.92	0.96	0.50	0.66
$set ({𝐌𝐀𝐉}_{𝟑})$	0.80	0.81	0.87	0.92	0.70	0.80
$set ({𝐌𝐀𝐉}_{𝟐})$	0.98	0.74	0.82	0.87	0.82	0.84

Table 9. Table 9: Results of the recovery strategy on the automatic transcriptions of the test set (confidence interval = 0.03)

test	asr
	cov.	$a c c_{1}$
set( ${𝐌𝐀𝐉}_{𝟒}$ )	0.54	0.89
set( ${𝐌𝐀𝐉}_{𝟑}$ )	0.80	0.86
set( ${𝐌𝐀𝐉}_{𝟐}$ )	0.98	0.80
All test	1	0.80

Table 10. Table 10: Number of concept instances for the dev and test sets

	man	asr one-best	asr lattice
dev	26,256	18,906	21,536
test	80,764	47,725	59,166

Equations99

G (f) = t \in T \sum P^{2} (t ∣ f) = t \in T \sum (\frac{d f _{t} ( f )}{d f _{C} ( f )})^{2}

G (f) = t \in T \sum P^{2} (t ∣ f) = t \in T \sum (\frac{d f _{t} ( f )}{d f _{C} ( f )})^{2}

w_{t} (f) = d f_{t} (f) . i d f^{2} (f) . G^{2} (f)

w_{t} (f) = d f_{t} (f) . i d f^{2} (f) . G^{2} (f)

sc (y, t) = cos (v_{y}, v_{t}) = \frac{\sum _{f \in y \cap t} w _{y} ( f ) . w _{t} ( f )}{\sum _{f} w _{y} ( f ) ^{2} . \sum _{f} w _{t} ( f ) ^{2}}

sc (y, t) = cos (v_{y}, v_{t}) = \frac{\sum _{f \in y \cap t} w _{y} ( f ) . w _{t} ( f )}{\sum _{f} w _{y} ( f ) ^{2} . \sum _{f} w _{t} ( f ) ^{2}}

t \in γ_{1} (y) ⟹ sc (y, t) ⩾ ρ . sc (y, \hat{t})

t \in γ_{1} (y) ⟹ sc (y, t) ⩾ ρ . sc (y, \hat{t})

t \in γ_{1} (y) ⟹ sc (y, t) ⩾ v t^{'} \in T \sum sc (y, t^{'})

t \in γ_{1} (y) ⟹ sc (y, t) ⩾ v t^{'} \in T \sum sc (y, t^{'})

w_{t} (p_{i}) = \frac{1}{∥ v _{t} ∥} f \in τ_{p_{i}} \sum w_{t} (f) (i = 1, ..., n)

w_{t} (p_{i}) = \frac{1}{∥ v _{t} ∥} f \in τ_{p_{i}} \sum w_{t} (f) (i = 1, ..., n)

d_{t} (p_{i}) = \frac{\sum _{j = 1}^{n} \frac{w _{t} ( p _{j} )}{λ ^{d_{j}}}}{\sum _{j = 1}^{n} \frac{1}{λ ^{d_{j}}}} (i = 1, ..., n)

d_{t} (p_{i}) = \frac{\sum _{j = 1}^{n} \frac{w _{t} ( p _{j} )}{λ ^{d_{j}}}}{\sum _{j = 1}^{n} \frac{1}{λ ^{d_{j}}}} (i = 1, ..., n)

t \in γ_{2} (y) ⟹ \exists i \in [[1; n]] : \forall t^{'} \in T, d_{t} (p_{i}) ⩾ d_{t^{'}} (p_{i})

t \in γ_{2} (y) ⟹ \exists i \in [[1; n]] : \forall t^{'} \in T, d_{t} (p_{i}) ⩾ d_{t^{'}} (p_{i})

t \in γ_{2} (y) ⟹ i \in I \sum d_{t} (p_{i}) ⩾ v j = 1 \sum n d_{t_{j}} (p_{j})

t \in γ_{2} (y) ⟹ i \in I \sum d_{t} (p_{i}) ⩾ v j = 1 \sum n d_{t_{j}} (p_{j})

\hat{t} = t arg max (P (t ∣ y)) = t arg max (lo g (P (y ∣ t)) + lo g (P (t)))

\hat{t} = t arg max (P (t ∣ y)) = t arg max (lo g (P (y ∣ t)) + lo g (P (t)))

P (F_{f} = r ∣ t) = \frac{e ^{- μ_{f, t}} μ _{f, t}^{r}}{r !}

P (F_{f} = r ∣ t) = \frac{e ^{- μ_{f, t}} μ _{f, t}^{r}}{r !}

lo g (P (y ∣ t)) ≃ f \sum (F_{f} lo g (μ_{f, t}) - μ_{f, t} - lo g (F_{f}!))

lo g (P (y ∣ t)) ≃ f \sum (F_{f} lo g (μ_{f, t}) - μ_{f, t} - lo g (F_{f}!))

\hat{t} = t arg max (lo g (P (t)) + f \sum (F_{f} lo g (μ_{f, t}) - μ_{f, t}))

\hat{t} = t arg max (lo g (P (t)) + f \sum (F_{f} lo g (μ_{f, t}) - μ_{f, t}))

R (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ M ( y ) ∣}

R (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ M ( y ) ∣}

P (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ γ ( y ) ∣}

P (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ γ ( y ) ∣}

F (γ, X) = \frac{2 P ( γ , X ) R ( γ , X )}{P ( γ , X ) + R ( γ , X )}

F (γ, X) = \frac{2 P ( γ , X ) R ( γ , X )}{P ( γ , X ) + R ( γ , X )}

a c c_{1} (γ, X) = \frac{1}{∣ X ∣} y \in X \sum g (γ (y), M (y))

a c c_{1} (γ, X) = \frac{1}{∣ X ∣} y \in X \sum g (γ (y), M (y))

g (γ (y), M (y)) = {10 if γ (y) = M (y) otherwise

g (γ (y), M (y)) = {10 if γ (y) = M (y) otherwise

a c c_{2} (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ γ ( y ) \cup M ( y ) ∣}

a c c_{2} (γ, X) = \frac{1}{∣ X ∣} y \in X \sum \frac{∣ γ ( y ) \cap M ( y ) ∣}{∣ γ ( y ) \cup M ( y ) ∣}

Y R Q_{1} := {y \in r s ∣ MA J_{4} (y, r s) = true}

Y R Q_{1} := {y \in r s ∣ MA J_{4} (y, r s) = true}

N R Q_{1} := {y \in r s ∣ MA J_{4} (y, r s) = false}

N R Q_{1} := {y \in r s ∣ MA J_{4} (y, r s) = false}

co v (Y R Q_{1}) = \frac{∣ Y R Q _{1} ∣}{∣ r s ∣}

co v (Y R Q_{1}) = \frac{∣ Y R Q _{1} ∣}{∣ r s ∣}

cor r (Y R Q_{1}) = y \in Y R Q_{1} \sum g (γ (y), M (y))

cor r (Y R Q_{1}) = y \in Y R Q_{1} \sum g (γ (y), M (y))

P 3 (L, Q) : (3 H \to L) \land (1 H \to L Q)

P 3 (L, Q) : (3 H \to L) \land (1 H \to L Q)

L = \textsc o bj t and Q = \textsc o bj t_\textsc n v g o

L = \textsc o bj t and Q = \textsc o bj t_\textsc n v g o

P C_{j (n)} ⟹ A C_{j (n)}

P C_{j (n)} ⟹ A C_{j (n)}

A C_{j (n)} : assert (H j (n)) \land evaluate (H j (n))

A C_{j (n)} : assert (H j (n)) \land evaluate (H j (n))

(λ (x y z) (theme (x) \land facts (x, y) \land compl_in f_{1} (x, y, z)))

(λ (x y z) (theme (x) \land facts (x, y) \land compl_in f_{1} (x, y, z)))

I (t_{k}, A_{i}) = H (t_{k}) - H (t_{k} ∣ A_{i})

I (t_{k}, A_{i}) = H (t_{k}) - H (t_{k} ∣ A_{i})

T = j arg max H (t_{j} ∣ A_{i})

T = j arg max H (t_{j} ∣ A_{i})

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Full text

Multiple topic identification in

human/human conversations

X. Bost

G. Senay

M. El-Bèze

R. De Mori

LIA, University of Avignon, 339 chemin des Meinajariès, 84000 Avignon, France

McGill University, School of Computer Science, Montréal, Québec, Canada

Abstract

**Cite as:

X. Bost, G. Senay, M. El-Bèze, R. De Mori.

Multiple topic identification in human/human conversations.

Computer Speech & Language, 2015, 34 (1), pp.18-42.

doi: 10.1016/j.csl.2015.03.006**

The paper deals with the automatic analysis of real-life telephone conversations between an agent and a customer of a customer care service (ccs). The application domain is the public transportation system in Paris and the purpose is to collect statistics about customer problems in order to monitor the service and decide priorities on the intervention for improving user satisfaction.

Of primary importance for the analysis is the detection of themes that are the object of customer problems. Themes are defined in the application requirements and are part of the application ontology that is implicit in the ccs documentation.

Due to variety of customer population, the structure of conversations with an agent is unpredictable. A conversation may be about one or more themes. Theme mentions can be interleaved with mentions of facts that are irrelevant for the application purpose. Furthermore, in certain conversations theme mentions are localized in specific conversation segments while in other conversations mentions cannot be localized. As a consequence, approaches to feature extraction with and without mention localization are considered.

Application domain relevant themes identified by an automatic procedure are expressed by specific sentences whose words are hypothesized by an automatic speech recognition (asr) system. The asr system is error prone. The word error rates can be very high for many reasons. Among them it is worth mentioning unpredictable background noise, speaker accent, and various types of speech disfluencies.

As the application task requires the composition of proportions of theme mentions, a sequential decision strategy is introduced in this paper for performing a survey of the large amount of conversations made available in a given time period. The strategy has to sample the conversations to form a survey containing enough data analyzed with high accuracy so that proportions can be estimated with sufficient accuracy.

Due to the unpredictable type of theme mentions, it is appropriate to consider methods for theme hypothesization based on global as well as local feature extraction. Two systems based on each type of feature extraction will be considered by the strategy. One of the four methods is novel. It is based on a new definition of density of theme mentions and on the localization of high density zones whose boundaries do not need to be precisely detected.

The sequential decision strategy starts by grouping theme hypotheses into sets of different expected accuracy and coverage levels. For those sets for which accuracy can be improved with a consequent increase of coverage a new system with new features is introduced. Its execution is triggered only when specific preconditions are met on the hypotheses generated by the basic four systems.

Experimental results are provided on a corpus collected in the call center of the Paris transportation system known as ratp. The results show that surveys with high accuracy and coverage can be composed with the proposed strategy and systems. This makes it possible to apply a previously published proportion estimation approach that takes into account hypothesization errors.

keywords:

Human/human conversation analysis; multi-topic identification; spoken language understanding; interpretation strategies

††journal: Computer Speech and Language

1 Introduction

In recent years, there has been an increasing research interest in the analysis of human/human spoken conversations. Advances in this research area are described in (Tur and De Mori, 2011) and (Tur and Hakkani-Tür, 2011). A scientifically interesting and practically important component of this research is topic identification for which an ample review of the state of the art can be found in (Hazen, 2011b). In spite of the relevant progress achieved so far, it is difficult to reliably identify multiple topics in real-life telephone conversations between casual speakers in unpredictable acoustic environments. Of particular interest are call-center conversations in which customers discuss problems in specific domains with an advisor. This is the case of the application considered in this paper. The purpose of the application is to monitor the effectiveness of a customer care service (ccs) of the ratp Paris transportation system by analyzing real-world human/human telephone conversations in which an agent solicits a customer to formulate a problem and attempts to solve it. The application task is to perform surveys of customer problems in different time periods. Proportions of problem themes computed with the survey data are used for monitoring user satisfaction and establishing priorities of problem solving interventions. Of primary importance for this task is the ability to automatically select, in a given period of time, a sufficiently large sample of conversations automatically analyzed with high accuracy. Application relevant information for evaluating proportions of problem items and solutions is described in the application requirements making evident, not formally defined, but useful, speech analytics. Themes mentioned in the application documentation appear to be the most important and general classes representing concerns that the survey has to address. Themes appear to be an adequate semantic representation of the types of problems in the application ontology that is inferred from the documentation.

Agents follow a pre-defined protocol to propose solutions to user problems about the transportation system and its services. In order to evaluate proportions of problem types, agents compile conversation reports following a domain ontology. Due to time constraints, reports compiled by the agents cover a small proportion of conversations. Furthermore, compiled reports are often incomplete and error-prone. Thus, an automatic classification of conversation themes would be useful for producing a quantity of accurate reports sufficient for performing a reliable survey.

The achievement of acceptable automatic solutions has to overcome problems and limits of known spoken language processing and interpretation systems. A fully automatic system must include an automatic speech recognition (asr) module for obtaining automatic transcriptions of the conversations. The acoustic environment on these conversations is unpredictable with a large variety of noise types and intensity. Customers may not be native French speakers and conversations may exhibit frequent disfluencies. The agent may call another service for gathering information. This may cause the introduction of different types of non-speech sounds that have to be identified and discarded. For all these reasons, the word error rate (wer) of the asr system is highly variable and can be very high. Conversations to be analyzed may be about one or more domain themes. Mentions of relevant themes may be interleaved with mentions of irrelevant comments. Mentions of multiple themes may appear in well separable conversation segments or they may be diffused in zones of a conversation that are difficult to localize. Moreover, mentions can be incomplete or highly imprecise involving repetitions, ambiguities, linguistic and pronunciation errors. Some mentions of an application relevant theme may become irrelevant in certain contexts. For example, a customer may inquiry about an object lost on a transportation mean that was late. In such a case, the loss should be considered as a more relevant theme than the traffic state delay.

In order to properly approach the problems in the above mentioned variety of scenarios and difficulties, methods relying on evaluating the evidence of each feature in the whole conversation should be combined with methods that evaluate feature evidence on specific, automatically detected, conversation segments. In fact, depending on the customer speaking style, mentions of the same pair of themes can be diffused in large portions of a conversation with not clearly defined location for each theme, or they may appear in other conversations in well-localized segments. Thus, it appears useful to consider algorithmic approaches for diffuse theme mentions and other approaches for mentions in specific, localized segments. Furthermore, in order to reduce the effect of asr and classification errors, it is proposed in this paper to investigate the possibility of using at least two different approaches for each of the two possible types of mentions. One of these methods is novel and is introduced, together with a review of the others, in section 4, while the corpus used for the experiments is described in section 3.

In order to obtain reliable surveys with error-prone approaches, conditions have to be automatically identified for selecting reliably annotated samples and for possibly triggering suitable refinements. An effective sequential theme hypothesization strategy will be introduced in section 6. A component of the strategy performs refinements consisting in recovering deletion or insertion errors in multiple theme hypotheses having a correct theme already hypothesized. The strategy also includes a new theme evaluation process to compensate some specific classification errors that may be caused by asr errors. The execution of this process is restricted to specific situations. The process action performs semantically coherent, compositions or decompositions of already generated theme hypotheses. New features are used in this process. They are designed to characterize diffuse theme mentions and are extracted from word lattices as described in (Wintrode and Kulp, 2009), (Hazen, 2011a). A moderate amount of human compiled knowledge is conceived to establish preconditions inspired by the application ontology for the application of the refinement process. The components of the theme hypothesization strategy are conceived to have low linear time complexity making their execution faster that the asr process. The experimental results reported in sections 5 and 6 show that topic identification errors increase with the disagreement among the four systems. Nevertheless, reliable surveys can be obtained with high annotation accuracy in a high proportion of the available conversations with a sequential decision strategy that progressively applies specific decision criteria and features from asr generated lattices of word hypotheses.

2 Related work

As mentioned in the topic identification review presented in (Hazen, 2011b) topic identification is performed with supervised and unsupervised classification methods. It is worth mentioning among them, especially for text documents, decision trees (Cohen and Singer, 1999), naive Bayes classifiers (Li and Jain, 1998), other probabilistic classifiers (Lewis and Gale, 1994), support vector machines (Joachims, 1998), k-nearest neighbors and example-based classifiers (Yang and Pedersen, 1997).

Popular features used by the classifiers are terms, for each of which measures such as term frequency (tf), inverse document frequency (idf), and tf-idf combinations are computed. A review on the unsupervised selection of terms such as keywords and word chunks can be found in (Chen et al., 2010). These features can be enriched with variations of left and right context and with parts of speech (pos) for each selected word. Other feature selection methods are discussed in (Yang and Pedersen, 1997).

In order to reduce the dimensionality of the feature space, latent semantic analysis (lsa) (Bellegarda, 2000), its probabilistic version (plsa) and latent Dirichlet allocation (lda) have been used for topic identification. In (Chien and Wu, 2008) a naive Bayes classifier based on adaptive lsa is proposed for topic identification in spoken documents. With lda (Blei et al., 2003), (Wintrode, 2011), the features representing a document are probabilities of the presence of hidden topics in a document. Hidden topics are determined by a specific learning process and differ from the topics to be identified. Approaches based on the assumption that a document is represented by a feature vector that is a combination of a small number of topic vectors are reviewed and compared in (Min and Ma, 2013).

Recently (Morchid et al., 2014a), it has been observed that theme identification accuracy may have large variations for different choices and dimensions of hidden topic spaces. Stable and superior performance has been achieved by introducing a new method called c-vector proposed for integrating features obtained in a large variety of these spaces.

Topic segmentation in spoken conversations is reviewed in (Purver, 2011). An evaluation of coarse-grain discourse segmentation can be found in (Niekrasz and Moore, 2010). Interesting solutions have been proposed for linear models of nonhierarchical segmentation. Some approaches propose inference methods for selecting segmentation points at the local maxima of cohesion functions. Some functions use features extracted in each conversation sentence or in a window including a few sentences. Some search methods detect cohesion in a conversation using language models and some others consider hidden topics shared across documents. These aspects are discussed for text processing in (Eisenstein and Barzilay, 2008) together with a critical review on lexical cohesion. In (Balchandran et al., 2010) techniques for topic detection have been proposed for obtaining topic specific language models (lms) used for re-scoring asr results.

Multi-label text classification is discussed in (de Carvalho and Freitas, 2009) and (Tsoumakas and Katakis, 2007) for large collections of text documents. In (de Carvalho and Freitas, 2009) a technique, called creation, is proposed. It consists in creating new composite labels for each association of multiple labels assigned to an instance. An important contribution with useful discussions and a thorough review on learning semantic structures from in-domain text document can be found in (Chen, 2010). A generative model of content structure is proposed based on an explicit representation of discourse constraints on regularities that are important for topic selection.

This paper adapts concepts found in the recent literature to the design of an automatic process for reliably annotating and selecting conversations for a ccs survey. A new sequential decision strategy is proposed for this purpose. It uses results of four different classification systems conceived for searching mentions of theme relevant features in an entire conversation or in specific segments of it. The motivation for considering these two possibilities is the unpredictable language style of casual real-world application users. One of the four systems uses a new method for extracting features from discourse segments. Rather than building segments by collapsing sentences or paragraphs or by identifying suitable segment bounds, the proposed approach looks for zones of a conversation dense of theme relevant features and uses these data for classification. This is motivated by the observation that often customers tend to introduce their problem with factual descriptions that are irrelevant for the task and agents tend to formulate a solution with a verbose introduction followed by a concise formal explanation dense of theme specific words and phrases.

In (Wintrode and Kulp, 2009), a method is introduced for performing a supervised classification of a spoken document into a single topic class. Features for the classification are posterior probabilities of word hypotheses computed over an entire conversation and obtained from word lattices generated by an asr system. These lattice features are also applied in (Hazen, 2011a) to supervised topic classification with minimum error classifiers. Hidden topic features obtained with lda have been combined with other features in support vector machine (svm) classifiers to assign spoken documents to relevance classes (Wintrode, 2011) using asr word lattice hypotheses.

The strategy proposed in this paper is based on an application independent criterion for composing conversations sets of increasing size with limited decrease in automatic annotation accuracy. A specific strategy process attempts to alleviate the effects of possible asr errors by using specific word and distance bigram features extracted from the lattice of word hypotheses generated by an asr system. These bigrams are selected to express distance relations between facts and actions that characterize each theme.

3 The application domain and the corpus used for the experiments

The application task considered in this paper is the automatic annotation of conversations between a customer and the agent of a ccs. Application requirements establish that the conversations have to be annotated in terms of application domain semantic contents. Essential contents are themes belonging to the following set (between parentheses the abbreviations used by the agents):

$\mathbb{T}:=\$ {itinerary (itnr), lost and found (objt), time schedules (horr), transportation card navigo (nvgo), traffic state (etfc), fares (tarf), contraventions (pv), transportation card vgc (vgc), out of domain (trsh)}

Identifying these themes is the purpose of the research described in this paper. Conversations may be about more than one theme and the co-presence of related themes appears to be essential to characterize the customer problem and to adequately monitor the time evolution of problem proportions in order to assess the increase or decrease of mentions of user concerns. Such an assessment can be made with surveys containing conversation samples. A method for obtaining acceptable estimations of proportion variability in time using surveys with samples that may contain annotation errors can be found in (Camelin et al., 2009). Starting with themes annotated with sufficient accuracy, more specific semantic information can be extracted and used to provide more details in conversation reports. These details can be used for other tasks such as the evaluation of the impact of strikes at specific dates. This aspect will be considered in future work.

Conversations are made available in successive time periods. Samples taken from different periods of a year have been selected to form a corpus of 1,654 telephone conversations collected at the call center of the public transportation service in Paris (ratp). The corpus has been manually transcribed after removing any relation to customer identity and other confidential information. In order to perform experiments with a fully automatic system, the corpus was split into three corpora, namely a train, a development and a test set containing respectively 880, 196 and 576 conversations. The sizes of the corpora in terms of conversations annotated with one or more theme labels are shown in Table 1. The table includes, for each set, the number of conversations annotated with a single theme label (#mono-label), the number of conversations annotated with multiple theme labels (#multi-label), the label cardinality (i.e. the average number of labels by conversation, denoted here label card.) and the percentage of conversations annotated with multiple labels.

Conversations have been independently annotated by three human annotators in terms of the main theme and possibly additional themes. When the annotators disagree, a consensus was found after discussion. The corpus with manual transcriptions and annotations will be made publicly available at the end of the decoda project pending the authorization of the data owners. In order to develop an effective strategy, four initial different systems have been considered and developed for automatic annotation. All the four systems use the same type of features extracted from the 1-best sequence of word hypotheses generated by an asr component described in (Linarès et al., 2007). The asr system is based on triphone acoustic hidden Markov models (hmm) with mixtures of 230,000 Gaussian distributions. Model parameters were estimated with maximum a posteriori probability (map) adaptation of 150 hours of speech in telephone bandwidth with the data of the train set. A 4-gram language model (lm) was obtained by adapting with the transcriptions of the train set a basic lm whose parameters have been estimated with other corpora. An initial set of experiments was performed with this system. The results show a wer on the test set of 57% (52% for agents and 62% for customers). These high error rates are mainly due to speech disfluencies and to adverse acoustic environments for some conversations when, for example, customers call from train stations or noisy streets with mobile phones. Furthermore, the signal of some sentences is saturated or of low intensity due to the distance between speakers and phones. Such difficult acoustic conditions have a significant impact on the density of the lattices with peaks of 2,600 links per second.

Two systems called density and hmm segment a conversation and attempt to hypothesize theme mentions in segments. The other two systems called cosine and poisson generate annotation hypotheses by cumulating scores for each feature in the entire conversation.

The asr component generates a lattice of word hypotheses using a vocabulary of a fixed number of words obtained from the content of the database of the application domain and words observed in the train corpus. The test data may contain out-of-vocabulary (oov) words not observed in the train set. It was observed that the train corpus vocabulary contains 7,920 words while the test corpus contains 3,806 words, only 70.8% of them occur in the train corpus and the model perplexity is 82 for the dev set and 86 for the test set. In spite of possibly frequent asr errors, it is worth investigating the possibility of performing reliable theme surveys considering that conversations contain mentions to rich sets of theme relevant facts, entities and actions and many small sets of them are sufficient conditions for theme identification.

Theme identification may use the most likely sequence (1-best sequence) of word hypotheses as features for theme identification. Using features from the lattice of word hypotheses may only reduce the effect of some errors in the 1-best sequence of words in the vocabulary, but may also introduce confusion in the classification decision performed by the theme identification strategy.

In addition to the oov problem, theme identification may also confront the code-switching problem occurring when speakers alternate between language varieties. An analysis of the train set shows that the code-switching problem is not frequent and appears only in some customer segments. This is explained by the fact that the agent has to follow a well-defined problem solving protocol and tends to drive the conversation according to it. Furthermore, most of the varieties and use of oov words relate to descriptions of facts that are irrelevant for the application purpose. In any case, in order to model possible code-switching effects, two systems, namely density and hmm, have been conceived to analyze conversation segments primarily for using localized features to model possible variations in theme mentions.

For the sake of comparison, the four systems perform classifications with different methods using the same features extracted from the 1-best sequence of word hypotheses.

As the four systems are based on different algorithms, it is expected that they make different types of classification errors and tend to produce different annotations mostly when the features are affected by drastic asr errors.

Parameter estimation for each system is based on the minimization of classification errors and comparisons of scores with thresholds for making decisions. For this reason, the consensus among the four systems is expected to be a good confidence index of the classification results. Consensus is thus used for an initial selection of the conversations to be inserted in the survey.

In order to find, in specific conditions of weak consensus, additional conversations to be placed in the survey, the strategy is refined by a conditional execution of a fifth new process based on a new method and new features. These features are specific for each theme and are extracted from the lattice of word hypotheses generated by the asr component. Details of these features will be provided later on with the description of the refinement strategy.

4 Multiple theme hypothesization architecture

An architecture for multiple theme hypothesization has been conceived for integrating the results of different systems implementing different global and segmental approaches for extracting mentions of application relevant themes.

The architecture has the scheme shown in Figure 1 and is based on three components, namely an asr system that generates word hypotheses, a theme hypothesization component that contains four systems whose results are further processed by a decision making strategy.

The four systems use the most likely (1-best) sequence of word hypotheses generated by the same asr component. Different features extracted from the lattice of word hypotheses are available. They are processed by a fifth system whose output can be used by the strategy for making decisions when there are doubts about the confidence of the outputs of the four other systems. The four basic systems use the same set of features made of words selected based on their purity, word classes and short distance word bigrams.

Theme hypotheses are generated by systems called cosine and poisson using theme mentions detected anywhere in a conversation, while hmm and density localize theme mentions in specific positions of a conversation and make decisions based on features extracted around these positions. The four systems are introduced below.

A short description and preliminary results obtained with cosine and density have been presented in (Bost et al., 2013). As density is a novel method, it will be described here in more details.

The features used by the four systems contain a selection of 7,217 words in the asr lexicon, a few word classes such as times and prices, and word bigrams. Distance bigrams made of pairs of words distant a maximum of two words are also used. A reduced feature set $V_{f}$ is obtained by selecting features based on their purity and coverage. Purity of a feature $f$ is defined with the Gini criterion as follows:

[TABLE]

where $df_{\mathbb{C}}(f)$ is the number of conversations of the train set $\mathbb{C}$ containing feature $f$ and $df_{t}(f)$ is the number of conversations of the train set containing feature $f$ and annotated with theme $t$ .

Considering a conversation as a document, a score $w_{t}(f)$ for feature $f$ is defined, using all the samples of $t$ in the train set, as follows:

[TABLE]

where $idf(f)$ is the inverse document frequency for feature $f$ .

4.1 Cosine

The cosine system generates a theme hypothesis with a classification process based on a global cosine measure of similarity $\text{sc}(y,t)$ between a conversation $y$ and a theme $t$ . Representing $y$ by a vector $\mathbf{v_{y}}$ and $t$ by a vector $\mathbf{v_{t}}$ , this measure of similarity is computed as follows:

[TABLE]

where $w_{y}(f)$ is a score for feature $f$ in conversation $y$ .

Let $\gamma_{1}(y)$ be the set of themes discussed in $y$ . The following decision rule for automatically annotating conversation $y$ with a theme class label $t$ is initially applied:

[TABLE]

where $\hat{t}:=\operatorname*{arg\,max}_{t^{\prime}\in\mathbb{T}}\text{sc}(y,t^{\prime})$ ; and $\rho\in[0;1]$ is an empirical parameter whose value is estimated by experiments on the development set. If the score of $\hat{t}$ is too low, then the application of the above rule is not reliable. To overcome this problem, the following additional rule is introduced:

[TABLE]

where $v\in[0;1]$ is another parameter whose value is estimated by experiments on the development set.

The optimal value for $\rho$ , the proportion of the highest score required for assigning themes to a conversation, has been evaluated to $\hat{\rho}=0.69$ , and the optimal value threshold $v$ to $\hat{v}=0.16$ .

4.2 Density

Rather than representing a conversation with a bag of words whose order is irrelevant, the density system analyzes sequences of transcribed words for extracting features at specific positions in a sequence. A conversation containing $N$ words is seen, including spaces, as a finite sequence $(p_{1},...,p_{n}$ ) of $n$ positions (with $n=2N-1$ ). Features are described taking into account the position where they are extracted. The $k$ -th word unigram in the feature sequence representation is located at position $p_{2k-1}$ , the $k$ -th bigram is at position $p_{2k}$ , the $k$ -th distant bigram with one intermediate unigram is at position $p_{2k+1}$ .

The contribution to theme $t$ of the features at the $i$ -th position in a conversation is:

[TABLE]

where $\tau_{p_{i}}$ is the set made of the features (unigram and associated bigrams) located at position $p_{i}$ in a conversation.

A thematic density $d_{t}(p_{i})$ of theme $t$ is associated with position $p_{i}$ and is defined as follows:

[TABLE]

where $\lambda\geqslant 1$ is a parameter of sensitivity to proximity whose value is estimated by experiments on the development set and $d_{j}:=|i-j|$ is the distance in a conversation between positions $p_{i}$ and $p_{j}$ .

4.2.1 Theme density computation

According to equation (7), the density for theme $t$ at the $i$ -th position of a conversation is obtained by adding the contributions of the features located at that point and the contributions of each surrounding $(i\pm d)$ position weighted according to the distance $d$ . Such a computation requires two nested loops over the $n$ conversation positions, resulting in a quadratic time complexity.

In order to reduce computation costs, recurrence relations have been introduced in the expression of equation 7, resulting in a linear computation algorithm.

This algorithm, as well as the recurrence formulas it relies on, are fully specified in A.

4.2.2 Conversation skeleton

The thematic density at a specific conversation position is used to construct a thematic skeleton by plotting the density of theme hypotheses as function of a position measured in number of words (including spaces) preceding the position.

An example of thematic skeleton for a particular conversation is reported in B. A similar example can be found in (Bost et al., 2013).

4.2.3 Decision making

A theme $t$ is considered as discussed in a conversation $y$ if it has dominant density at a position of the conversation (rule (8)) and if the sum of its densities in positions where it is dominant exceeds an empirically determined threshold (rule (9)):

[TABLE]

where $\gamma_{2}(y)$ is the set of themes hypothesized for conversation $y$ ; $v\in[0,1]$ is a parameter whose value is estimated by experiments on the development set; $t_{j}$ is the theme of dominant density at the $j$ -th position in the conversation; and $I:=\{i\in\llbracket{1};{n}\rrbracket\ |\ \forall t^{\prime}\in\mathbb{T},\ d_{t}(p_{i})\geqslant d_{t^{\prime}}(p_{i})\}$ .

4.3 hmm

hmms have been widely used for topic identification (see, for example, (Mittendorf and Schäuble, 1994) and (Wallace et al., 2013) for a recent application).

For each theme $t_{k}$ , a language model $\textsc{lm}_{k}$ is obtained with the train set. The language model $\textsc{lm}_{k}$ is made of unigram probabilities and of probabilities $\mathbb{P}_{k}(wh)$ , where histories are obtained from chunks automatically selected with the same procedure used in (Maza et al., 2011). Conditional probabilities $\mathbb{P}(t_{i}|t_{j})$ are estimated from the conversations of the train set annotated with two themes. These probabilities are used together with probabilities $\mathbb{P}(t_{k})$ that there is a unique theme in a conversation.

A network is then constructed by connecting in parallel theme hmms modeling single theme conversations and sequences of these hmms for modeling multiple theme conversations. Probabilities of having a unique theme or pairs of themes are associated with each branch of the network. Word generation probabilities of these models are provided by the theme lms. The Viterbi algorithm is then applied using the network as model and the words of the 1-best sequence of conversation $y$ as observations for computing the highest probability among the following candidates $\mathbb{P}(y|t_{k})$ , $\mathbb{P}(y|t_{m}t_{j})$ , for every unique theme $(\forall k)$ and for every pair $(\forall(m,j))$ . When the hypothesis of multiple themes is predominant, segments expressing pairs of hypothesized themes are hypothesized by the search algorithm, together with a score for each segment. In this way, segments of theme mentions are hypothesized using models of short term sequential constraints.

4.4 poisson

The optimal hypothesis $\hat{t}$ consisting of a single theme or a sequence of themes is obtained with the following equation:

[TABLE]

Given the relatively small size of the available corpus, the Poisson law is well suited to take into account the sparse distribution of features $f$ in a theme $t$ . The approach followed for finding the results reported in this paper is inspired by (Bahl et al., 1988). Relying on the histogram estimate $F$ of feature occurrences, with mean $\mu_{f,t}$ , a topic model has been conceived based on the following probability:

[TABLE]

According to (Bahl et al., 1988) it is assumed that the feature frequencies have independent Poisson distribution and $\log(\mathbb{P}(y|t))$ can be estimated as follows:

[TABLE]

The last logarithm in the summation is independent of $t$ and can be ignored leading to:

[TABLE]

5 Individual evaluation of the four systems for theme hypothesization

5.1 Performance measures

The proposed approaches have been evaluated following procedures discussed in (Tsoumakas and Katakis, 2007) with measures used in Information Retrieval (ir) and accuracy as defined in the following for a corpus $\mathbb{X}$ and a decision strategy $\gamma$ .

Recall:

[TABLE]

where $M(y)$ indicates the set of themes manually annotated for conversation $y$ and $\gamma(y)$ the set of themes automatically hypothesized for the same conversation.

Precision:

[TABLE]

F-score:

[TABLE]

Accuracy:

The traditional way of computing the accuracy (denoted hereafter $acc_{1}$ ) is defined as:

[TABLE]

where:

[TABLE]

For multiple topic identification, it is also interesting to compute the accuracy in another way (denoted $acc_{2}$ ), as proposed by (Tsoumakas and Katakis, 2007):

[TABLE]

In the following if only accuracy is mentioned, it has to be assumed to refer to $acc_{1}$ .

5.2 Results

The four systems density, hmm, poisson, and cosine have been separately evaluated on the development and test sets. A linear interpolation of the results obtained with the four systems has also been evaluated and results are reported in the line labelled comb. As the dev set is too small for estimating the weights of the linear combination, equal weights have been used for the scores of the four systems.

As the parameters of the four systems are estimated for separately optimizing the accuracy of each system, a linear combination of decision scores of each system is not as efficient as consensus. The reason is that consensus is based only on each system decision that is reasonable to consider as the most reliable indicator of the evidence ascribed by each system to the theme hypotheses it generates after classifying with a specific approach an entire conversation.

The results are reported for the dev set in Tables 2 and 3, and for the test set in Tables 4 and 5. Label man refers to automatic theme annotation of manual transcriptions, while label asr refers to automatic theme annotation of asr system transcription.

For the sake of comparison, the results obtained with the proposed classification approaches have been compared with those obtained with a support vector machine (svm) using the same features (unigrams and bigrams with possible gap of one word) and a linear kernel. Denoting by $\mathbb{X}$ the set of conversations, a binary classifier $\gamma_{k}:\mathbb{X}\rightarrow\{t_{k},\overline{t_{k}}\}$ is defined for every theme $t_{k}\in\mathbb{T}$ . For every pair $(y,t_{k})$ containing the conversation $y$ and the $k$ -th theme, a score is computed by using this $k$ -th classifier. The candidate theme hypotheses for a conversation are those whose score is in an interval corresponding to an empirically determined proportion of the highest one. In addition to that, the hypothesis with the highest score must be above a threshold empirically determined for this purpose.

The toolkit used to perform svm-based categorization, svmlight, is described in (Joachims, 2002a), (Joachims, 1999) and (Joachims, 2002b).

5.3 Analysis of the performance of each proposed method and the linear combination of their results

The results reported in Tables 2, 3, 4 and 5 show that the results obtained with the proposed classification methods are not inferior and in many cases are significantly better than those obtained with svm. Furthermore, they also show that just a linear combination of the classifier outputs does not provide significant improvements over the best results obtained with one of the four proposed classifiers.

Some discrepancies between the results obtained with the dev and the test sets can be explained by the fact that parts of the test data were collected during the summer while all the train and dev data were collected in other seasons. Such a consideration is important since during the summer the proportion of traveling visitors with specific problems, speaking styles and accents is significantly higher than in the other seasons. Based on these considerations, lm probabilities are expected to exhibit some variation in the summer period. In order to evaluate this expectation, samples of the same time period have been analyzed in a sequence corresponding to date and time of their collection. A set of adaptation data has been collected using words of the 1-best sequence of word hypotheses generated by the asr component. Using these data for lm adaptation resulted in minor reductions of the wer and minor theme classification improvements that are not worth reporting here, over representing contexts with statistics of words that coexist in a conversation.

The possibility of modeling sequences of theme mentions in a multi-theme conversation has been also considered and the manual transcriptions of the entire train set have been analyzed for this purpose. Unfortunately no reliable cues have been identified to model a theme switch by the customer. Nevertheless, such an analysis suggested that hypothesization of concepts about specific theme contents may benefit from the knowledge of some extent of the concept mention. Such an investigation is part of future work.

A careful analysis has also been made for errors due to discrepancies between a correct theme hypothesization from manual (man) and automatic (asr) annotations. It was observed that most of these errors concern the hypothesization of the trsh class that is not a theme but an indication that the conversation content does not concern the application domain. Errors may concern the hypothesization of trsh. This is the cause of 40% of the errors in dev and 36,59% of the errors in test. The main reason for this is the large variability of the trsh lexicon that makes it difficult to model and classify it as a theme. A model and a classifier that are suitable for theme classification may not work well for rejecting out of domain conversations and different approaches should probably be investigated to further improve theme classification even if the results reported in this paper are more than satisfactory for a complete real-life application.

6 Decision strategy

The design of the decision strategy is inspired by the application requirements. The purpose is to obtain, with little or no human effort, a survey with enough correctly annotated samples for estimating, with sufficient confidence, proportions of user problem themes in a given time period. For this purpose, a large enough and accurately annotated portion of a specific corpus (dev, test) has to be obtained from the entire corpus. Such a selection is composed with a sequential decision strategy conceived to progressively augment the survey with samples that cause an acceptable accuracy decrease. The strategy is represented by a tree. A predicate having a conversation $y$ and a set is associated with each node of the tree. Node predicates describe binary relations having $\mathbf{pred}(y,rs)$ where $\mathbf{pred}$ represents the relation type and $rs$ is a variable that takes values in sets of conversations. For the predicate associated with the root, values of $rs$ can be the dev set for validating the strategy or the test set for evaluating it. A predicate associated with a tree node returns $\mathbf{true}$ after evaluating a specific function on specific properties extracted from specific bindings of its arguments.

An initial version of the strategy is constructed as described in the following subsection. For this strategy, the truth of the node predicates is evaluated by asking a question formulated in terms of agreement among the classifications of the four different systems introduced in section 4. The motivation is that when the same result is obtained by the four systems using diffuse and localized features the consensus should correspond to a reliable decision. Furthermore, confidence of partial consensus based decisions is expected to decrease with the level of consensus.

6.1 Initial consensus strategy

Three predicates about full and partial consensus among the four systems are associated to consensus sets obtained with the initial version of the strategy tree. Each predicate has the form $\mathbf{MAJ_{m}}(y,rs)\ (m=4,3,2)$ .

Predicate $\mathbf{MAJ_{4}}(y,rs)$ is associated with the root of the strategy tree and is evaluated to $\mathbf{true}$ when the four systems hypothesize the same themes for conversation $y$ in set $rs$ . The entire development set (dev) is initially used as value of $rs$ for a preliminary assessment of the conceived strategy. Let $\{\mathbf{true},\mathbf{false}\}$ be the possible values of $\mathbf{MAJ_{4}}(y,rs)$ . Based on these values, the following subsets of $rs$ are formed:

[TABLE]

Sets $YRQ_{1}$ and $NRQ_{1}$ are subsets of the value of variable $rs$ . This value indicates the set associated with the mother node of $YRQ_{1}$ and $NRQ_{1}$ in the decision tree. Associations of this type can be inferred from the decision tree structure and will not be mentioned in the following for the sake of simplicity.

A decision consisting in making a survey only with conversations in $YRQ_{1}$ is evaluated by two parameters, namely accuracy ( $acc_{1}$ ), as defined in equation 17, and coverage, computed as follows:

[TABLE]

Let $\text{set}(\mathbf{MAJ_{4}}):=YRQ_{1}$ , $\text{size}(\text{set}(\mathbf{MAJ_{4}})):=|YRQ_{1}|$ and let

[TABLE]

where $g(\gamma(y),M(y))$ has the same meaning as in equation 18.

For $rs:=\textsc{dev}$ , the following values have been observed:

$\text{size}(\text{set}(\mathbf{MAJ_{4}}))=126,\ corr(\text{set}(\mathbf{MAJ_{4}}))=116$

2.

$\text{size}(NRQ_{1})=70,\ \text{size}(\textsc{dev})=196$

from which the following evaluation of using $YRQ_{1}$ as survey is obtained:

$cov(\text{set}(\mathbf{MAJ_{4}}))=0.64$

2.

$acc_{1}(\text{set}(\mathbf{MAJ_{4}}))=0.92$

Based on the above results, motivations for starting the strategy with predicate $\mathbf{MAJ_{4}}(y,rs)$ are listed in the following:

Coverage and accuracy are pretty high and more suitable than what could be achieved by playing with thresholds on system combinations as combination weights could not be accurately estimated with a small development set. 2. 2.

All but one conversations in $NRQ_{1}$ contain only one correct theme hypothesis instead of two, missing a second semantically coherent theme hypothesis. For example, a conversation is automatically annotated with only itinerary while it was manually annotated with itinerary and time schedule.

Predicates $\mathbf{MAJ_{3}}$ and $\mathbf{MAJ_{2}}$ to represent consensus of respectively three or two systems are introduced and the corresponding sets are evaluated as described in C.

The results obtained with the initial strategy, considering four possible surveys made respectively with sets based on $\mathbf{MAJ_{4}},\ \mathbf{MAJ_{3}},\ \mathbf{MAJ_{2}}$ are reported in Table 6 for $rs:=\textsc{dev}$ with details of coverage, total accuracy ( $acc_{1}$ ), partial accuracy ( $acc_{2}$ ), precision, total recall, and F-measure.

An analysis of the errors in the dev set shows that many of them can be clustered and annotated for further processing. This possibility was investigated leading to an augmentation of the decision tree as described in the next subsection.

6.2 Strategy refinements

The errors observed with the application of the initial strategy are due to limits of the classifiers, the types of features used, out of vocabulary words (oov), the errors in the 1-best sequences of words hypotheses generated by the asr system and the simplicity of the strategy itself.

It appears reasonable and interesting to refine the strategy with the purpose of recovering some of the above mentioned errors. The recovery strategy adds nodes and branches to the initial decision strategy tree and executes additional decision processes specific for each of the consensus sets. Each process is conceived after a human analysis of error types in each of the above sets in the train and dev sets. For the sake of clarity, in the following a conversation will be indicated by $y_{i}$ and a theme will be indicated by adding an index to the variable $t$ .

The considered types of possible errors are listed below.

DT: the deletion of a theme $t_{b}$ in a multiple theme conversation manually annotated with $t_{a}t_{b}$ for which only $t_{a}$ has been automatically annotated. Notice that the order of theme hypotheses is not relevant for the application task even if it could be inferred using the results of density and hmm.

2.

ST: the substitution of a theme $t_{c}$ with a theme $t_{d}$ .

3.

IT: the insertion of a theme $t_{i}$ in a conversation manually annotated with $t_{a}$ and automatically annotated with $t_{a}t_{i}$ .

The observation of the errors in the train and dev sets shows that most of them are of the type DT, and the errors of the type ST and IT appear in a limited number of specific conditions, the most frequent of which concerns annotations with trash (trsh), a class that is difficult to characterize because of the variability of its semantic content.

The errors in the DT cases automatically annotated with trsh appear to be mostly due to asr errors or to words such as street names and lost objects whose mention is not present in the asr vocabulary. Other frequent errors are due to the detection of a fare without an indication to what item it applies to. This is explained by the failure to hypothesize a type of transportation card, whose mention is present in the lattice of word hypotheses but not in the 1-best sequence.

The error analysis of the train and dev sets reveals that the just mentioned types of errors appear for theme hypotheses specific to each consensus set. It is thus reasonable to perform recovery with a small set of precondition-action rules in which a precondition is a logical expression of hypotheses generated by the four systems and appearing in a specific consensus set. The consequent action consists in performing insertions and/or deletions of theme hypotheses if the modifications are supported by the evidence of new, theme specific features evaluated with the lattice of word hypotheses. Furthermore, the actions must be coherent with the application ontology and the precondition content.

The introduction of a specific set of features for each theme is an important novelty of the proposed approach. These thematic feature sets have a limited size and are obtained automatically from the lattices of all the train samples. Features for each theme are selected based on their purity in the train set. Features are words, word classes and bigrams with a variable distance between the mention of their constituents in a time window that may involve up to three conversation turns. These distance bigrams appear to have high purity expressing distance relations between facts and actions that characterize each theme.

In this way, it is expected that a theme hypothesis receives a high score when the features characterizing it have high evidence in contrast with features characterizing other themes. Features are scored with their contribution to reduction of equivocation (re) when they are used to hypothesize the theme they belong to.

Relevant facts and complementary information pertinent for each theme are concepts listed in Table 7. The expression of these concepts is assumed to be made of words and (distant) bigrams.

The expressions of each concept are listed in a record of features associated with the concept. Concepts such as time and price are decomposed into constituents. Each constituent is represented by a word or a bigram feature.

In some cases the detection of only part of the constituents of a concept may be useful for recovering a deletion. In other cases it may introduce a false alarm. For this reason, the use of lattice features has been constrained with specific preconditions expressing the detection of a significant but incomplete amount of theme features.

Other types of features could be considered but have not been used. For example, prosodic features may contribute to characterize complaints or requests of indulgence. The main reason is that it is difficult to extract reliable prosodic features in the customer turns because the acoustic environments may be affected by a large variety of noise types and intensity.

In order to introduce the form of preconditions let us define:

$H_{c}(i,cs)$ : the automatic annotation by system cosine of conversation $y_{i}$ in the consensus set $cs$ .

2.

$H_{d}(i,cs)$ : the automatic annotation by system density of conversation $y_{i}$ in the consensus set $cs$ .

3.

$H_{h}(i,cs)$ : the automatic annotation by system hmm of conversation $y_{i}$ in the consensus set $cs$ .

4.

$H_{P}(i,cs)$ : the automatic annotation by system poisson of conversation $y_{i}$ in the consensus set $cs$ .

Let $(mH\rightarrow A),\ (m=2,3,4)$ represent the fact that $m$ and only $m$ annotations of the four systems are equal to $A$ . For example, let assume that, for a conversation in the dev, it has been observed $H_{d}(i,cs)=H_{h}(i,cs)=H_{P}(i,cs)=\textsc{objt}$ (lost_and_found theme) and $H_{c}(i,cs)=\textsc{objt}\_\textsc{nvgo}$ (lost a nvgo transportation card).

This situation is generalized leading to the following precondition representation:

[TABLE]

Five general forms have been introduced, one for $YRQ_{1}$ (see Subsection 6.1), one for $YRQ_{2}$ (see C) and three for $NRQ_{2}$ (see C).

A general form can be specialized by binding the variables $L$ and $Q$ to specific values as in the previous example:

[TABLE]

Specialized preconditions have been manually derived from the semantic coherence relations introduced later on after the actions.

An action is introduced as consequence of each specialized precondition.

An action $AC_{j(n)}$ asserting an annotation modification $H_{i}{j(n)}$ of a theme annotation for conversation $y_{i}$ includes two processes, namely:

assert a semantically coherent modification $H_{i}{j(n)}$ ,

2.

evaluate the evidence of $H_{i}{j(n)}$ .

A precondition-action rule is written as follows:

[TABLE]

here $j(n)$ indicates the $j$ -th specification of the $n$ -th general precondition. Symbol $\Longrightarrow$ links a precondition to its corresponding action.

Action $AC_{j(n)}$ is formally represented as follows:

[TABLE]

The process $\text{assert}(H{j(n)})$ generates hypotheses coherent with the application ontology according to the content of a look-up table corresponding to precondition $j(n)$ . The application ontology has been inferred from the application documentation and the look-up table has been compiled based on the analysis of the errors of the train set. Both activities have been performed by a human expert. Common to many application domains is the use semantic knowledge containing structures representing a theme with its related facts and some complementary information. These structures are represented by lambda-expressions of this type:

[TABLE]

The semantic content of a conversation is obtained by binding the variables $x,y,z$ to specific values. Notice that a value can be another semantic structure with all variables bound to specific values. For example, if $x=\textit{traffic\_state}$ , then $y$ may be bound to the elements of the set $ts:=\{\text{strike},\text{delay},\text{accident},\text{anomaly}\}$ . Variable $z$ can be bound by other complementary information such as time and location. In many application domains, there are specific facts for each theme and specific relations between different themes.

Given the type of application in which conversations have to follow a defined protocol, it is reasonable to assume that co-existence of mentions of fact predicates and coherent argument values are sufficient conditions for expressing themes characterized by these predicates.

The conversation protocol implies that only themes with semantically coherent facts may coexist in a conversation. This may not be always the case in practice. Nevertheless, this type of semantic coherence can be used for imposing constraints for recovering hypothesized theme errors. For example, a mention of the theme objt may share the fact that the lost object is a type of nvgo card, thus the two themes may coexist. In order to further constrain the application of the recovery strategy, frequent errors observed in the train and dev sets have been clustered into types. The types of errors listed below have been used in the introduced coherent recovery actions:

fare in conjunction with itinerary and transportation cards.

2.

time in conjunction with itinerary and traffic_state.

3.

lost_found in conjunction with cards.

4.

itinerary in conjunction with traffic_state, cards, lost_found.

5.

fine in conjunction with cards and loss.

Refinement strategy actions are executed only if specific conditions and the asserted hypotheses ( $H{j(n)}$ ) are verified.

The procedure for hypothesis evaluation is now described.

A feature set $\Phi_{k}:=\{\varphi_{1}^{k},....,\varphi_{n}^{k},.....,\varphi^{k}_{N_{\Phi}}\},\ k\in\{1,...,K\}$ of size $N_{\Phi}$ is defined for each theme $t_{k}$ . Only conversations in the train set annotated with a single theme are considered and used for inferring features of $\Phi_{k}$ for each $k$ . The $i$ -th conversation of the entire corpus is described by the acoustic features $A_{i}$ . The mutual information between the annotated theme $t_{k}$ and $A_{i}$ is defined as:

[TABLE]

where $H(t_{k})$ is the entropy of theme $t_{k}$ and $H(t_{k}|A_{i})$ is the conditional entropy of theme $t_{k}$ computed with features $\Phi_{k}$ extracted from $A_{i}$ .

Details on the computation of $H(t_{k}|A_{i})$ are given in D.

The automatic feature selection process starts by considering a word vocabulary $V_{1}$ obtained from the application vocabulary $V$ (5,782 words) after removal of the words of a stop list. A list $LB$ of distant bigrams have been added to $V_{1}$ by considering the association of a word in $V_{1}$ with any word in V in a window spanning three turns. The elements of $V_{1}$ , $LB$ and a small set of abstract classes such as time and price are considered as possible features belonging to an initial set $\Phi_{0}$ . Set $\Phi^{k}$ is formed by computing, for every element $\varphi_{x}\in\Phi_{0}$ the purity measured by the probability $\mathbb{P}(t_{k}|\varphi_{x})$ , whose computation is described in D.

The size of the feature set has been obtained by automatically annotating the conversations in the train set manually annotated with only one theme. Automatic annotation was based on the following decision rule:

[TABLE]

After ordering by feature purity the feature set of each theme the same size of feature set candidates was considered for all themes. The size was progressively increased from 50 to 1,000 by steps of 5. For each step, the accuracy of theme hypothesization was computed in the train set. The process of growing the feature set size stopped when no tangible improvements in the classification accuracy was observed. A value $N_{\Phi}=420$ was found in this way.

Overall, 1,440 word features and 1,657 distant bigram features were obtained. The highest word purity for each theme varies between 0.6 and 0.91 depending on the theme, while the lowest word purity varies between 0.12 and 0.4.

Even with these features, some combinations of theme hypotheses were not well characterized. For example, it was not possible to have an exhaustive list of lost objects. For this purpose, it was considered useful to infer complete topic mentions. Complete topic mentions were manually compiled by selecting, abstracting and combining into patterns some automatically obtained features. For example, pattern structures were derived for abstractions of mentions of departure, arrival, connection to describe an itinerary. Patterns characterizing actions describing loss of an object without specifying the object type were also compiled into an abstract class. Some minor improvements were observed after the introduction of complete topic mentions features.

As an example of how recovery actions are applied, let us consider the case of recovering the deletion of a theme $AA$ when theme $B$ has been hypothesized. The following decision rule is applied:

[TABLE]

Scores are computed with the reduction of equivocation (re) process. These scores are obtained with features of a conversation whose mention is omitted in the above relation for the sake of simplicity. $\text{TH}(B)$ is a threshold evaluated from the dev set to optimize separation between positive and negative examples. With a larger dev set, $\text{TH}(B)$ could be estimated as the value that minimizes the relative entropy between manually and automatically annotated distributions of theme proportions. This threshold is applied to all consensus sets. Predicates $\mathbf{rank\_RE\_FIRST}(B)$ and $\mathbf{rank\_RE\_SECOND}(B)$ are evaluated to true when theme $B$ is respectively scored first or second by the re process. Decision of removing false insertion themes is applied in specific conditions when the theme position in the re ranking is not among the first three.

For specific conditions, these types of decisions were also considered:

composition of themes hypothesized by different systems,

2.

recovery of a theme from trsh.

Composition decisions were made based on positions and scores of the themes to be combined in the following situations:

highest re score for single insertion or trsh recovery,

2.

first two positions and co-presence in the hypotheses for the composition of two themes,

3.

composition of three themes in case of co-presence in different system hypotheses of only three semantically coherent themes and high re scores for them.

Based on the frequency of the observed errors in the dev and train, recovery for data in $\text{set}(\mathbf{MAJ_{4}})$ was considered only for the following semantic relations:

$B=\mathit{fare}$ in conjunction with $AA=\textsc{nvgo}$ (transportation card),

2.

$B=\mathit{time}\text{ or }B=\textsc{nvgo}$ in conjunction with $AA=\mathit{itinerary}$ ,

3.

recovery of $B=\mathit{fine}\text{ from }AA=\mathit{trash}$ .

The same cases as for $\text{set}(\mathbf{MAJ_{4}})$ were considered for $YRQ_{2}$ with the addition of

$B=\mathit{traffic\_state}$ in conjunction with $AA=\mathit{itinerary}$ ,

2.

$B=\mathit{lost}$ in conjunction with $AA=\textsc{nvgo}$ ,

3.

recovery of $B=\textsc{vgc}\text{ card}\text{ or }B=\mathit{traffic\_state}\text{ from }AA=\mathit{trash}$ .

Recovery in $cs_{3}$ (subset defined in C) is more difficult since possible major problems may be caused by errors in the 1-best sequence or by limits of some of the four systems. When the hypotheses generated by the four systems are all different it is likely that the 1-best sequence contains too many errors. In both cases re ranking and scores are used for recovering deletions and also for obtaining a new majority vote with the addition of the re ranking.

The results obtained with the initial strategy on the test set are reported in Table 8 and those obtained with the addition of the recovery strategy are reported in Table 9.

Results show a statistically significant contribution of the recovery strategy on $acc_{1}$ for high coverage with respect to the use of just the initial consensus strategy.

The numbers of concept instances annotated in the manual transcriptions (indicated in the following as man), and detected from the word hypotheses generated by the asr (most likely sequence and lattice) for the dev (8,646 turns – 196 dialogues) and the test (27,497 turns – 578 dialogues) sets are reported in Table 10.

The increase of relevant concepts detected in the lattice with respect to the one best sequence is significant. This is done at the expenses of 30% insertions that have little impact on theme recovery since most of them are filtered out by the selective recovery procedure.

By observing the concept mentions in the manually annotated concepts it appears that errors in automatic concept hypothesization are essentially due to asr errors.

Published experimental results on the decoda corpus refer to the identification of the dominant theme. Among them, the highest accuracy of 85% is reported in (Morchid et al., 2014a).

This value has been obtained with a subset of the corpus used in this paper annotated with a single theme label and a large number of lda feature sets extracted in hidden spaces of different size. This single theme is annotated with one of 8 theme labels. This label set is obtained from the one used in the experiments of this paper by removing trsh and vgc. Such a classification is useful in practice if out domain conversations are discarded by another method and proportions are evaluated for problems of transportation cards ignoring the card type.

Considering that the confidence interval of the results is 0.03, the accuracy $acc_{1}$ reported in this paper for multiple theme labelling is slightly inferior to the best accuracy reported for just one theme labelling with a smaller set of labels and a smaller corpus. It is worth mentioning that the results reported in this paper for multiple theme are superior or inside the confidence interval when compared with results reported in (Morchid et al., 2014b) for the identification of one of the above mentioned 8 theme labels obtained with Gaussian and svm classifiers using different types of features including lda hidden topic features computed in a single hidden space.

6.3 Error analysis

The errors observed in the test set of $YRQ_{1}$ using the initial strategy were 45 in 321 conversations. Among them, 6 deletions were corrected to $X\_Y$ when $X$ was hypothesized. Using only the complete mentions, 4 more errors of the same type were corrected while 1 false insertion was generated. Among the remaining errors, there were 15 deletion errors of $Y$ with $X$ correct and two insertion errors with $X$ correct. The other errors where 9 substitutions involving trash indicating difficulties in characterizing this type of rejection and the others were confusions between itinerary and traffic_state and between lost_and_found and cards. Listening to these conversations, it appeared that the two themes were mentioned, but the annotators considered one of them not sufficiently relevant.

The errors observed in the test set of $YRQ_{2}$ were 44 in 131 conversations. Among them, 10 deletions were corrected to $X\_Y$ when $X$ was hypothesized. Using only the complete mentions 5 more errors of the same type were corrected.

The errors observed in the test set of $YRQ_{3}$ were 71 in 117 conversations. Among them, no false recoveries were observed, 5 deletions were corrected to $X\_Y$ when $X$ was hypothesized, 6 compositions were correctly executed by merging hypotheses from different systems, 7 recoveries from trash were correctly performed. A minor number of corrections for $YRQ_{3}$ were made using the additional contribution of re ranking and complete mentions.

For $YRQ_{2}$ and $YRQ_{3}$ , corrected and remaining errors were of the same nature as those in $YRQ_{1}$ .

A large majority of the remaining errors are deletions of a theme in a conversation in which at least one of the annotated themes was correctly hypothesized. By listening to the conversations annotated with errors it appears that, in many of them, the annotators have made an intelligent choice of what was relevant, while the system ignored part of it or inserted themes that were mentioned, but were not relevant. Based on the analysis it is reasonable to conclude that, with the approach described in (Camelin et al., 2009), significant theme proportion variations can be detected in a time period only a little longer than if the manual annotations were used.

7 Conclusion and future work

Four systems for multiple theme hypothesization in human/human call center conversations have been considered. Two of them, cosine and poisson extract features for theme mentions in an entire conversation. Other two systems density and hmm attempt to localize theme mentions in specific zones of a conversation. A comparison of the results separately obtained with each system show little performance difference, while a study on the consensus among the results of the four systems show that there is a non-negligible proportion of conversations for which the four systems do not completely agree. Nevertheless, in such a case it is still possible to perform reliable decisions on specific conversation sets by considering majority votes. In making these decisions all four systems appear to be useful indicating that their combined use contribute to effectively process diffuse as well as localized theme mentions.

For a significant proportion of those cases for which reliable decisions cannot be made with majority vote strategy it is still possible to trigger a new process with new features for obtaining a high coverage survey with accurately annotated themes. The overall result is that the proposed approach makes it possible to produce practically useful theme proportions in spite of asr errors and the imprecision of classification methods.

The recovery strategy has some precondition-action rules that depend on the application ontology and other that are applicable to a large variety of application domains. This latter set contains rules for recovering false classification as trsh due to concept deletion errors and false insertions of concepts leading to the hypothesization of a domain theme rather than classifying the conversation as trsh. Another general case is the deletion of concepts whose mentions are short words that are likely to be deleted in the 1-best sequence of word hypotheses. These short words are often confused by inserting or deleting one short phoneme such as a plosive consonant. Specific recovery rules can then be conceived by observing a limited number of examples in the train set and using them for setting appropriate contexts for searching instances in the lattice of word hypotheses.

Future work should attempt to provide accurate estimations of the amount of time periods in which corpora have to be collected in order to extract practically useful surveys.

More detailed semantic information has also to be extracted under the control of the application domain ontology. New methods have to be introduced for estimating the amount and degree of user problem solution and for composing all the extracted semantic contents to produce conversation reports. These reports are not necessarily summaries. They should rather be semantic structures of strictly relevant information for the application task.

Acknowledgements

This work has been supported by the French National Research Agency (anr) with Project decoda, contract anr-09-coord-005, and the French cluster scs (Secured Communicating Solutions). The corpus used for the experiments has been provided by the ratp (Paris public transportation system).

Appendix A Theme density: computation details

A.1 Recurrence formulas

According to equation (7), the density for theme $t$ at the $i$ -th position of a conversation is evaluated as follows:

[TABLE]

where $w_{t}(p_{j})$ denotes the contribution to theme $t$ of the features located at the $j$ -th position; $\lambda\geqslant 1$ is a parameter of sensitivity to proximity and $d_{j}:=|i-j|$ is the distance between positions $p_{i}$ and $p_{j}$ .

The density $d_{t}(p_{i})$ for theme $t$ at the $i$ -th position can first be expressed as follows:

[TABLE]

With reference to the $i$ -th position, let $L(p_{i})=\sum_{j=1}^{i-1}\frac{w_{t}(p_{j})}{\lambda^{i-j}}$ be the weighted sum of the left-hand side thematic contributions and $R(p_{i})=\sum_{j=i+1}^{n}\frac{w_{t}(p_{j})}{\lambda^{j-i}}$ be the weighted sum of the right-hand side thematic contributions. Let $NL(p_{i})=\sum_{j=1}^{i-1}\frac{1}{\lambda^{i-j}}$ be the left-hand side summation of the normalization factors and let $NR(p_{i})=\sum_{j=i+1}^{n}\frac{1}{\lambda^{j-i}}$ be the corresponding right-hand side summation. Introducing these notations in equation (28), one gets for $i=1,...,n$ :

[TABLE]

The quantities $L(p_{i}),R(p_{i}),\text{ and }NL(p_{i})$ can be computed as follows by simple recurrence on the neighbor positions:

Initialization:

[TABLE]

Recurrence:

[TABLE]

In a similar way, one gets:

[TABLE]

A.2 Computation algorithm

The use of these recurrences for computing the thematic density is shown in the description of Algorithm 1.

Positions $p_{i}\ (i=1,...,n)$ are represented by their respective indices and the density $d_{t}(p_{i})$ of the theme $t$ at position $p_{i}$ is indicated as $dens[t,i]$ . Similarly, the contribution $w_{t}(p_{i})$ to the theme $t$ of the features located at position $p_{i}$ is indicated as $w[t,i]$ .

The number of the instructions of the algorithm 1 is given by:

[TABLE]

The time complexity is thus a linear function of the number of conversation positions.

Appendix B Thematic skeleton of a conversation

The thematic skeleton of a conversation for three themes is shown in Figure 2.

The skeletons for $\lambda=1.04$ (Fig. B.2-a) and $\lambda=3$ (Fig. B.2-b) were obtained using a manual word transcription of the conversation, which is about a fine (theme indicated as pv) caused by an identity theft. The customer has been fined because he used his mother’s transportation card (theme indicated as nvgo). Two functions are plotted for the annotated themes transportation card and fine and a third function is plotted for the theme fare (indicated as tarf) that is not annotated because not completely mentioned in the conversation and considered to be irrelevant for the task.

The figures have been obtained from a conversation whose an excerpt, as manually transcribed, is reported in the following. The conversation positions corresponding to each turn are in brackets.

–

Customer [22–116]: I am calling because I had a problem yesterday with my nvgo card (…) I used my mother’s card (…) The controller confiscated the card.

–

(…)

–

Agent [320–338]: We are going to receive your card at the collection center.

–

(…)

–

Customer [544–630]: How much the fine will be? (…) I have been told I have to pay forty euros (…)

–

Agent [642–656]: Right. It should be forty euros.

Distant contextual relations, corresponding to high values of $\lambda$ , such as $3$ in Fig. B.2-b, tend to be neglected. In these cases, incorrect decisions may be caused by isolated features that are not relevant for theme hypothesization. For example, the expressions how much (combien in French) in turn [544–630] and forty euros, repeated twice in turns [544–630] and [642–656], tend to show evidence for fare with a peak of density at position 630, whereas it is just, in this case, part of a request about fine.

Local context, more appropriate for theme mention detection, is appropriately represented by a value of $\lambda=1.04$ as in Fig. B.2-a. This has the effect of reducing the relevance of the fare hypothesis only supported by the expressions how much and forty euros.

In conclusion, values of $\lambda$ such as 1.04 correspond to thematic coherence leading to more accurate decisions.

Appendix C Consensus strategy with three or two systems

A predicate $\mathbf{MAJ_{3}}(y,NRQ_{1})$ takes value $\mathbf{true}$ if there is a consensus among only three systems. According to this value, two new subsets $YRQ_{2}$ and $NRQ_{2}$ are obtained.

Considering now a new survey set, larger than $YRQ_{1}$ and defined as $\text{set}(\mathbf{MAJ_{3}}):=(YRQ_{1}\cup YRQ_{2})$ , the following values are found:

$\text{size}(YRQ_{2})=43,\ corr(YRQ_{2})=31$

2.

$cov(\text{set}(\mathbf{MAJ_{3}}))=(126+43)/196=0.86$

3.

$acc_{1}(\text{set}(\mathbf{MAJ_{3}}))=(116+31)/(126+43)=0.87$

A high accuracy is still obtained with a coverage much greater than that of $YRQ_{1}$ .

With similar motivations, the predicate $\mathbf{MAJ_{2}}(y,NRQ_{2})$ , was considered and applied. It is worth considering three possible types of samples for which the predicate is asserted true. They are:

$XY_{3}$ : the automatic annotation of two systems is $X$ and the automatic annotation of the other two systems is $Y$ .

2.

$Za_{3}$ : the automatic annotation of two systems is $Z$ and the automatic annotations of the other two systems are respectively $Z_{1}$ and $Z_{2}$ , both different from $Z$ and such that $Z_{1}\neq Z_{2}$ .

3.

$N_{3}$ : the four systems provide four different automatic annotations.

Let $cs_{3}\subseteq NRQ_{2}$ be a subset of $NRQ_{2}$ defined as follows:

[TABLE]

where $LC_{3}$ is the set $XY_{3}$ automatically annotated with the results of the linear combination of the four systems. Let: $\text{set}(\mathbf{MAJ_{2}}):=(YRQ_{1}\cup YRQ_{2}\cup cs_{3})$ .

In such a case:

$\text{size}(cs_{3})=26,\ corr(cs_{3})=11$

2.

$cov(\text{set}(\mathbf{MAJ_{2}}))=1,\ acc_{1}(\text{set}(\mathbf{MAJ_{2}}))=0.81$

Appendix D Recovery strategy: computation details

Conditional entropy $H(t_{k}|A_{i})$ is computed as follows:

[TABLE]

$\mathbb{P}(t_{k}|\varphi_{n}^{k},A_{i})$ is approximated with $\mathbb{P}(t_{k}|\varphi_{n}^{k})$ .

This conditional entropy can be seen as a reduction of equivocation (re) in the estimation of $t_{k}$ knowing $A_{i}$ . The same number $N_{\Phi}$ of features is used for all themes.

Let $\mathbb{P}(t_{k}|\varphi_{n}^{k})$ be represented as $\mathbb{P}(t_{k}|\varphi_{x})$ for the sake of simplicity. Probability $\mathbb{P}(t_{k}|\varphi_{x})$ is computed as follows:

[TABLE]

where $c(\varphi_{x},t_{k})$ is the count of the times $\varphi_{x}$ appears in all the conversations in the train set manually annotated with $t_{j}$ .

Features are ranked based on their purity in each theme $t_{k}$ and an ordered list $L_{k}$ is compiled for each theme.

The presence of a candidate feature $\varphi_{x}^{k}\in L_{k}$ in a conversation is measured by the probability $\mathbb{P}(\varphi_{x}^{k}|A_{i})$ . If the feature is a distant bigram involving a word $w_{m}$ and its context $c_{m,m-\tau}$ the feature posterior probability is computed by the following product:

[TABLE]

Features are considered as terms in a term frequency-inverse document frequency score defined according to (Hazen, 2011a) as:

[TABLE]

where $Z=\overset{K}{\underset{j=1}{\sum}}\underset{g\in G_{u,j}}{\sum}\lambda_{g}(\varphi_{j}^{k})$ is the sum of the scores of all features competing with $\varphi_{x}^{k}$ . $G_{u,j}$ is the set of arcs in the graph labelled with the hypothesis.

References

Bahl et al. (1988)

Bahl, R., Bakis, R., De Souza, P., Mercer, R., 1988.

Obtaining candidate words by polling in a large vocabulary speech recognition system, in: Int. Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 489–492.

Balchandran et al. (2010)

Balchandran, R., Rachevsky, L., Ramabhadran, B., Novak, M., 2010.

Techniques for topic detection based processing in spoken dialog systems., in: INTERSPEECH, pp. 82–85.

Bellegarda (2000)

Bellegarda, J.R., 2000.

Exploiting latent semantic information in statistical language modeling.

Proceedings of the IEEE 88, 1279–1296.

Blei et al. (2003)

Blei, D.M., Ng, A.Y., Jordan, M.I., 2003.

Latent dirichlet allocation.

the J. of machine Learning res. 3, 993–1022.

Bost et al. (2013)

Bost, X., El-Bèze, M., De Mori, R., 2013.

Multiple topic identification in telephone conversations, in: INTERSPEECH, pp. 3756–3760.

Camelin et al. (2009)

Camelin, N., De Mori, R., Béchet, F., Damnati, G., 2009.

Error correction of proportions in spoken opinion surveys., in: INTERSPEECH, pp. 2715–2718.

de Carvalho and Freitas (2009)

de Carvalho, A.C., Freitas, A.A., 2009.

A tutorial on multi-label classification techniques, in: Found. of Computational Intelligence Volume 5. Springer, pp. 177–195.

Chen (2010)

Chen, H., 2010.

Learning semantic structures from in-domain documents.

Ph.D. thesis. Massachusetts Institute of Technology.

Chen et al. (2010)

Chen, Y.N., Huang, Y., Kong, S.Y., Lee, L.S., 2010.

Automatic key term extraction from spoken course lectures using branching entropy and prosodic/semantic features, in: Spok. Lang. Technology Workshop (SLT), IEEE. pp. 265–270.

Chien and Wu (2008)

Chien, J.T., Wu, M.S., 2008.

Adaptive bayesian latent semantic analysis.

Transactions on Audio, Speech, and Lang. Processing 16, 198–207.

Cohen and Singer (1999)

Cohen, W.W., Singer, Y., 1999.

Context-sensitive learning methods for text categorization.

ACM Transactions on Inf. Syst. (TOIS) 17, 141–173.

Eisenstein and Barzilay (2008)

Eisenstein, J., Barzilay, R., 2008.

Bayesian unsupervised topic segmentation, in: Proceedings of the 2008 Conference on Emp. Methods in Nat. Lang. Processing, Association for Computational Linguistics. pp. 334–343.

Hazen (2011a)

Hazen, T., 2011a.

Mce training techniques for topic identification of spoken audio documents.

Transactions on Audio, Speech, and Lang. Processing 19, 2451–2460.

Hazen (2011b)

Hazen, T.J., 2011b.

Topic identification, in: Spok. Lang. Underst.. John Wiley & Sons, Ltd, pp. 319–356.

Joachims (1998)

Joachims, T., 1998.

Text categorization with support vector machines: Learning with many relevant features.

Springer.

Joachims (1999)

Joachims, T., 1999.

Making large scale SVM learning practical.

Universität Dortmund.

Joachims (2002a)

Joachims, T., 2002a.

Learning to classify text using support vector machines: Methods, theory and algorithms.

Kluwer Academic Publishers.

Joachims (2002b)

Joachims, T., 2002b.

Optimizing search engines using clickthrough data, in: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, Association for Computational Linguistics. pp. 133–142.

Lewis and Gale (1994)

Lewis, D.D., Gale, W.A., 1994.

A sequential algorithm for training text classifiers, in: Proceedings of the 17th annual Int. ACM SIGIR conference on Res. and development in inf. retr., Springer-Verlag New York, Inc.. pp. 3–12.

Li and Jain (1998)

Li, Y.H., Jain, A.K., 1998.

Classification of text documents.

The Computer J. 41, 537–546.

Linarès et al. (2007)

Linarès, G., Nocéra, P., Massonie, D., Matrouf, D., 2007.

The lia speech recognition system: from 10xrt to 1xrt, in: Text, Speech and Dialogue, Springer. pp. 302–308.

Maza et al. (2011)

Maza, B., El-Bèze, M., Linarès, G., De Mori, R., 2011.

On the use of linguistic features in an automatic system for speech analytics of telephone conversations, in: INTERSPEECH, pp. 2049–2052.

Min and Ma (2013)

Min, K., Ma, Y., 2013.

Joint topic-document modeling via low-dimensional sparse models, in: Int. Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 8590–8594.

Mittendorf and Schäuble (1994)

Mittendorf, E., Schäuble, P., 1994.

Document and passage retrieval based on hidden markov models, in: In Proceedings of the 17th Annual Int. ACM SIGIR Conference on Res. and Development in Inf. Retr., Springer. pp. 318–327.

Morchid et al. (2014a)

Morchid, M., Bouallegue, M., Dufour, R., Linarès, G., Matrouf, D., de Mori, R., 2014a.

An i-vector based approach to compact multi-granularity topic spaces representation of textual documents, in: Proceedings of the 2014 Conference on Emp. Methods in Nat. Lang. Processing, Association for Computational Linguistics. pp. 443–454.

Morchid et al. (2014b)

Morchid, M., Dufour, R., Bousquet, P.M., Bouallegue, M., Linarès, G., De Mori, R., 2014b.

Improving dialogue classification using a topic space representation and a gaussian classifier based on the decision rule, in: Int. Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 126–130.

Niekrasz and Moore (2010)

Niekrasz, J., Moore, J.D., 2010.

Unbiased discourse segmentation evaluation, in: Spok. Lang. Technology Workshop (SLT), IEEE. pp. 43–48.

Purver (2011)

Purver, M., 2011.

Topic segmentation.

Spok. Lang. Underst.: Syst. for Extracting Semantic Inf. from Speech , 291–317.

Tsoumakas and Katakis (2007)

Tsoumakas, G., Katakis, I., 2007.

Multi-label classification: An overview.

Int. J. of Data Warehous. and Min. (IJDWM) 3, 1–13.

Tur and De Mori (2011)

Tur, G., De Mori, R., 2011.

Spoken language understanding: Systems for extracting semantic information from speech.

John Wiley & Sons.

Tur and Hakkani-Tür (2011)

Tur, G., Hakkani-Tür, D., 2011.

Human/human conversation understanding.

Spok. Lang. Underst.: Syst. for Extracting Semantic Inf. from Speech , 225–255.

Wallace et al. (2013)

Wallace, B.C., Trikalinos, T.A., Laws, M.B., Wilson, I.B., Charniak, E., 2013.

A generative joint, additive, sequential model of topics and speech acts in patient-doctor communication, in: Proceedings of the 2013 Conference on Emp. Methods in Nat. Lang. Processing, Association for Computational Linguistics. pp. 1765–1775.

Wintrode (2011)

Wintrode, J., 2011.

Using latent topic features to improve binary classification of spoken documents, in: Int. Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 5544–5547.

Wintrode and Kulp (2009)

Wintrode, J., Kulp, S., 2009.

Techniques for rapid and robust topic identification of conversational telephone speech, in: INTERSPEECH, pp. 1471–1474.

Yang and Pedersen (1997)

Yang, Y., Pedersen, J.O., 1997.

A comparative study on feature selection in text categorization, in: ICML, pp. 412–420.

Vitae

**Xavier Bost

**With a background in Humanities, Xavier Bost began taking interest in computer science when attempting to automate processes used in historical linguistics. He is interested in language and image understanding working as a Ph.D. student at the lia (University of Avignon, France) on automatic moving picture summarization.

**Grégory Senay

**Grégory Senay received his Ph.D. in 2011 from lia at the University of Avignon, France and then worked as a postdoctoral fellow for two years. His research interests are in the areas of automatic speech recognition, spoken document retrieval, keyword spotting and topic classification with emphasis on speech confidence measure and audio indexing.

**Marc El-Bèze

**Marc El-Bèze is a full professor at the University of Avignon (uapv), France. He is an expert in several fields related to natural language processing: automatic text categorization, question answering, opinion analysis, and recommender systems. With some of his colleagues, he has participated to many evaluation campaigns such as trec, clef, duc, and more recently to RepLab. He has also been responsible for the French part of the European project EuroWordNet.

**Renato De Mori

**Renato De Mori is emeritus professor at Mc Gill University (Canada) and at the University of Avignon (France). He is a Fellow of the Computer Society and has been distinguished lecturer of the Signal Processing Society of the Institute of Electrical and Electronic Engineers (ieee). He is actually an Associate Editor of the ieee Transactions on Audio, Speech and Language.

His major contributions have been in the area of Automatic Speech Recognition and Understanding, Signal Processing, Computer Arithmetic, Software Engineering and Human-Machine Interfaces.

Bibliography35

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1Bahl et al. (1988) Bahl, R., Bakis, R., De Souza, P., Mercer, R., 1988. Obtaining candidate words by polling in a large vocabulary speech recognition system, in: Int. Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 489–492.
2Balchandran et al. (2010) Balchandran, R., Rachevsky, L., Ramabhadran, B., Novak, M., 2010. Techniques for topic detection based processing in spoken dialog systems., in: INTERSPEECH, pp. 82–85.
3Bellegarda (2000) Bellegarda, J.R., 2000. Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE 88, 1279–1296.
4Blei et al. (2003) Blei, D.M., Ng, A.Y., Jordan, M.I., 2003. Latent dirichlet allocation. the J. of machine Learning res. 3, 993–1022.
5Bost et al. (2013) Bost, X., El-Bèze, M., De Mori, R., 2013. Multiple topic identification in telephone conversations, in: INTERSPEECH, pp. 3756–3760.
6Camelin et al. (2009) Camelin, N., De Mori, R., Béchet, F., Damnati, G., 2009. Error correction of proportions in spoken opinion surveys., in: INTERSPEECH, pp. 2715–2718.
7de Carvalho and Freitas (2009) de Carvalho, A.C., Freitas, A.A., 2009. A tutorial on multi-label classification techniques, in: Found. of Computational Intelligence Volume 5. Springer, pp. 177–195.
8Chen (2010) Chen, H., 2010. Learning semantic structures from in-domain documents. Ph.D. thesis. Massachusetts Institute of Technology.