Bayesian Tensor Factorisation for Bottom-up Hidden Tree Markov Models

TL;DR

This paper introduces a tensor factorisation-based approximation for Bottom-Up Hidden Tree Markov Models, enabling practical use and improved performance on tree-structured data tasks.

Contribution

It proposes a novel Tucker tensor factorisation approach to approximate the intractable state-transition matrix in Hidden Tree Markov Models, with a new probabilistic model and learning algorithm.

Findings

The new model outperforms existing approximations on two tasks.

Empirical results demonstrate improved accuracy and efficiency.

The approach enables practical application of complex tree models.

Abstract

Bottom-Up Hidden Tree Markov Model is a highly expressive model for tree-structured data. Unfortunately, it cannot be used in practice due to the intractable size of its state-transition matrix. We propose a new approximation which lies on the Tucker factorisation of tensors. The probabilistic interpretation of such approximation allows us to define a new probabilistic model for tree-structured data. Hence, we define the new approximated model and we derive its learning algorithm. Then, we empirically assess the effective power of the new model evaluating it on two different tasks. In both cases, our model outperforms the other approximated model known in the literature.

Figures

Tables

Table 1. TABLE I : Average accuracy and entropy over 5 runs (std in brackets) on INEX05 dataset. In bold the best result for each model

	INEX05
Hidden	Accuracy ($\%$)		Entropy ($\%$)
	SP-BU	TU-BU	TF-BU	TU-BU
$C=2$	$87.55\left(4.13\right)$	$91.41\left(3.62\right)$	$34.19\left(5.80\right)$	$31.99\left(10.37\right)$
$C=4$	$90.68\left(5.70\right)$	$93.65\left(1.64\right)$	$28.92\left(5.16\right)$	$25.00\left(4.13\right)$
$C=6$	$93.81\left(1.15\right)$	$95.10\left(0.27\right)$	$24.30\left(2.90\right)$	$20.91\left(0.93\right)$
$C=8$	$93.15\left(1.69\right)$	$94.28\left(2.27\right)$	$23.59\left(1.29\right)$	$23.12\left(6.35\right)$
$C=10$	$93.30\left(3.09\right)$	$95.21\left(0.17\right)$	$21.77\left(1.64\right)$	$20.60\left(0.46\right)$

Table 2. TABLE II : Average accuracy and entropy over 5 runs (std in brackets) on INEX06 dataset. In bold the best result for each model

	INEX06
Hidden	Accuracy ($\%$)		Entropy ($\%$)
	SP-BU	TU-BU	SP-BU	TF-BU
$C=2$	$21.44\left(4.54\right)$	$27.94\left(2.62\right)$	$287.40\left(7.24\right)$	$275.28\left(2.07\right)$
$C=4$	$24.84\left(3.15\right)$	$29.83\left(3.06\right)$	$281.23\left(3.46\right)$	$279.82\left(7.11\right)$
$C=6$	$25.57\left(2.12\right)$	$30.65\left(2.36\right)$	$277.99\left(2.81\right)$	$283.47\left(5.34\right)$
$C=8$	$26.43\left(2.47\right)$	$26.48\left(2.84\right)$	$278.17\left(1.12\right)$	$292.36\left(5.56\right)$
$C=10$	$22.89\left(3.33\right)$	$26.94\left(2.77\right)$	$289.40\left(6.28\right)$	$291.03\left(7.12\right)$

Table 3. TABLE III : Average label accuracy over 5 runs (std in brackets) on the synthetic dataset. In bold the best result for each model

	Synthetic dataset
Label	Accuracy ($\%$)		Entropy ($\%$)
	SP-BU	TU-BU	SP-BU	TF-BU
$0$	$55.47\left(12.90\right)$	$99.67\left(0.11\right)$	$35.49\left(8.25\right)$	$1.34\left(2.03\right)$
$1$	$60.15\left(2.22\right)$	$79.59\left(22.35\right)$	$151.51\left(13.14\right)$	$65.61\left(61.25\right)$
$2$	$45.84\left(4.43\right)$	$64.71\left(27.17\right)$	$180.88\left(5.40\right)$	$93.69\left(51.19\right)$
$3$	$17.04\left(2.05\right)$	$29.68\left(38.22\right)$	$184.08\left(4.93\right)$	$97.71\left(44.38\right)$
All	$53.08\left(4.83\right)$	$80.68\left(15.77\right)$	$140.38\left(10.39\right)$	$51.42\left(34.91\right)$

Equations

$$\begin{array}{rl}& \mathcal{L}(\mathbf{x},\mathbf{Q})=P(\mathbf{x},\mathbf{Q})=\prod _{u\in \mathcal{L}\mathcal{F}}{\pi }^l(j)b_j(x_u)\\ & \times \prod _{u\notin \mathcal{L}\mathcal{F}}{A}_{j_1,\dots ,j_L}(j)b_j(x_u),\end{array}$$

$$\begin{array}{rl}& P(Q_u=j\mid Q_{c{h}_1(u)}=j_1,\dots ,Q_{c{h}_L(u)}=j_L)\\ & =\sum _{l=1}^{L}P(S_u=l)P(Q_u=j\mid S_u=l,Q_{c{h}_l(u)}=j_l).\end{array}$$

$$a_{x_1,\dots ,x_p}=\sum _{h_1=1}^{J_1}\cdots \sum _{h_p=1}^{J_p}{\lambda }_{h_1,\dots ,h_p}\prod _{z=1}^p{\kappa }_{h_z,x_z}^{(z)},$$

$$\mathcal{A}=\mathbf{\lambda }{\times }_1{\mathbf{\kappa }}^{(1)}{\times }_2{\mathbf{\kappa }}^{(2)}{\times }_3\cdots {\times }_p{\mathbf{\kappa }}^{(p)}.$$

$$\begin{array}{rl}& P(Q_u=j\mid Q_{c{h}_1(u)}=j_1,\dots ,Q_{c{h}_L(u)}=j_L)=\\ & \sum _{i_1=1}^{k_1}\cdots \sum _{i_L=1}^{k_L}P(Q_u=j\mid z_{c{h}_1(u)}=i_1,\dots ,z_{c{h}_L(u)}=i_L)\\ & \times \prod _{l=1}^{L}P(z_{c{h}_l(u)}=i_l\mid Q_{c{h}_l(u)}=j_l)=\\ & \sum _{i_1=1}^{k_1}\cdots \sum _{i_L=1}^{k_L}{\lambda }_{i_1,\dots ,i_L}(j)\prod _{l=1}^L{\kappa }_{j_l}^l(i_l).\end{array}$$

$$\begin{array}{rl}& \mathcal{L}(\mathbf{x},\mathbf{Q},\mathbf{z}\mid \theta )=P(\mathbf{x},\mathbf{Q},\mathbf{z}\mid \theta )=\prod _{u\in \mathcal{L}\mathcal{F}}{\pi }^l(j)b_j(x_u)\\ & \times \prod _{u\in \mathcal{U}\setminus \mathcal{L}\mathcal{F}}{\lambda }_{i_1,\dots ,i_L}(j)\prod _{l=1}^L{\kappa }_{j_l}^l(i_l)b_j(x_u).\end{array}$$

$$p(k_l)$$

$$\sum _{l=1}^L\mathbb{I}[k_l\ne 1]$$

$${\lambda }_{i_1,\dots ,i_L}$$

$${\mathbf{\lambda }}_0$$

$$\min \left\{{\left[\frac{{\prod }_u{\lambda }_{z_l^{\prime }}(Q_u^{\prime })}{{\prod }_u{\lambda }_{z_l}(Q_u)}\frac{{\prod }_u{\lambda }_{z_l^{\prime }}(Q_u)}{{\prod }_u{\lambda }_{z_l}(Q_u^{\prime })}\right]}^{1/\mathcal{T}(m)},1\right\},$$

$$\min \left\{{\left[\frac{\mathcal{L}({\mathbf{k}}^{\prime })}{\mathcal{L}(\mathbf{k})}\frac{{\prod }_{l}p(k_l^{\prime })}{{\prod }_{l}p(k_l)}\right]}^{1/\mathcal{T}(m)},1\right\},$$

$$\mathcal{L}(\mathbf{k})=\prod _{{\mathbf{i}}_{\mathbf{u}}}\frac{\mathbf{B}(\alpha {\lambda }_0(1)+n_{{\mathbf{i}}_{\mathbf{u}}}(1),\dots ,\alpha {\lambda }_0(C)+n_{{\mathbf{i}}_{\mathbf{u}}}(C))}{\mathbf{B}(\alpha {\lambda }_0(1),\dots ,\alpha {\lambda }_0(C))},$$

$${\pi }^l$$

$$b_j$$

$${\lambda }_{{\mathbf{i}}_{\mathbf{u}}}$$

Full text

I Introduction

Trees are complex data which represent hierarchical information. They are composed of atomic entities, called nodes, combined together through a parent-child relationship which defines the tree structure. Developing a Machine Learning model which is able to deal with such rich representation poses two main challenges [11]: (1) the model must be able to adapt itself on different tree structures and (2) the model must exploit the structure information to absolve the given task.

Both challenges have been tackled over the years by a variety of approaches ranging from early works on recursive neural networks [11], to their more recent deep learning style re-factoring [17] (see [2] for a recent survey), and including kernel-based approaches [12, 16, 1] and generative models [8, 5, 6]. In this work, we focus in particular on the latter models, which are referred to as Hidden Tree Markov Models (HTMM), being obtained as a generalisation of the Hidden Markov Model (HMM) to the tree domain. Like their sequential counterpart, HTMM defines a generative process for tree structures regulated by discrete-valued hidden state variables, typically from a finite alphabet (but can be extended to non-parametric models along the lines of [3]). Structural knowledge is captured by such hidden states through their transition distribution, whose representational power and complexity depend on the direction of context propagation. In the literature, we mainly refer to two types of HTMM associated with different context propagation strategies. The Top-Down Hidden Tree Markov Model (TD-HTMM) [8] defines a generative process propagating from the root to the leaves of the tree. The top-down direction defines a causal relationship among nodes which goes from the parent to each child independently. On the other hand, the Bottom-Up Hidden Tree Markov Model (BU-HTMM) [5] defines a generative process which goes in the opposite direction: from the leaves to the root. The different direction induces more complex causal relationships between nodes, pointing from the joint state of the children towards the common parent. Different casual relationships create differences in the local Markov properties of the two approaches, which influence the way the nodes exchange information during inference and learning [5].

While possessing several interesting properties, e.g. compositionality [6], the bottom-up HTMM has practical limitations due to the size of the state-transition distribution which is exponential with respect to the output degree (i.e. the maximum number of child nodes). To overcome this limitation, the authors in [5] introduced an approximation, named Switching Parent (SP), which factorises the complex joint state transition distribution as a mixture of simpler multinomials. While having demonstrated its effectiveness in a number of applications, including generalisation to graphs [4], such an approximation limits the amount of information shared between the children of common parents.

The goal of this paper is to introduce a new approximation for the BU-HTMM state-transition distribution which relies on a tensor decomposition known as Tucker decomposition [18]. This approximation is interesting because of its probabilistic interpretation, highlighted in [19, 15], as well as for the increased information sharing between children with respect to the SP factorisation. Such an advantage comes at the cost of a more complex learning algorithm, due to the use of Gibbs sampling when estimating model parameters from data. As part of this work, we validate our proposed Bayesian tensor factorisation approach on two different learning tasks. The former is a standard tree classification task, while the latter tests the model ability to predict node labels given a tree structure. In both tasks, we compare our model to the BU-HTMM model with SP approximation introduced in [5]. Results highlight how the proposed tensor factorisation outperforms the SP model, especially in the label prediction task.

The rest of the paper is organised as follows. In Section II, we introduce the two building blocks of our new model. In Section III we formally introduce our model and we derive its learning algorithm. In Section IV we report the empirical analysis, while in Section V we draw our conclusions.

II Background

II-A Bottom-Up Hidden Tree Markov Model

The Bottom-Up Hidden Tree Markov Model (BHTMM) [5] defines a probability distribution over tree-structured data by postulating the existence of a hidden generative process regulated by unobserved Markov state random variables.

The terms Bottom-Up refers to the direction of the conditional dependence among tree nodes. In particular, the BHTMM assumes that the hidden state of a node depends on the joint hidden state of its direct descendants in the tree.

Given a labelled tree $t$, we denote with $\mathbf{x}=\{x_{1},\dots,x_{|t|}\}$ the set of visible labels in $t$. Then, its BHTMM graphical model is built associating a discrete hidden random variable $Q_{u}\in[1,C]$ to each label $x_{u}$ in the tree. All hidden variables are linked together reproducing the same structure of the visible tree $t$; the direction of links goes from leaves to the root in order to represent the dependence between the hidden state of a node and the joint configuration of its hidden child nodes (see Fig. 1).

Given the graphical model, the derivation of the complete likelihood associated to labels $\mathbf{x}$ is straightforward:

[TABLE]

where we use $\mathbf{Q}=\{Q_{1},\dots,Q_{|t|}\}$ to indicate the set of the hidden variables introduced by the BHTMM. The term $L$ is the maximum tree output degree (i.e. the maximum number of children).

The distributions in (1), as in every HMM, are: the priori distribution $\pi^{l}(j)=P^{l}(Q_{u}=j)$, the state-transition distribution $A_{j_{1},\dots,j_{L}}(j)=P(Q_{u}=j\mid Q_{ch_{1}(u)}=j_{1},\dots,Q_{ch_{L}(u)}=j_{L})$, and the emission distribution $b_{j}(x_{u})=P(x_{u}\mid Q_{u}=j)$. In our case, the priori distribution is defined on leaf nodes: we use the symbol $\mathcal{LF}(t)$ to indicate the set of leaf nodes in $t$. Moreover, we use a different priori distribution for each node position: the $l$ superscript indicates the positional dependence of the priori distribution. To our purpose, the position of a node is computed with respect to its siblings: we denote by $ch_{l}(u)$ the $l$-th child node of $u$ and by $l=pos(u)$ the position of node $u$ w.r.t. its siblings. We will omit the dependence on $u$ when it clear from the context or it is not relevant.

The state-transition distribution models the relationships between the hidden state of a node and the joint hidden state of its child nodes; unfortunately, this relationship cannot be represented explicitly. In fact, given $C$ the number of hidden states and $L$ the maximum tree output degree, the state-transition distribution is represented by a tensor which contains $C^{L+1}$ elements. The exponential growth with respect to the number of child nodes $L$ is a direct consequence of the assumption made by the model since there is an exponential number of joint state configurations.

To this end, an approximation is required to make the Bottom-Up approach feasible in practice. The first approximation introduced is called Switching-Parent [5]: the idea is to approximate of the joint transition distribution as a mixture of simpler distributions. In practice, such approximation assumes the existence of a mixture variable $S_{u}$ for each tree internal nodes. Then, the joint transition matrix factorises as a convex combination of $L$ elementary transition matrices (one for each child), where the mixture distribution $P(S_{u}=l)$ represents the weight of such combination. In formula, the following simplification is applied:

[TABLE]

This approximated model is referred to as SP-BHTMM.

II-B Tucker Decomposition

Let $\mathcal{A}\in\mathbb{R}^{I_{1}\times\dots\times I_{p}}$ a $p$-way tensor, each element $a_{x_{1},\dots,x_{p}}$ in $\mathcal{A}$ can be factorised as [9]

[TABLE]

where $\lambda_{h_{1},\dots,h_{p}}$ are elements of a new $p$-way tensor $\boldsymbol{\lambda}\in\mathbb{R}^{J_{1}\times\dots\times J_{p}}$ called core tensor and $\kappa^{(z)}_{h_{z},x_{z}}$ are elements of matrices $\boldsymbol{\kappa}^{(z)}\in\mathbb{R}^{I_{z}\times J_{z}}$, called mode matrices.

A more compact way to represent the same decomposition can be obtained using the operator $\times_{n}$, which represents $n$-mode product of a tensor by a matrix along the $n$-th dimension [9]:

[TABLE]

An example of decomposition can be visualised in Figure 2.

This factorisation is called Tucker decomposition, since it was originally developed by Tucker to obtain a method for searching relations in a three-way tensor of psychometric data [18]. Later, the same idea has been generalised to $p$-way tensor proving that it is a multi-linear generalisation of the matrix Singular Value Decomposition (SVD) [9]. Hence, the same decomposition is also called High-Order Singular Value Decomposition (HOSVD) [9].

The major advantage of the Tucker decomposition is that it may lead to a compressed representation of the original tensor. In fact, storing the tensor $\mathcal{A}$ in its explicit form requires $\prod_{z=1}^{p}I_{z}$ space. Instead, storing its factorised form requires $\prod_{z=1}^{p}J_{z}$ space to store $\boldsymbol{\lambda}$ tensor plus $\sum_{z=1}^{p}I_{z}\times J_{z}$ to store all the $\boldsymbol{\kappa}^{(.)}$ matrices. Thus, the Tucker factorisation is advantageous if and only if $\prod_{z=1}^{p}J_{z}\ll\prod_{z=1}^{p}I_{z}$.

Even if we have the guarantee that a Tucker decomposition exists for every tensor [9], we have no guarantee on the size of its core tensor. In the worst case, the core tensor has the same dimension as the initial one; hence, there is no compression using the Tucker decomposition.

III The Tensor Factorisation BHTMM

III-A Model

The Tensor Factorisation Bottom-up Hidden Tree Markov Model (TF-BHTMM) defines a probability distribution over tree-structured data. As for BHTMM, the direction of conditional dependencies among tree nodes goes from the leaves to the root. Moreover, the Tucker decomposition is applied in order to reduce the complexity required to store a BHTMM state-transition distribution.

To this end, given a labelled tree $t$, the TF-BHTMM introduces a new set of discrete hidden random variable $\mathbf{z}=\{z_{1},\dots,z_{|t|}\}$. The goal of these variables is to break the dependence between a node and the joint configuration of its children. Hence, the state of a variable $z_{u}$ depends only on the state of the variable $Q_{u}$, while the state of a variable $Q_{u}$ depends on the joint configuration $(z_{ch_{1}},\dots,z_{ch_{L}})$ associated to its children. In Figure 3 we show the Markov blanket associated to a generic tree node $u$. By observing this figure, it is clear that the every path between the variable $Q_{u}$ and every child variable $Q_{ch_{l}}$ is blocked by the variable $z_{ch_{l}}$. Hence, we can argue that the variable $Q_{u}$ is conditionally independent of the joint configuration $(Q_{ch_{1}},\dots,Q_{ch_{L}})$ when the joint configuration $(z_{ch_{1}},\dots,z_{ch_{L}})$ is observed.

The interaction between the $\mathbf{z}$ and the $\mathbf{Q}$ variables is modelled trough the categorical distributions $\lambda_{i_{1},\dots,i_{L}}(j)=P(Q_{u}=j\mid z_{ch_{1}(u)}=i_{1},\dots,z_{ch_{L}(u)}=i_{L})$ and $\kappa^{l}_{j}(i)=P^{l}(z_{u}=i\mid Q_{u}=j)$. The symbols uses to represent these distributions are intentionally the same as those of the Tucker decomposition parameters since there is a strong connection between them. In fact, if we derive the state-transition distribution $P(Q_{u}\mid Q_{ch_{1}(u)},\dots,Q_{ch_{L}(u)})$ according to the Markov blanket in Figure 3, we obtain:

[TABLE]

The decomposition obtained in (5) is similar to the one obtained in (3): the only difference is that in the latter one both the core tensor $\boldsymbol{\lambda}$ and the mode matrices $\boldsymbol{\kappa}^{l}$ are probability distributions. The value $k_{l}$ represents the size of the core tensor $\boldsymbol{\lambda}$ along the $l$-th dimension. This Bayesian counterpart of the Tucker decomposition has been already introduced to approximate the conditional distribution in a classification problem [19] and the state-transition distribution of High-Order Hidden Markov Model [15].

To deeply understand the probabilistic interpretation of our approximation, it is worth to observe that the states of $z_{ch_{l}}$ represent clusters of $Q_{ch_{l}}$ states. If we denote with $\mathcal{C}^{l}_{i}$ the cluster associated to the $i$-th value of $z_{ch_{l}}$, the value $\kappa^{l}_{j}(i)$ indicates the probability that the state $Q_{ch_{l}}=j$ belongs to the $i$-th cluster. All the states in the same cluster are indistinguishable from the state-transition point of view since the parent state depends only on the cluster (i.e. the state of $z_{ch_{l}}$). Hence, each cluster contains all the $Q_{ch_{l}}$ states that bring the same information to the parent state transition. This interpretation makes clear why the Tucker approximation is able to compress the space required to store the state-transition distribution preserving a probabilistic interpretation.

Finally, using the BHTMM complete likelihood equation in (1) and the approximation introduced in (5), we can derive the complete likelihood equation for the TF-BHTMM:

[TABLE]

We use the symbol $\theta=\{\boldsymbol{\pi},\mathbf{b},\boldsymbol{\lambda},\boldsymbol{\kappa^{1}},\dots,\boldsymbol{\kappa^{L}}\}$ to denote the set of parameters which define a TF-BHTMM.

III-B Decomposition Size

The values $\mathbf{k}=\{k_{1},\dots,k_{L}\}$ (i.e. the size of the core tensor $\boldsymbol{\lambda}$) play a fundamental role in the compression of the state-transition distribution. In fact, as we discussed in Section II-B, the space required to store the Tucker factorisation depends on the size of the core tensor. In our case, it is $C\times\prod_{l=1}^{L}k_{l}$. Moreover, these quantities have a nice probabilistic interpretation [15]: the value $k_{l}$ measures how strong is the dependence between the hidden state $Q_{ch_{l}}$ and its hidden parent state $Q_{u}$. In fact, a value of $k_{l}=C$, means that it is important to know the exact child state $Q_{ch_{l}}$ in order to determine the parent state. On the contrary, if $k_{l}=1$, the child state does not affect the parent one: no matter the true value of $Q_{ch_{l}}$, it will collapse in the unique state available for $z_{ch_{l}}$.

For ease of exposition, in Section III-A, we have assumed the true size of the core tensor to be known. In practical application, these quantities are unknown. To overcome this limitation, we treat them as discrete hidden random variables, hence the Bayesian nature of our model. To this end, we specify a priori distribution on them and we learn the most suitable values during the training procedure.

The prior distributions on such variables are very important since they force the model to focus either on compression (i.e. the $\mathbf{k}$ values are almost $1$) or data representation (i.e. the $\mathbf{k}$ values are almost $C$). Moreover, we can easily insert prior knowledge: for example, we can build a prior distribution which is position dependent to indicate there are children positions that are more informative than the others.

In our model, we decide to use the following position-independent priori:

[TABLE]

The value $\varphi$ is an hyper-parameter which regulates the distribution decay. The hyper-parameters $L_{\text{min}}$ and $L_{\text{max}}$ denote the minimum and the maximum number of important positions, respectively. The utility of $L_{\text{min}}$ and $L_{\text{max}}$ is twofold: (i) we can avoid that the parent hidden state is completely independent from the child hidden states (by setting $L_{\text{min}}\geq 1$), and (ii) we can control the worst case space-complexity (which is $O(C^{L_{\text{max}}+1})$).

By varying the value of $k_{l}$, we change the dimension of the core tensor $\boldsymbol{\lambda}$. Hence, we need to specify a prior distribution on it in order to generate new entries on the fly. Since $\boldsymbol{\lambda}$ is a categorical distribution, we use its conjugate prior, i.e. the Dirichlet distribution

[TABLE]

The value $\alpha$ and $\alpha_{0}$ are hyper-parameters which regulate the shape of the $\boldsymbol{\lambda}$ distribution.

III-C Parameter Learning

Learning of TF-BHTMM parameters is straightforward if the vector dimension $\mathbf{k}$ is known. As we discussed in the previous section, in practical application these quantities are unknown. Therefore, we develop a learning procedure which is able to learn the approximation size from data.

Unfortunately, the estimation of the vector $\mathbf{k}$ worsens the learning problem. To this end, we introduce an approximation on the mode matrices $\boldsymbol{\kappa^{1}},\dots,\boldsymbol{\kappa^{L}}$. The approximation consists in adding the constraint $\tilde{\kappa}^{l}_{j}({i})=\{0,1\}$ to each entry of the mode matrices [15]. This new constraint makes the clustering discussed in Section III-A deterministic. In fact, we can associate each state to a single cluster and define each cluster as $\mathcal{C}^{l}_{i}=\{j\mid\tilde{\kappa}^{l}_{j}({i})=1\}$. The approximated mode matrices $\tilde{\mathbf{\kappa}}^{l}$ take the name of hard clustering, which is opposed to the soft clustering defined by the mode matrices without approximation.

The approximation above allows us to develop a Gibbs sampling algorithm for fitting the unknown quantities from data. The procedure comprises the following steps:

update the hidden state $\mathbf{Q},\mathbf{z}$; 2. 2.

update the approximation size $\mathbf{k}^{\prime}$ and the mode matrices $\boldsymbol{\kappa^{1}},\dots,\boldsymbol{\kappa^{L}}$; 3. 3.

update the model parameters $\boldsymbol{\pi},\boldsymbol{\lambda},\boldsymbol{\lambda_{0}},\mathbf{b}$.

For the sake of simplicity, we discuss the learning algorithm restricting to training set with a single tree. The extension to a generic i.i.d. dataset is straightforward.

In the first step we perform a Simulated Annealing [14] update of latent variables. Given the current values of $(\mathbf{Q},\mathbf{z},\mathbf{k},\theta)$, we compute the new values of $\mathbf{Q}^{\prime},\mathbf{z}^{\prime}$ through an ancestor sampling procedure [7] starting from leaf nodes. In particular, we sample the new $\mathbf{Q}^{\prime}$ fixing the old values $\mathbf{z}$. Then, we use $\mathbf{Q}^{\prime}$ to sample the new values $\mathbf{z}^{\prime}$. More details are given in Algorithm 1. The new values $\mathbf{Q}^{\prime},\mathbf{z}^{\prime}$ are accepted with probability [15]

[TABLE]

where $\mathcal{T}(0)$ and $\mathcal{T}(m)=\max\{\mathcal{T}_{0}^{1-m/m_{0}},1\}$ denotes the initial and the current annealing temperature, $m$ is the current iteration number and $m_{0}$ is the iteration at which the temperature reduces to one.

In the second step, we perform a Stochastic Search for Variable Selection (SSVS) [13] to update the compression size. The approximation size $\mathbf{k}$ is updated choosing a random position $l$. Then, we decide to do an increase (or decrease) move on the position $l$ by a coin toss. The increasing move consists in adding a new cluster for the $l$ position. Hence, we randomly select a cluster $\mathcal{C}^{l}_{i}$ and we randomly split it into two clusters. The splitting process consists to randomly select a set of element in the cluster $\mathcal{C}^{l}_{i}$ and to move them in the new cluster $\mathcal{C}^{l}_{i^{\prime}}$. In practice, it is implemented modifying the hard clustering $\tilde{\mathbf{\kappa}}^{l}$: for every state $j$ which is moving from cluster $i$ to cluster $i^{\prime}$, we set $\tilde{\kappa}^{l}_{j}(i)=0$ and $\tilde{\kappa}^{l}_{j}(i^{\prime})=1$. On the contrary, the decrease move merges together two randomly selected clusters $\mathcal{C}^{l}_{i}$ and $\mathcal{C}^{l}_{i^{\prime}}$. Again, the merging operation is implemented modifying the hard clustering $\tilde{\mathbf{\kappa}}^{l}$: for every state $j$ in the cluster $i^{\prime}$, we set $\tilde{\kappa}^{l}_{j}(i^{\prime})=0$ and $\tilde{\kappa}^{l}_{j}(i)=1$.

Finally, we should guarantee that the constraint (8) is satisfied: Algorithm 2 provides more details on the process. The new values $\mathbf{k}^{\prime}$ are accepted with probability [15]

[TABLE]

where $\mathcal{T}(m)$ is the current annealing temperature which is computed as in the previous step. The priori $p(k_{l})$ is defined in (7), while the marginal likelihood is given by [15]:

[TABLE]

where $\mathbf{i_{u}}=(i_{1},\dots,i_{L})$. The function $\mathbf{B}(\cdot)$ represents the multivariate Beta function and the value $n_{\mathbf{i_{u}}}(j)$ counts how many times the joint configuration $(Q_{u}=j,Q_{ch_{1}}=i_{1},\dots,Q_{ch_{L}}=i_{l})$ appears.

In the last step, we update the model parameters sampling from their multinomial conditional. We assume that both the priori distribution $\boldsymbol{\pi}$ and the emission distribution $\mathbf{b}$ have a Dirichlet priori with parameters $\gamma$ and $\beta$, respectively, yielding

[TABLE]

where $n^{l}(c)=\sum_{u\in\mathcal{LF}}\mathbb{I}[Q_{u}=c\,\wedge\,pos(u)=l]$ and $n_{j}(d)=\sum_{u}\mathbb{I}[Q_{u}=j\wedge x_{u}=d]$. The operator $\mathbb{I}$ is the indicator function whose values is $1$ if and only if the condition in its square brackets is true.

The sampling of the base distribution $\boldsymbol{\lambda_{0}}$ requires a more complex procedure which is outlined in Algorithm 3.

IV Experimental Analysis

In this section, we provide an experimental evaluation of the proposed model, with the aim of characterising the properties of the approximation introduced against the SP one. In particular, we evaluate the model on two different tasks: (1) a classification task on XML tree-data and (2) a labelling task on a synthetic dataset. In both tasks, we use two measures to evaluate the model performance: the accuracy, which assess the correctness of model’s answers, and the entropy, which measures the amount of uncertainty in them. For the accuracy measures, higher values are better; for the entropy, lower values are better.

All the reported results are averaged over five executions, to account for randomisation effects due to initialisation. The MATLAB code used can be found on a public GitLab repository111https://gitlab.itc.unipi.it/d.castellana/TF_bhtmm.

IV-A Classification Task

The classification task consists in predicting the class a tree-structured sample belongs to. We test both the SP-BHTMM and TF-BHTMM on two datasets taken from the INEX 2005 (INEX05) and INEX 2006 (INEX06) competition [10]. The INEX05 dataset is based on the (m-db-s-0) corpus, comprising $9631$ XML-formatted documents represented as trees with maximum output degree $L=32$ and assigned to $11$ different cluster. Node labels represent $366$ different XML tags. The dataset is split in training set ($4820$ trees) and test set ($4811$ trees) [10]. The INEX06 is composed of $12107$ XML- formatted documents representing scientific articles, each from one of $18$ different IEEE journals which represent the different cluster. Again, the node labels represent different $65$ XML tags and maximum output degree $L=66$. The dataset is split in training set ($6,053$ trees) and test set ($6,054$ trees) [10].

On both datasets, we train a single model for each class. Each model is trained on the elements in the training set associated with the same model class; at test time, we compute the likelihood according to each model and we assign the class of the model which scores the highest sample likelihood. For all the models, we set a flat prior on model parameters. Also we set $\varphi=2$ (as in [15]), $L_{\text{min}}=1$, and $L_{\text{max}}=5$ for the TF-BHTMM. The values assigned to $L_{\text{min}}$ and $L_{\text{max}}$ allow to bound the space complexity of the model to $O(C^{6})$.

To evaluate the impact of the hidden state size, we vary the number of hidden states $C\in\{2,4,6,8,10\}$. Both training algorithms are executed for $100$ iterations.

In Table I we report the accuracy and the entropy obtained on the test set. The results show that the TF-BHTMM always outperforms the SP-BHTMM, both in accuracy and in entropy: the difference is higher when the number of hidden states is small. This results is quite surprising if we consider that the TF-BHTMM considers at most $5$ child nodes among $32$ (we set $L_{\text{max}}=5$); on the contrary, the SP-BHTMM always consider the contribution of all child nodes. Hence, we deduce that not all the child nodes contain useful information for the tree-classification and therefore can be ignored.

The results obtained on the INEX06 dataset (see Table II) are similar to the ones obtained on the INEX05 dataset: the TF-BHTMM always reach a higher accuracy than the SP-BHTMM. On the contrary, the entropy values are high for both model due to the intrinsic difficulty of the INEX06 dataset [5].

IV-B Labelling Task

The labelling task consists in predicting the visible label associated with the nodes of a given tree structure. We test both the SP-BHTMM and the TF-BHTMM on a controlled dataset which contains ternary trees (i.e. L=3), comprising left-asymmetric, symmetric and right-asymmetric tree. A tree is defined as left-asymmetric (right-asymmetric) if the number of nodes in the leftmost (rightmost) position is greater than the number of nodes in the opposite position. In a symmetric tree, the number of nodes is almost equivalent for each position. The label of each node encodes structural information since it represents the number of children of the node: therefore the label goes from [math] (i.e. no child nodes) to $3$ (i.e. a child node in each position). We generate $780$ trees ($260$ for each type) and split them in training set ($600$ trees, $200$ for each type) and test set ($180$ trees, $60$ for each type).

We train a SP-BHTMM and a TF-BHTMM with $C=10$ hidden states. In this experiments, we do not test multiple hyper-parameter configurations: for both models, we use flat priors to generate the initial probability distribution. Also we set $\varphi=2$ (as in [15]), $L_{\text{min}}=1$, and $L_{\text{max}}=3$ for the TF-BHTMM. The values assigned to $L_{\text{min}}$ and $L_{\text{max}}$ allow to consider all child position. Both training algorithms are executed for $100$ iteration. At test time, we remove the visible labels from the test data and we generate them using the probability distribution learned by both models. In Table III, we report accuracy and entropy obtained by both models on each label type.

The overall accuracy reached by SP-BHTMM and TF-BHTMM is $53.08\%$ and $80.68\%$ respectively, showing the effectiveness of the approximation introduced. Even if the prediction of the label [math] should be the easiest one (it appears only on leaf nodes), the accuracy obtained by the SP-BHTMM is only around $55\%$ while the accuracy of TF-BHTMM is around $99\%$. The high entropy value suggests that the SP-BTHMM attaches the [math] label also to internal nodes (see Figure 4(a)).

In Figure 5 we report the confusion matrix obtained by the best SP-BHTMM and the best TF-BHTMM. The best TF-BHTMM reaches an accuracy of $99.1\%$, while the best SP-BHTMM reaches only an accuracy of $57.5\%$. This huge difference is due to the SP approximation which mixes together the contributions from child nodes. On the contrary, the TF-BHTMM is able to distinguish the contribution of each child due to the core tensor $\boldsymbol{\lambda}$ which models all the possible joint configurations of the $z$ variables on child nodes.

Moreover, there is consistent difference between the accuracy obtained by the best TF-BHTMM model and the results obtained averaging TF-BHTMM over $5$ runs. This is confirmed by the high standard deviation reported in Table III. We believe that the high variance in the results is due to strong dependence between the first two step of the learning algorithm.

V Conclusion

In this paper, we have introduced a new generative model which defines a probability distribution over tree-structured data. The novelty of our model is the approximation introduced to tackle the well-known computational problems which arise in BU-HTMMs due to the exponential nature of the bottom-up state transition distribution. We have also introduced a new training algorithm which is able to learn the approximation size directly from data, simplifying the model selection step. Finally, we have shown the potential of our model by achieving better results than SP-BHTMM in two different tasks. In particular, the results obtained on the labelling task show clearly the advantage given by the new approximation.

The emphasis of this paper is on the approximation which exhibits nice property and promising results to further develop the model. One of the first improvement concerns the learning algorithm, which exhibits a high variance on the labelling task. Further study should be conducted in order to reduce the dependence between the first two steps of the Gibbs sampler.

Also, the model can be naturally extended in a Bayesian non-parametric fashion, allowing to learn the size of its latent space directly from data. Thanks to the approximation, the size of the latent space does not affect the complexity of the model which is regulated by the approximation size. In this sense, the non-parametric model will be able to separate the label generation dynamics, which depends on the size of the latent space, from the state-transition dynamic, which depends on the approximation size.

Acknowledgment

The work is supported by the Italian Ministry of Education, University, and Research (MIUR) under project SIR 2014 LIST-IT (grant n. RBSI14STDE).