Using Background Knowledge to Rank Itemsets

TL;DR

This paper introduces methods to incorporate various types of background knowledge into the ranking of itemsets using a maximum entropy approach, improving the assessment of pattern interestingness in data mining.

Contribution

It presents efficient techniques for infusing diverse background knowledge into itemset evaluation, extending beyond simple independence models, with polynomial-time solutions.

Findings

More sophisticated models better fit the data.

Using additional background knowledge improves frequency prediction.

Efficient polynomial-time algorithms are developed for maximum entropy models.

Abstract

Assessing the quality of discovered results is an important open problem in data mining. Such assessment is particularly vital when mining itemsets, since commonly many of the discovered patterns can be easily explained by background knowledge. The simplest approach to screen uninteresting patterns is to compare the observed frequency against the independence model. Since the parameters for the independence model are the column margins, we can view such screening as a way of using the column margins as background knowledge. In this paper we study techniques for more flexible approaches for infusing background knowledge. Namely, we show that we can efficiently use additional knowledge such as row margins, lazarus counts, and bounds of ones. We demonstrate that these statistics describe forms of data that occur in practice and have been studied in data mining. To infuse the information efficiently we use a maximum entropy approach. In its general setting, solving a maximum entropy model is infeasible, but we demonstrate that for our setting it can be solved in polynomial time. Experiments show that more sophisticated models fit the data better and that using more information improves the frequency prediction of itemsets.

Tables

Table 1. Table 1: Time and memory complexity of the ComputeProb algorithm for various count statistics. N 𝑁 N is the number of attributes, and K 𝐾 K is the number of distinct values the statistic can assume.

Statistic	memory	time
row margins (from scratch)	$O\left(K\right)$	$O\left(KN\right)$
row margins (backward method)	$O\left(K\right)$	$O\left(N\right)$
lazarus counts	$O\left(K\right)$	$O\left(KN\right)$
joint row margins and lazarus counts	$O\left(N^2\right)$	$O\left(K^{2}N\right)$
joint transaction bounds	$O\left(N^2\right)$	$O\left(N^2\right)$
joint row margins, transaction bounds	$O\left(N^3\right)$	$O\left(K{N}^2\right)$

Table 2. Table 2: The datasets used in our experiments. Shown are the number of transactions | D | 𝐷 |D| and the number of attributes N 𝑁 N , the time required to learn the various models, and the number of iterations until the algorithm convergences.

					Margins			Lazarus			Bounds
Dataset		$\left|D\right|$	$N$		iter	time		iter	time		iter	time
Independent		100000	20		2	0.01 s		2	0.02 s		2	0.02 s
Clusters		100000	20		2	0.01 s		9	0.07 s		10	0.07 s
Markov		100000	20		2	0.01 s		8	0.05 s		12	0.07 s
Chess		3196	75		3	0.6 s		400	153 s		28	8 s
DNA		4590	391		8	313 s		96	90 m		119	66 m
Retail		81998	221		4	26 s		11	110 s		19	171 s
Webview-1		52840	150		3	5 s		14	45 s		93	267 s

Table 3. Table 3: BIC scores (Negative log-likelihood and BIC penalty) of the datasets for different models. The best (smallest) values are indicated in bold.

Dataset	Independent	Margins	Lazarus	Bounds
Independent	$\mathrm{1\hspace{0.17em}658\hspace{0.17em}486}$	$\mathrm{1\hspace{0.17em}658\hspace{0.17em}630}$	$\mathrm{1\hspace{0.17em}658\hspace{0.17em}621}$	$\mathrm{1\hspace{0.17em}658\hspace{0.17em}779}$
Clusters	$\mathrm{2\hspace{0.17em}000\hspace{0.17em}159}$	1 719 959	$\mathrm{1\hspace{0.17em}889\hspace{0.17em}308}$	$\mathrm{1\hspace{0.17em}946\hspace{0.17em}942}$
Markov	$\mathrm{2\hspace{0.17em}000\hspace{0.17em}159}$	$\mathrm{1\hspace{0.17em}938\hspace{0.17em}960}$	1 861 046	$\mathrm{1\hspace{0.17em}890\hspace{0.17em}648}$
Chess	$\mathrm{142\hspace{0.17em}054}$	$\mathrm{132\hspace{0.17em}921}$	131 870	$\mathrm{137\hspace{0.17em}213}$
DNA	$\mathrm{185\hspace{0.17em}498}$	$\mathrm{173\hspace{0.17em}305}$	107 739	$\mathrm{109\hspace{0.17em}572}$
Retail	$\mathrm{1\hspace{0.17em}796\hspace{0.17em}126}$	1 774 291	$\mathrm{1\hspace{0.17em}783\hspace{0.17em}054}$	$\mathrm{1\hspace{0.17em}775\hspace{0.17em}588}$
Webview-1	$\mathrm{836\hspace{0.17em}624}$	$\mathrm{778\hspace{0.17em}361}$	$\mathrm{783\hspace{0.17em}733}$	774 773

Table 4. Table 4: Average absolute and relative error of the top 10 000 10000 10\,000 closed frequent itemsets in the test data. Lower scores are better.

		Independent			Margins
Dataset		absolute	relative		absolute	relative
Independent		0.11% $\pm$ 0.10%	1.54% $\pm$ 1.23%		0.11% $\pm$ 0.10%	1.52% $\pm$ 1.21%
Clusters		9.39% $\pm$ 0.73%	63.08% $\pm$ 11.81%		0.20% $\pm$ 0.11%	1.37% $\pm$ 0.86%
Markov		4.79% $\pm$ 2.29%	47.80% $\pm$ 20.86%		2.19% $\pm$ 1.64%	21.11% $\pm$ 13.41%
Chess		1.81% $\pm$ 1.35%	2.35% $\pm$ 1.80%		1.94% $\pm$ 1.43%	2.52% $\pm$ 1.91%
DNA		0.58% $\pm$ 1.08%	85.89% $\pm$ 30.91%		0.56% $\pm$ 1.04%	84.27% $\pm$ 31.76%
Retail		0.05% $\pm$ 0.11%	48.89% $\pm$ 27.95%		0.04% $\pm$ 0.09%	37.70% $\pm$ 28.64%
Webview-1		0.11% $\pm$ 0.07%	92.61% $\pm$ 17.27%		0.10% $\pm$ 0.06%	79.77% $\pm$ 23.94%
		Lazarus			Bounds
Dataset		absolute	relative		absolute	relative
Independent		0.11% $\pm$ 0.10%	1.54% $\pm$ 1.23%		0.12% $\pm$ 0.10%	1.61% $\pm$ 1.30%
Clusters		7.00% $\pm$ 1.12%	47.04% $\pm$ 10.77%		8.21% $\pm$ 0.93%	55.33% $\pm$ 11.92%
Markov		2.27% $\pm$ 1.61%	22.41% $\pm$ 14.88%		3.36% $\pm$ 2.24%	33.54% $\pm$ 21.12%
Chess		2.06% $\pm$ 1.49%	2.68% $\pm$ 1.99%		1.81% $\pm$ 1.35%	2.35% $\pm$ 1.79%
DNA		0.45% $\pm$ 0.75%	80.24% $\pm$ 72.73%		0.54% $\pm$ 0.94%	82.63% $\pm$ 31.38%
Retail		0.04% $\pm$ 0.10%	42.90% $\pm$ 27.81%		0.04% $\pm$ 0.09%	43.71% $\pm$ 27.27%
Webview-1		0.11% $\pm$ 0.06%	88.35% $\pm$ 20.87%		0.11% $\pm$ 0.06%	90.60% $\pm$ 18.63%

Table 5. Table 5: The average improvement of itemset log-likelihoods over the independence model for the top 10 000 10000 10\,000 closed frequent itemsets. Higher scores are better.

Dataset		Margins	Lazarus	Bounds
Independent		0.03 $\pm$ 0.12	-0.00 $\pm$ 0.09	-0.10 $\pm$ 0.36
Clusters		4517.4 $\pm$ 1168.3	2412.6 $\pm$ 1030.1	1353.2 $\pm$ 660.4
Markov		1585.9 $\pm$ 1310.5	1536.9 $\pm$ 1283.2	931.5 $\pm$ 854.1
Chess		-0.40 $\pm$ 0.52	-0.81 $\pm$ 0.98	0.01 $\pm$ 0.17
DNA		119.9 $\pm$ 235.7	133.1 $\pm$ 274.5	106.2 $\pm$ 224.8
Retail		9.69 $\pm$ 22.49	4.62 $\pm$ 11.05	5.37 $\pm$ 24.94
Webview-1		135.79 $\pm$ 90.32	64.72 $\pm$ 40.41	57.71 $\pm$ 62.95

Equations

$$q_D\left(A=v\right)=\left|\left\{t\in D\mid t=v\right\}\right|/\left|D\right|.$$

$$l\left(t\right)=\left|\left\{t_i=0\mid \text{there exist }a,b\text{ s.t. }a<i<b,t_a=t_b=1\right\}\right|.$$

$$first\left(t\right)=\min \{i\mid t_i=1\}\text{ and }last\left(t\right)=\max \{i\mid t_i=1\}.$$

$$\mathcal{C}=\left\{p\mid p\left(a_i=1\right)=m_i,p\left(S(A)=k\right)=n_k\text{ for }i=1,\dots ,N,k=1,\dots ,K\right\}.$$

$$p^{\ast }=\underset{p}{\mathrm{argmax}}\left\{H\left(p\right)\mid p\in \mathcal{C}\right\}.$$

$$p^{*}\mathopen{}\left(A=t\right)=\left\{\begin{array}[]{ll}u_{0}\prod_{i=1}^{N}u_{i}^{M_{i}(t)}\prod_{k=1}^{K}v_{k}^{T_{k}(t)},&\text{ if }t\notin Z\\ 0,&\text{ if }t\in Z\\ \end{array}\right.$$

$$q\left(A=t\right)=Z_q^{-1}\prod _{i=1}^N{u}_i^{M_i(t)},$$

$$r\left(S(A)=k\right)=Z_r^{-1}{v}_{k}q\left(S(A)=k\right),$$

$$p^{\ast }\left(A=t\right)=\frac{v_k}{Z_r}q\left(A=t\right)=r\left(S(A)=k\right)\frac{q\left(A=t\right)}{q\left(S(A)=k\right)},$$

$$p^{\ast }\left(X=1\right)=\sum _{k=1}^K{p}^{\ast }\left(X=1,S(A)=k\right)=\sum _{k=1}^K\frac{v_k}{Z_r}q(X=1,S(A)=k).$$

$$p(o\left(A_i;K\right)=k)=p_{i}p(o\left(A_{i-1};K\right)=k-1)+(1-p_i)p(o\left(A_{i-1};K\right)=k).$$

$$p(o\left(A-a_i\right)=k)=\frac{1}{1-p_i}\left(p(o\left(A\right)=k)-p_{i}p(o\left(A-a_i\right)=k-1)\right),$$

$$p^{(i)}(k,i)=p_i\sum _{l=0}^k{p}^{(i-1)}(l,i+l-k-1),$$

$$p^{(i)}(0,i)=p_i{p}^{i-1}(0,i-1)+p_i{p}^{(i-1)}(0,0),$$

$$p^{(1)}(0,1)=p(l\left(a_1\right)=0,last\left(a_1\right)=1)=p_1.$$

$$\begin{array}{rl}{p}_i\sum _{l=0}^k{p}^{(i-1)}(l,i+l-k-1)& =p_i{p}^{(i-1)}(k,i-1)+p_i\sum _{l=0}^{k-1}{p}^{(i-1)}(l,i+l-k-1)\\ & =p_i{p}^{(i-1)}(k,i-1)+p_i\frac{1-p_{i-1}}{p_{i-1}}{p}^{(i-1)}(k-1,i-1).\end{array}$$

$$p_i{p}^{(i-1)}(1,i-1)+p_i\frac{1-p_{i-1}}{p_{i-1}}{p}^{(i-1)}(0,i-1)-p_i{p}^{(i-1)}(0,0).$$

$$p(l\left(A\right)=k,last\left(A\right)\le j)=p(l\left(A\right)=k,last\left(A\right)\le j-1)+p^{(j)}(k,j)\prod _{i=j+1}^N(1-p_i),$$

$$p(i,j)=\{\begin{array}{ll}{\prod }_{k=1}^N(1-p_k)& \text{if i=j=0}\\ p_i{\prod }_{k\ne i}(1-p_k)& \text{if i=j=0}\\ {\prod }_{k=1}^{i-1}(1-p_k)p_i{p}_j{\prod }_{k=j+1}^N(1-p_k)& \text{if 0<i<j}\end{array}$$

Full text

In this paper we study techniques for more flexible approaches for infusing background knowledge. Namely, we show that we can efficiently use additional knowledge such as row margins, lazarus counts, and bounds of ones. We demonstrate that these statistics describe forms of data that occur in practice and have been studied in data mining.

To infuse the information efficiently we use a maximum entropy approach. In its general setting, solving a maximum entropy model is infeasible, but we demonstrate that for our setting it can be solved in polynomial time. Experiments show that more sophisticated models fit the data better and that using more information improves the frequency prediction of itemsets.

1 Introduction

Discovering interesting itemsets from binary data is one of the most studied branches in pattern mining. The most common way of defining the interestingness of an itemset is by its frequency, the fraction of transactions in which all items co-occur. This measure has a significant computational advantage since frequent itemsets can be discovered using level-wise or depth-first search strategies [2, 21]. The drawback of frequency is that we cannot infuse any background knowledge into the ranking. For example, if we know that items $a$ and $b$ occur often, then we should expect that $ab$ also occurs relatively often.

Many approaches have been suggested in order to infuse background knowledge into ranking itemsets. The most common approach is to compare the observed frequency against the the independence model. This approach has the advantage that we can easily compute the estimate and that the background knowledge is easily understandable. The downside of the independence model is that it contains relatively little information. For example, if we know that most of the data points contain a small number of ones, then we should infuse this information for ranking patterns. For a more detailed discussion see Section 6.

Assessing the quality of patterns can be seen as a part of the general idea where we are required to test whether a data mining result is statistically significant with respect to some background knowledge (see [10] as an example of such a framework). However, the need for such assessment is especially important in pattern mining due to two major problems: Firstly, the number of discovered patterns is usually large, so a screening/ranking process is needed. Secondly, many of the discovered patterns reflect already known information, so we need to incorporate this information such that we can remove trivial results.

Our goal in this work is to study how we can infuse background knowledge into pattern mining efficiently. We will base our approach on building a statistical model from the given knowledge. We set the following natural goals for our work:

The background knowledge should be simple to understand. 2. 2.

We should be able to infer the model from the data efficiently. 3. 3.

We should be able to compute expected frequencies for itemsets efficiently.

While these goals seem simple, they are in fact quite strict. For example, consider modeling attribute dependencies by using Bayesian networks. First of all, inferring the expected frequency of an itemset from the Bayesian network is done using a message passing algorithm, and is not guaranteed to be computationally feasible [5]. Finally, understanding parent-child relationships in a Bayesian network can be discombobulating.

In this work we will consider the following simple statistics: column margins, row margins, number of zeroes between ones (lazarus counts), and the boundaries of ones. We will use these statistics individually, but also consider different combinations. While these are simple statistics, we will show that they describe many specific dataset structures, such as banded datasets or nested datasets.

We will use these statistics and the maximum entropy principle to build a global model. In a general setting inferring such a model is an infeasible problem. However, we will demonstrate that for our statistics inferring the model can be done in polynomial time. Once this model is discovered we can use it to assess the quality of an itemset by comparing the observed frequency against the frequency expected by the model. The more the observed frequency differs from the expected value, the more significant is the pattern.

We should point out that while our main motivation is to assess itemsets, the discovered background model is a true statistical global model and is useful for other purposes, such as model selection or data generation.

2 Statistics as Background Knowledge

In this section we introduce several count statistics for transactional data. We also indicate for what kinds of datasets they are useful. We begin by presenting preliminary notations that will be used throughout the rest of the paper.

A binary dataset $D$ is a collection of ${\left|D\right|}$ transactions, binary vectors of length $N$. The set of all possible transactions is written as $\Omega=\left\{0,1\right\}^{N}$. The $i$th element of a transaction is represented by an attribute $a_{i}$, a Bernoulli random variable. We denote the collection of all the attributes by $A=\left\{a_{1},\ldots,a_{N}\right\}$. An itemset $X=\left\{x_{1},\ldots,x_{L}\right\}\subseteq A$ is a subset of attributes. We will often use the dense notation $X={{x_{1}}\cdots{x_{L}}}$. Given an itemset $X$ and a binary vector $v$ of length $L$, we use the notation $p\mathopen{}\left(X=v\right)$ to express the probability of $p\mathopen{}\left(x_{1}=v_{1},\ldots,x_{L}=v_{L}\right)$. If $v$ contains only 1s, then we will use the notation $p\mathopen{}\left(X=1\right)$.

Given a binary dataset $D$ we define $q_{D}$ to be an empirical distribution,

[TABLE]

We define the frequency of an itemset $X$ to be $fr\mathopen{}\left(X\right)=q_{D}\mathopen{}\left(X=1\right)$. A statistic is a function ${S}:{\Omega}\to{\mathbb{N}}$ mapping a transaction $t$ to a integer. All our statistics will be of form $q_{D}(S(A)=k)$, that is, our background knowledge will be the fraction of transactions in the data for which $S(t)=k$.

Column margins

The simplest of statistics one can consider are the column margins or item probabilities. These probabilities can be used to define an independence model. This model has been used before in other works to estimate itemset frequencies [4, 1]. It has the advantage of being computationally fast and easy to interpret. However, it is a rather simplistic model and few datasets actually satisfy the independence assumption. We will include the item probabilities in all of our models.

Row margins

We define $o\mathopen{}\left(A\right)$ to be a random variable representing the number of ones in a random transaction. We immediately see that $o\mathopen{}\left(A\right)$ obtains integer values between [math] and $N$. Consequently, $p(o\mathopen{}\left(A\right)=k)$ is the probability of a random transaction having $k$ ones. Given a transaction $t\in\Omega$, we will also denote by $o\mathopen{}\left(t\right)$ the number of ones in $t$, i.e. $o\mathopen{}\left(t\right)=\sum_{i=1}^{N}t_{i}$.

The use of row margins (in conjunction with column margins) has been proposed before by Gionis et al. [9], to asses the significance of (among others) frequent itemsets. However, their approach is different from ours (see Section 6). It was shown that for datasets with very skewed row margin distribution, most frequent itemsets, clusterings, correlations, etc. can be explained entirely by row and column margins alone. Supermarket basket data falls into this category, since the transactions there typically contain only a handful of items.

Lazarus counts

A lazarus event in a transaction is defined as the occurrence of a zero within a string of ones. This requires that a total order is specified on the attributes $a_{i}\in A$. For simplicity, we assume that this order is $a_{1}<\cdots<a_{N}$. Let $t\in\Omega$ be a transaction, then the lazarus count of $t$ is defined as

[TABLE]

Clearly, the lazarus count of a transaction $t$ ranges from [math] to $N-2$. If the lazarus count of $t$ is [math], then $t$ is said to satisfy the consecutive-ones property.

A specific case of datasets with the consecutive-ones property are banded datasets. These are datasets whose rows and columns can be permuted such that the non-zeroes form a staircase pattern. The properties of such banded binary matrices have been studied by Garriga et al., who presented algorithms to determine the minimum number of bit flips it would take for a dataset to be (fully) banded [8]. A banded dataset can be characterized by the following statistics: The major part of the transactions will have low lazarus counts, and typically the row margins are low as well.

Transaction bounds

For certain types of data it can be useful to see at which positions the ones in the transactions of a dataset begin and end. Given a transaction $t\in\Omega$, we define the $\mathit{first}$ and $\mathit{last}$ statistics as

[TABLE]

If $t$ contains only [math]s, then we define $\mathit{first}\mathopen{}\left(t\right)=\mathit{last}\mathopen{}\left(t\right)=0$. A dataset is called nested if for each pair of rows, one is always a subset of the other [14]. For such data, the rows and columns can be permuted such that the rows have consecutive ones starting from the first column. Thus, assuming the permutation is done, transactions in nested datasets will have a low lazarus count and low left bound $\mathit{first}\mathopen{}\left(t\right)$. Nestedness has been studied extensively in the field of ecology for absence/presence data of species (see, for example, [14] for more details).

3 Maximum Entropy Model

The independence model can be seen as the simplest model that we can infer from binary data. This model has many nice computational properties: learning the model is trivial and making queries on this model is a simple and fast procedure. Moreover, a fundamental theorem shows that the independent model is the distribution maximizing the entropy among all the distributions that have the same frequencies for the individual attributes.

Our goal is to define a more flexible model. We require that we should be able to infer such model efficiently and we should be able to make queries. In order to do that we will use the maximum entropy approach.

3.1 Definition of the Maximum Entropy Model

Say that we have computed a set of certain statistics from a dataset and we wish to build a distribution, such that this distribution satisfies the same statistics. The maximum entropy approach gives us the means to do that.

More formally, assume that we are given a function ${S}:{\Omega}\to{\mathbb{N}}$ mapping a transaction $t$ to an integer value. For example, this function can be $o\mathopen{}\left(t\right)$, the number of ones in a transaction. Assume that we are given a dataset $D$ with $N$ attributes and let $q_{D}$ be its empirical distribution. We associate a statistic $n_{k}$ with $S$ for any $k=1,\ldots,K$ to be the proportion of transactions in $D$ for which $S(t)=k$, that is, $n_{k}=q_{D}\mathopen{}\left(S(A)=k\right)$.

In addition to the statistics $\left\{n_{k}\right\}$ we always wish to use the margins of the individual attributes, that is, the probability of an attribute having a value of 1 in a random transaction. We denote these column margins by $m_{i}=q_{D}\mathopen{}\left(a_{i}=1\right)$, where $i=1,\ldots,N$.

The maximum entropy model $p^{*}$ is derived from the statistics $m_{i}$ and $n_{k}$. The distribution $p^{*}$ is the unique distribution maximizing the entropy among the distributions having the statistics $m_{i}$ and $n_{k}$. To be more precise, we define $\mathcal{C}$ to be the set of all distributions having the statistics $m_{i}$ and $n_{k}$,

[TABLE]

The maximum entropy distribution maximizes the entropy in $\mathcal{C}$,

[TABLE]

Note that $\mathcal{C}$ and consequently $p^{*}$ depend on the statistics $m_{i}$ and $n_{k}$, yet we have omitted them from the notation for the sake of clarity.

Our next step is to demonstrate a classic result that the maximum entropy distribution has a specific form. In order to do so we begin by defining indicator functions ${M_{i}}:{\Omega}\to{\left\{0,1\right\}}$ such that $M_{i}(t)=t_{i}$, that is, $M_{i}$ indicates whether the attribute $a_{i}$ has a value of $1$ in a transaction $t$. We also define ${T_{k}}:{\Omega}\to{\left\{0,1\right\}}$ such that $T_{k}(t)=1$ if and only if $S(t)=k$.

Theorem 3.1 (Theorem 3.1 in [6])

The maximum entropy distribution given in Eq. 1 has the form

[TABLE]

for some specific set of parameters $u_{i}$, $v_{k}$ and a set $Z\subseteq\Omega$. Moreover, a transaction $t\in Z$ if and only if $p(A=t)=0$ for all distributions $p$ in $\mathcal{C}$.

It turns out that we can represent our maximum entropy model as a mixture model. Such a representation will be fruitful when we are solving and querying the model. To see this, let $u_{i}$ and $v_{k}$ be the parameters in Eq. 2. We define a distribution $q$ for which we have

[TABLE]

where $Z_{q}$ is a normalization constant so that $q$ is a proper distribution. Since $M_{i}$ depends only on $a_{i}$ we see that $q$ is actually an independence distribution. This allows us to replace the parameters $u_{i}$ with more natural parameters $q_{i}=q(a_{i}=1)$. We should stress that $q_{i}$ is not equal to $m_{i}$.

Our next step is to consider the parameters $v_{k}$. First, we define a distribution

[TABLE]

where $Z_{r}$ is a normalization constant such that $r$ is a proper distribution. We can now express the maximum entropy model $p^{*}$ using $r$ and $q$. By rearranging the terms in Eq. 2 we have

[TABLE]

where $k=S(t)$. The right side of the equation is a mixture model. According to this model, we first sample an integer, say $1\leq k\leq K$, from $r$. Once $k$ is selected we sample the actual transaction from $q\mathopen{}\left(A=t\mid S(A)=k\right)$.

We should point out that we have some redundancy in the definition of $r$. Namely, we can divide each $v_{k}$ by $Z_{r}$ and consequently remove $Z_{r}$ from the equation. However, keeping $Z_{r}$ in the equation proves to be handy later on.

3.2 Using Multiple Statistics

We can generalize our model by allowing multiple statistics together. By combining statistics we can construct more detailed models, which are based on these relatively simple statistics. In our case, building and querying the model remains polynomial in the number of attributes, when combining multiple statistics.

We distinguish two alternatives to combine count statistics. Assume, for the sake of exposition, we have two statistics ${S_{1}}:{\Omega}\to{\mathbb{N}}$ and ${S_{2}}:{\Omega}\to{\mathbb{N}}$, with $S_{1}(t)\leq K_{1}$ and $S_{2}(t)\leq K_{2}$ for all $t\in\Omega$. Then we can either consider using the joint probabilities $q_{D}(S_{1}(A)=k_{1},S_{2}(A)=k_{2})$ for $k_{1}\leq K_{1}$ and $k_{2}\leq K_{2}$; or, we may use the marginal probabilities $q_{D}(S_{1}(A)=k_{1})$ and $q_{D}(S_{2}(A)=k_{2})$ separately. The joint case can be easily reduced to the case of a single statistic by considering $S_{1}+(K_{1}+1)S_{2}$. Solving and querying the model in the marginal case can be done using the same techniques and the same time complexity as with the joint case.

4 Solving the Maximum Entropy Model

In this section we introduce an algorithm for finding the correct distribution. We will use the classic Iterative Scaling algorithm. For more details of this algorithm and the proof of correctness we refer to the original paper [7].

4.1 Iterative Scaling Procedure

The generic Iterative Scaling algorithm is a framework that can be used for discovering the maximum entropy distribution given any set of linear constraints. The idea is to search the parameters and update them in an iterative fashion so that the statistics we wish to constrain will converge towards the desired values. In our case, we update $u_{i}$ and $v_{k}$ iteratively so that the statistics of the distribution will converge into $m_{i}$ and $n_{k}$. The sketch of the algorithm is given in Algorithm 1.

In order to use the Iterative Scaling algorithm we need techniques for computing the probabilities $p^{*}\mathopen{}\left(a_{i}=1\right)$ and $p^{*}\mathopen{}\left(S(A)=k\right)$ (lines 3 and 7). The former is a special case of computing the frequency of an itemset $p^{*}\mathopen{}\left(X=1\right)$ and is detailed in the following subsection. For the latter, assume that we have an algorithm, say ComputeProb, which given a set of probabilities $p_{1},\ldots,p_{N}$ will return the probabilities $p\mathopen{}\left(S(A)=k\right)$ for each $k=1,\ldots,K$, where $p$ is the independence model parameterized by $p_{i}$, that is, $p(a_{i}=1)=p_{i}$. We will now show that using only ComputeProb we are able the obtain the required probabilities.

Note that the first equality in Eq. 5 implies that we can compute $p^{*}\mathopen{}\left(S(A)=k\right)$ from $q(S(A)=k)$. To compute the latter we call ComputeProb with $p_{i}=q_{i}$ (for $i=1,\ldots,N$) as parameters.

4.2 Querying the Model

We noted before that computing $p^{*}\mathopen{}\left(a_{i}=1\right)$ is a special case of computing the frequency of an itemset $p^{*}\mathopen{}\left(X=1\right)$. In order to do that let us write

[TABLE]

Let us write $y=\prod_{a_{j}\in X}q_{j}$. Note that since $q$ is the independence model, we have $q(S(A)=k,X=1)=yq(S(A)=k\mid X=1)$. To compute $q(S(A)=k\mid X=1)$ we call ComputeProb with parameters $p_{j}=q_{j}$ if $a_{j}\notin X$ and $p_{j}=1$ if $a_{j}\in X$.

4.3 Computing Statistics

In order to use the Iterative Scaling algorithm we need an implementation of ComputeProb, a routine that returns the probabilities of statistic $S$ with respect to the independence model. Since ComputeProb is called multiple times, the runtime consumption is pivotal. Naturally, memory and time requirements depend on the statistic at hand. In Table 1 the complexities of ComputeProb are listed for several statistics and joint statistics. All statistics are computed using the same general dynamic programming idea: To compute the statistics for items $\left\{a_{1},\ldots,a_{i}\right\}$ we first solve the problem for items $\left\{a_{1},\ldots,a_{i-1}\right\}$ and using that result we will be able to compute efficiently the statistics for $\left\{a_{1},\ldots,a_{i}\right\}$. The details of these calculations are given in the appendix.

5 Experiments

In this section we present the results of experiments on synthetic and real data. The source code of the algorithm can be found at the authors’ website111http://www.adrem.ua.ac.be/implementations/.

5.1 Datasets

The characteristics of the datasets we used are given in Table 2.

We created three synthetic datasets. The first one has independent items with randomly chosen frequencies. The second dataset contains two clusters of equal size. In each cluster the items are independent with a frequency of 25% and 75% respectively. Hence, the row margin distribution has two peaks. In the third synthetic dataset, the items form a Markov chain. The first item has a frequency of 50%, and then each subsequent item is a noisy copy of the preceding one, that is, the item is inverted with a 25% probability.

The real-world datasets we used are obtained from the FIMI Repository222 http://fimi.cs.helsinki.fi/data/ and [16]. The Chess data contains chess board descriptions. The DNA dataset [16] describes DNA copy number amplifications. The Retail dataset contains market basket data from an anonymous Belgian supermarket [3] and the Webview-1 dataset contains clickstream data from an e-commerce website [13]. For both datasets the rare items with a frequency lower than 0.5% were removed. This resulted in some empty transactions which were subsequently deleted.

Note that for all of the models, except the margin model, an order is needed on the items. In this paper we resort to using the order in which the items appear in the data. Despite the fact that this order is not necessarily optimal, we were able to improve over the independence model.

5.2 Model Performance

We begin by examining the log-likelihood of the datasets, given in Table 3. We train the model on the whole data and then compute its likelihood, giving it a BIC penalty [18], equal to $\frac{k}{2}\log|D|$ where $k$ is the number of free parameters of each model, and $|D|$ is the number of transactions. This penalty rewards models with few free parameters, while penalizing complex ones.

Compared to the independence model, the likelihoods are all better on all datasets with the exception of the Independent data. This is expected since the Independent data is generated from an independence distribution so using more advanced statistics will not improve the BIC score.

When looking at the other two synthetic datasets, we see that the margin model has the highest likelihood for the Clusters data, and the lazarus model for the Markov data. The Clusters data has two clusters, in both of which the items are independent. The distribution of the row margins has two peaks, one for each cluster. This information cannot be explained by the independence model and adding this information improves the log-likelihood dramatically. The items in the Markov dataset are ordered since they form a Markov chain. More specifically, since each item is a (noisy) copy of the previous one, we expect the transactions to consist of only a few blocks of consecutive ones, which implies that their lazarus count is quite low, and hence the lazarus model performs well.

Chess is originally a categorical dataset, which has been binarized to contain one item for each attribute-value pair. Hence, it is a rather dense dataset, with constant row margins, and lazarus counts and bounds centered around a peak. That is, Chess does not obey the independence model and this is seen that the likelihoods of our models are better than that of the independence model.

The likelihood of the lazarus model for the DNA dataset, which is very close to being fully banded [8], is significantly lower than that of the independence model and the margin model, which indicates that using lazarus counts are a good idea in this case. The bounds model comes in second, also performing very well, which can again be explained by the bandedness of the data.

Finally, Retail and Webview-1 are sparse datasets. The margin model performs well for both datasets. Therefore we can conclude that a lot of the structure of these datasets is captured in the row and column margins. Interestingly, although close to the margin model, the bounds model is best for Webview-1.

5.3 Frequency Estimation of Itemsets

Next, we perform experiments on estimating the supports of a collection of itemsets. The datasets are split in two parts, a training set and a test set. We train the models on the training data, and use the test data to verify frequency estimates of itemsets. We estimate the frequencies of the top $10\,000$ closed frequent itemsets in the test data (or all closed itemsets if there are less).

Table 4 reports the average absolute and relative errors of the frequency estimates, for the independence, margins, lazarus and bounds models. For the Independent data, the independence model performs best, since the other models overlearn the data. For all other datasets, except Chess, we see that using more information reduces both the average absolute and relative error of the frequency estimates. For instance, for Clusters the average absolute error is reduced from $9.39\%$ to $0.2\%$ using row and column margins, and likewise for Markov the average relative error is reduced from $47.8\%$ to $21.11\%$. The DNA, Retail and Webview-1 data are sparse. Therefore, the itemset frequencies are very low, as well as the absolute errors, even for the independence model. However, the relative errors are still quite high. In this case our models also outperform the independence model. For example, the relative error is reduced from $92.61\%$ to $79.77\%$ by using the margins model on the Webview-1 dataset. For DNA, the average relative error drops $5\%$ using lazarus counts.

The only exception is the Chess dataset where the average absolute and relative errors do not improve over the independence model. Note, however, that this dataset is originally categorical, and contains a lot of dependencies between the items. Interestingly enough, our models perform better than the independence model in terms of (penalized) likelihood. This suggests that in this case using additional information makes some itemsets interesting.

Our final experiment (given in Table 5) is the average improvement of log-likelihood of each itemset compared to the independence model: Let $f$ be the empirical frequency of itemset $X$ from the test data, and let $p$ be the estimate given by one of the models, then the log-likelihood of $X$ is computed as $|D|(f\log p+(1-f)\log(1-p))$. We compute the difference between the log-likelihoods for the independence model and the other models, and take the average over the top $10\,000$ closed frequent itemsets. Again, for Independent and Chess, the independence model performs the best. For all the other datasets, we clearly see an improvement with respect to the independence model. For Clusters and Markov, the average log-likelihood increases greatly, and is the highest for the margin model. For DNA, Retail and Webview-1, the increase in likelihood is somewhat lower. The reason for this is that both the estimates and observed frequencies are small and close to each other. For DNA the average increase is highest when using lazarus counts, while for Retail and Webview-1 the margin model is best.

6 Related Work

A special case of our framework greatly resembles the work done in [9]. In that work the authors propose a procedure for assessing the results of a data mining algorithm by sampling the datasets having the same margins for the rows and columns as the original data. While the goals are similar, there is a fundamental difference between the two frameworks. The key difference is that we do not differentiate individual rows. Thus we do not know that, for example, the first row in the data has $5$ ones but instead we know how many rows have $5$ ones. The same key difference can be seen between our method and Rasch models where each individual row and column of the dataset is given its own parameter [17].

Our approach and the approach given in [9] complement each other. When the results that we wish to assess do not depend on the order or identity of transactions, it is more appropriate to use our method. An example of such data mining algorithms is frequent itemset mining. On the other hand, if the data is to be treated as a collection of transactions, e.g. for segmentation, then we should use the approach in [9]. Also, our approach has a more theoretically sound ground since for sampling datasets the authors in [9] rely on MCMC techniques with no theoretical guarantees that the actual mixing has happened.

Comparing frequencies of itemsets to estimates has been studied in several works. The most common approach is to compare the itemset against the independence model [4, 1]. A more flexible approach has been suggested in [12, 11] where the itemset is compared against a Bayesian network. In addition, approaches where the maximum entropy models derived from some given known itemsets are suggested in [15, 20]. A common problem for these more general approaches is that the deriving of probabilities from these models is usually too complex. Hence, we need to resort to either estimating the expected value by sampling, or build a local model using only the attributes occurring in the query. In the latter case, it is shown in [19] that using only local information can distort the frequency estimate and that the received frequencies are not consistent with each other. Our model does not suffer from these problems since it is a global model from which the frequency estimates can be drawn efficiently.

7 Conclusions

In this paper we considered using count statistics to predict itemset frequencies. To this end we built a maximum entropy model from which we draw estimates for the frequency of itemsets and compare the observed value against the estimate. We introduced efficient techniques for solving and querying the model. Our experiments show that using these additional statistics improves the model in terms of likelihood and in terms of predicting itemset frequencies.

Appendix 0.A Computing Statistics

To simplify notation, we define $A_{i}=\left\{a_{1},\ldots,a_{i}\right\}$ to be the set of the first $i$ items.

Row Margins

Our first statistic is the number of ones in a transaction $o\mathopen{}\left(t\right)$.

We consider a cut version of $o\mathopen{}\left(t\right)$ by defining $o\mathopen{}\left(t;K\right)=\min\mathopen{}\left(o\mathopen{}\left(t\right),K\right)$. Hence, $p(o\mathopen{}\left(A;K\right)=k)$ is the probability of a random transaction having $k$ ones, if $k<K$. On the other hand $p(o\mathopen{}\left(A;K\right)=K)=p(o\mathopen{}\left(A\right)\geq K)$ is the probability of a random transaction having at least $K$ ones. We can exploit the fact that $p(o\mathopen{}\left(A\right)\geq K)=1-\sum_{k=0}^{K-1}p(o\mathopen{}\left(A\right)=k)$ to reduce computation time. The range of $o\mathopen{}\left(t\right)$ is $0,\ldots,N$, whereas the range of $o\mathopen{}\left(A;K\right)$ is $0,\ldots,K$.

Our next step is to compute the probabilities $p(o\mathopen{}\left(A;K\right)=k)$ when $p$ is the independence model. Let us write $p_{i}=p(a_{i}=1)$, the probability of $a_{i}$ attaining the value of 1. We will first introduce a way of computing the probabilities from scratch. In order to do that, note the following identity

[TABLE]

This identity holds since $p$ is the independence model. Hence, to compute $p(o\mathopen{}\left(A;K\right)=k)$ we start with $p(o\mathopen{}\left(a_{1};K\right)=k)$, add $a_{2}$, $a_{3}$, and so on, until we have processed all variables and reached $A$. Note that we are computing simultaneously the probabilities for all $k$. We can perform the computation in $O(NK)$ steps and $O(K)$ memory slots.

We can improve this further by analyzing the flow in the Iterative Scaling Algorithm. The algorithm calls ComputeProb either using parameters $p_{1}=q_{1},\ldots,p_{N}=q_{N}$ or $p_{1}=q_{1},\ldots,p_{i}=1,\ldots,p_{N}=q_{N}$.

Assume that we have $q(o\mathopen{}\left(A\right)=k)$ from the previous computation. These probabilities will change only when we are updating $q_{i}$. To achieve more efficient computation we will first compute $q(o\mathopen{}\left(A\right)=k\mid a_{i}=1)$ from $q(o\mathopen{}\left(A\right)=k)$, update $q_{i}$ and then update $q(o\mathopen{}\left(A\right)=k)$. To do that we can reverse the identity given in Eq. 6 into

[TABLE]

if $k=1,\ldots,N$. When $k=0$ we have $p(o\mathopen{}\left(A-a_{i}\right)=0)=p(o\mathopen{}\left(A\right)=0)/(1-p_{i})$. Using these identities we can take a step back and remove $a_{i}$ from $A$. To compute $q(o\mathopen{}\left(A\right)=k\mid a_{i}=1)$ we can apply the identity in Eq. 6 with $p_{i}=1$. Once $q_{i}$ is updated we can again use Eq. 6 to update $q(o\mathopen{}\left(A\right)=k)$. All these computations can be done in $O(N)$ time. On the other hand, in order to use Eq. 7 we must remember the probabilities $q(o\mathopen{}\left(A\right)=k)$ for all $k$, so consequently our memory consumption rises from $O(K)$ to $O(N)$.

Lazarus Events

Our aim is to efficiently compute the probabilities $p(l\mathopen{}\left(A\right)=k)$ where $p$ is an independence distribution. Again, let us write $p_{i}=p(a_{i}=1)$. The desired probabilities are computed incrementally in $N$ steps, starting from $p(l\mathopen{}\left(a_{1}\right)=k)$, up to $p(l\mathopen{}\left(A\right)=k)$. In order to determine the Lazarus count probabilities, we use an auxiliary statistic, the last occurrence $\mathit{last}\mathopen{}\left(t\right)$.

We will compute the probabilities $p(l\mathopen{}\left(A\right)=k,\mathit{last}\mathopen{}\left(A\right)=j)$ and then marginalize them to obtain $p(l\mathopen{}\left(A\right)=k)$. To simplify the following notation, we define the probability $p^{(i)}(k,j)=p(l\mathopen{}\left(A_{i}\right)=k,\mathit{last}\mathopen{}\left(A_{i}\right)=j)$.

First, assume that $\mathit{last}\mathopen{}\left(A_{i}\right)=i$, which implies that $a_{i}=1$. In such case the Lazarus count increases by $i-\mathit{last}\mathopen{}\left(A_{i-1}\right)-1$. We can write the following identity

[TABLE]

for $k=1,\ldots,i-2$. To handle the boundary cases, for $k=0$ and $i>1$ we have

[TABLE]

and finally when $k=0$ and $i=1$ we obtain

[TABLE]

Secondly, consider the instances where $\mathit{last}\mathopen{}\left(A_{i}\right)<i$, in which case $a_{i}$ must be equal to [math]. We obtain $p^{(i)}(k,j)=(1-p_{i})p^{(i-1)}(k,j)$ for all $k=0,\ldots,i-2$ and $j=0,\ldots,i-1$.

Using the equations above, we are able to compute $p(l\mathopen{}\left(A\right)=k)$ using $O(N^{2})$ memory slots. Specifically, for $N(N-1)/2+2$ out of all $N^{2}-1$ combinations of $j$ and $k$, is $p(l\mathopen{}\left(A\right)=k,\mathit{last}\mathopen{}\left(A\right)=j)$ potentially nonzero. Since in each step a quadratic number of probabilities is updated, $O(N^{3})$ time is needed. However, we can reduce this to quadratic time and linear space. First of all, note that the righthand size of Equation 8, being a sum of size linear in $k$, can be rewritten as a sum of constant size. Moreover, this sum only uses probabilities involving $\{a_{1},\ldots,a_{i-1}\}$ having $a_{i}=1$, which were computed in the previous step. Hence for $k>1$, the probability $p^{(i)}(k,i)$ is equal to

[TABLE]

For $k=1$, $p^{(i)}(1,i)=p_{i}p^{(i-1)}(1,i-1)+p_{i}p^{(i-1)}(0,i-2))$ is equal to

[TABLE]

Finally, noting that

[TABLE]

we can compute $p(l\mathopen{}\left(A\right)=k)$ by gradually summing the second terms. We can compute $p^{(j)}(k,j)$ in constant time using only $p^{(j-1)}(k,j-1)$ and $p^{(j-1)}(k,j-1)$. Hence we can discard the terms $p^{(n)}(k,n)$, where $n<j-1$. Therefore, only $O(N)$ memory and $O(N^{2})$ time is needed.

Joint Transaction Bounds

We need to compute $p(\mathit{first}\mathopen{}\left(A\right)=i,\mathit{last}\mathopen{}\left(A\right)=j)$ for an independence distribution $p$, with $i,j=0,\ldots,N$. For the sake of brevity, let us denote $p(i,j)=p(\mathit{first}\mathopen{}\left(A\right)=i,\mathit{last}\mathopen{}\left(A\right)=j)$. Note that $p(i,j)$ is nonzero if $i=j=0$ or $0<i\leq j$. Hence there are $(N^{2}+N)/2+1$ probabilities to compute. We distinguish three cases

[TABLE]

We can construct the $p(i,j)$ in quadratic time, by looping over $i$ and $j$, and maintaining the products $\prod_{k=1}^{i}(1-p_{k})$ and $\prod_{k=j}^{N}(1-p_{k})$, which can be updated in constant time in each iteration. Using these products and the individual item probabilities, we can construct $p(i,j)$.