Learning what matters - Sampling interesting patterns

TL;DR

This paper introduces LetSIP, a novel interactive pattern sampling algorithm that learns user interests through feedback, enabling efficient, personalized data exploration with improved quality and diversity of discovered patterns.

Contribution

It presents a new sampling approach combining weighted sampling and learning to rank, allowing direct adaptation to user interests in pattern mining.

Findings

LetSIP outperforms state-of-the-art methods in quality-diversity trade-offs.

The system enables efficient, user-specific, anytime data exploration.

It effectively learns user preferences through feedback during pattern sampling.

Abstract

In the field of exploratory data mining, local structure in data can be described by patterns and discovered by mining algorithms. Although many solutions have been proposed to address the redundancy problems in pattern mining, most of them either provide succinct pattern sets or take the interests of the user into account-but not both. Consequently, the analyst has to invest substantial effort in identifying those patterns that are relevant to her specific interests and goals. To address this problem, we propose a novel approach that combines pattern sampling with interactive data mining. In particular, we introduce the LetSIP algorithm, which builds upon recent advances in 1) weighted sampling in SAT and 2) learning to rank in interactive pattern mining. Specifically, it exploits user feedback to directly learn the parameters of the sampling distribution that represents the user's interests. We compare the performance of the proposed algorithm to the state-of-the-art in interactive pattern mining by emulating the interests of a user. The resulting system allows efficient and interleaved learning and sampling, thus user-specific anytime data exploration. Finally, LetSIP demonstrates favourable trade-offs concerning both quality-diversity and exploitation-exploration when compared to existing methods.

Tables

Table 1. Table 1: Dataset properties.

	$\left|ℐ\right|$	$\left|𝒟\right|$	$\theta$	Frequent
				patterns
anneal	$93$	$812$	$660$	$\mathrm{149\hspace{0.17em}331}$
australian	$125$	$653$	$300$	$\mathrm{141\hspace{0.17em}551}$
german	$112$	$1000$	$300$	$\mathrm{161\hspace{0.17em}858}$
heart	$95$	$296$	$115$	$\mathrm{153\hspace{0.17em}214}$
hepatitis	$68$	$137$	$48$	$\mathrm{148\hspace{0.17em}289}$
lymph	$68$	$148$	$48$	$\mathrm{146\hspace{0.17em}969}$
primary	$31$	$336$	$16$	$\mathrm{162\hspace{0.17em}296}$
soybean	$50$	$630$	$28$	$\mathrm{143\hspace{0.17em}519}$
vote	$48$	$435$	$25$	$\mathrm{142\hspace{0.17em}095}$
zoo	$36$	$101$	$10$	$\mathrm{151\hspace{0.17em}806}$

Table 2. Table 2: Effect of LetSIP ’s parameters on regret w.r.t. three performance measures. Results are aggregated over datasets, quality measures, and other parameters.

		Regret: avg.$\phi$	Regret: max.$\phi$	Regret: $H_J$
Query size $k$	5	$6.35\pm 1.04$	$1.13\pm 0.52$	$13.28\pm 0.89$
	10	$5.91\pm 0.59$	$0.47\pm 0.18$	$17.44\pm 0.45$
All results below are for query size of $k=5$
Features	I	$8.17\pm 0.96$	$1.35\pm 0.56$	$13.64\pm 0.90$
	ILF	$6.30\pm 1.36$	$1.16\pm 0.59$	$13.15\pm 0.96$
	ILFT	$4.60\pm 0.78$	$0.87\pm 0.40$	$13.06\pm 0.81$
Range $A$	0.5	$6.43\pm 1.06$	$1.15\pm 0.52$	$13.20\pm 0.86$
	0.1	$6.26\pm 1.01$	$1.11\pm 0.51$	$13.36\pm 0.91$
Query retention $l$	0	$8.19\pm 1.21$	$2.53\pm 0.72$	$13.38\pm 0.69$
	1	$6.78\pm 0.99$	$0.53\pm 0.34$	$13.06\pm 0.72$
	2	$5.61\pm 0.94$	$0.61\pm 0.42$	$13.56\pm 1.05$
	3	$4.80\pm 1.00$	$0.80\pm 0.57$	$13.33\pm 1.22$
Cell sampling $\varsigma$	Random	$10.60\pm 0.71$	$1.89\pm 0.64$	$12.15\pm 0.59$
	Top(1)	$5.14\pm 1.13$	$0.81\pm 0.45$	$13.70\pm 1.00$
	Top(2)	$5.45\pm 1.06$	$0.87\pm 0.47$	$13.60\pm 0.98$
	Top(3)	$5.95\pm 1.20$	$0.95\pm 0.50$	$13.57\pm 0.96$

Table 3. Table 3: LetSIP has considerably lower regrets than alternatives w.r.t. quality and, for samplers, diversity as quantified by joint entropy. (For φ = s ​ u ​ r ​ p 𝜑 𝑠 𝑢 𝑟 𝑝 \varphi=surp (marked by *), IPM fails for 7 out of 10 datasets due to double overflow of multiplicative weights.)

	Regret: avg.$\phi$				Regret: joint entropy $H_J$
	$freq$	${\chi }^2$	$surp$		$freq$	${\chi }^2$	$surp$
LetSIP	$2.4\pm 0.5$	$2.4\pm 0.1$	$4.5\pm 1.4$*		$11.7\pm 0.6$	$11.7\pm 0.5$	$15.9\pm 1.1$*
IPM	$15.5\pm 1.8$	$12.8\pm 2.3$	$15.5\pm 1.8$*		$15.7\pm 1.9$	$15.4\pm 1.9$	$19.8\pm 2.1$*
APLe	$0.0\pm 0.0$	$4.5\pm 3.8$	$5.3\pm 3.9$*		–	–	–*

Equations

$${\phi }_{\text{logistic}}\left(p;w,A\right)=A+\frac{1-A}{1+e^{-\vec{w}\cdot \vec{p}}}$$

$${\chi }^2\left(p\right)=$$

Full text

To address this problem, we propose a novel approach that combines pattern sampling with interactive data mining. In particular, we introduce the LetSIP algorithm, which builds upon recent advances in 1) weighted sampling in SAT and 2) learning to rank in interactive pattern mining. Specifically, it exploits user feedback to directly learn the parameters of the sampling distribution that represents the user’s interests.

We compare the performance of the proposed algorithm to the state-of-the-art in interactive pattern mining by emulating the interests of a user. The resulting system allows efficient and interleaved learning and sampling, thus user-specific anytime data exploration. Finally, LetSIP demonstrates favourable trade-offs concerning both quality–diversity and exploitation–exploration when compared to existing methods.

1 Introduction

†† This document is an extended version of a conference publication [14].

Imagine a data analyst who has access to a medical database containing information about patients, diagnoses, and treatments. Her goal is to identify novel connections between patient characteristics and treatment effects. For example, one treatment may be more effective than another for patients of a certain age and occupation, even though the latter is more effective at large. Here, age and occupation are latent factors that explain the difference in treatment effect.

In the field of exploratory data mining, such hypotheses are represented by patterns [1] and discovered by mining algorithms. Informally, a pattern is a statement in a formal language that concisely describes the structure of a subset of the data. Unfortunately, in any realistic database the interesting and/or relevant patterns tend to get lost among a humongous number of patterns.

The solutions that have been proposed to address this so-called pattern explosion, caused by enumerating all patterns satisfying given constraints, can be roughly clustered into four categories: 1) condensed representations [10], 2) pattern set mining [9], 3) pattern sampling [5], 4) and—most recently—interactive pattern mining [20]. As expected, each of these categories has its own strengths and weaknesses and there is no ultimate solution as of yet.

That is, condensed representations, e.g., closed itemsets, can be lossless but usually still yield large result sets; pattern set mining and pattern sampling can provide succinct pattern sets but do not take the analyst into account; and existing interactive approaches take the user into account but do not adequately address the pattern explosion. Consequently, the analyst has to invest substantial effort in identifying those patterns that are relevant to her specific interests and goals, which often requires extensive data mining expertise.

Aims and contributions Our overarching aim is to enable analysts—such as the one described in the medical scenario above—to discover small sets of patterns from data that they consider interesting. This translates to the following three specific requirements. First, we require our approach to yield concise and diverse result sets, effectively avoiding the pattern explosion. Second, our method should take the user’s interests into account and ensure that the results are relevant. Third, it should achieve this with limited effort on behalf of the user.

To satisfy these requirements, we propose an approach that combines pattern sampling with interactive data mining techniques. In particular, we introduce the LetSIP algorithm, for Learn to Sample Interesting Patterns, which follows the Mine, Interact, Learn, Repeat framework [13]. It samples a small set of patterns, receives feedback from the user, exploits the feedback to learn new parameters for the sampling distribution, and repeats these steps. As a result, the user may utilize a compact diverse set of interesting patterns at any moment, blurring the boundaries between learning and discovery modes.

We satisfy the first requirement by using a sampling technique that samples high quality patterns with high probability. While sampling does not guarantee diversity per se, we demonstrate that it gives concise yet diverse results in practice. Moreover, sampling has the advantage that it is anytime, i.e., the result set can grow by user’s request. LetSIP’s sampling component is based on recent advances in sampling in SAT [12] and their extension to pattern sampling [15].

The second requirement is satisfied by learning what matters to the user, i.e., by interactively learning the distribution patterns are sampled from. This allows the user to steer the sampler towards subjectively interesting regions. We build upon recent work [13, 7] that uses preference learning to learn to rank patterns.

Although user effort can partially be quantified by the total amount of input that needs to be given during the analysis, the third requirement also concerns the time that is needed to find the first interesting results. For this it is of particular interest to study the trade-off between exploitation and exploration. As mentioned, one of the benefits of interactive pattern sampling is that the boundaries between learning and discovery are blurred, meaning that the system keeps learning while it continuously aims to discover potentially interesting patterns.

We evaluate the performance of the proposed algorithm and compare it to the state-of-the-art in interactive pattern mining by emulating the interests of a user. The results confirm that the proposed algorithm has the capacity to learn what matters based on little feedback from the user. More importantly, the LetSIP algorithm demonstrates favourable trade-offs concerning both quality–diversity and exploitation–exploration when compared to existing methods.

2 Interactive pattern mining: Problem definition

Recall the medical analyst example. We assume that after inspecting patterns, she can judge their interestingness, e.g., by comparing two patterns. Then the primary task of interactive pattern mining consists in learning a formal model of her interests. The second task involves using this model to mine novel patterns that are subjectively interesting to the user (according to the learned model).

Formally, let $\mathcal{D}$ denote a dataset, $\mathcal{L}$ a pattern language, $\mathcal{C}$ a (possibly empty) set of constraints on patterns, and $\succ$ the unknown subjective pattern preference relation of the current user over $\mathcal{L}$, i.e., $p_{1}\succ p_{2}$ implies that the user considers pattern $p_{1}$ subjectively more interesting than pattern $p_{2}$:

Problem 1 (Learning)

Given $\mathcal{D}$, $\mathcal{L}$, and $\mathcal{C}$, dynamically collect feedback $U$ with respect to patterns in $\mathcal{L}$ and use $U$ to learn a (subjective) pattern interestingness function $h:\mathcal{L}\rightarrow\mathbb{R}$ such that $h\left(p_{1}\right)>h\left(p_{1}\right)\Leftrightarrow p_{1}\succ p_{2}$.

The mining task should account for the potential diversity of user’s interests. For example, the analyst may (unwittingly) be interested in several unrelated treatments with disparate latent factors. An algorithm should be able to identify and mine patterns that are representative of these diverse hypotheses.

Problem 2 (Mining)

Given $\mathcal{D}$, $\mathcal{L}$, $\mathcal{C}$, and $h$, mine a set of patterns $\mathcal{P}_{h}$ that maximizes a combination of interestingness $h$ and diversity of patterns.

The interestingness of $\mathcal{P}$ can be quantified by the average quality of its members, i.e., $\left.\sum_{p\in\mathcal{P}}{h\left(p\right)}\mid/{\left|\mathcal{P}\right|}\right.$. Diversity measures quantify how different patterns in a set are from each other. Joint entropy is a common diversity measure [24] (see Section 4 for the definition).

3 Related work

In this paper, we focus on two classes of related work aimed at alleviating the pattern explosion, namely

1. pattern sampling and
2. interactive pattern mining.

Pattern sampling. First pattern samplers are based on Markov Chain Monte Carlo (MCMC) random walks over the pattern lattice [5, 17, 4]. Their main advantage is that they support “black box” distributions, i.e., they do not require any prior knowledge about the target distribution, a property essential for interactive exploration. However, they often converge only slowly to the desired target distribution and require the selection of the “right” proposal distributions.

Samplers that are based on alternative approaches include direct two-step samplers and XOR samplers. Two-step samplers [6, 8], while provably accurate and efficient, only support a limited number of distributions and thus cannot be easily extended to interactive settings. Flexics [15] is a recently proposed pattern sampler based on the latest advances in weighted constrained sampling in SAT [12]. It supports black-box target distributions, provides guarantees with respect to sampling accuracy and efficiency, and has been shown to be competitive with the state-of-the-art methods described above.

Interactive pattern mining. Most recent approaches to interactive pattern mining are based on learning to rank patterns. They first appeared in Xin et al. [25] and Rueping [22] and were independently extended by Boley et al. [7] and Dzyuba et al. [13]. The central idea behind these algorithms is to alternate between mining and learning. Priime [3] focuses on advanced feature construction for interactive mining of structured data, e.g., sequences or graphs.

To the best of our knowledge, IPM [2] is the only existing approach to interactive itemset sampling. It uses binary feedback (“likes” and “dislikes”) to update weights of individual items. Itemsets are sampled proportional to the product of weights of constituent items. Thus, the model of user interests in IPM is fairly restricted; moreover, it potentially suffers from convergence issues typical for MCMC. We empirically compare LetSIP with IPM in Section 6.

4 Preliminaries

Pattern mining and sampling. We focus on itemset mining, i.e., pattern mining for binary data. Let $\mathcal{I}=\left\{1\ldots M\right\}$ denote a set of items. Then, a dataset $\mathcal{D}$ is a bag of transactions over $\mathcal{I}$, where each transaction $t$ is a subset of $\mathcal{I}$, i.e., $t\subseteq\mathcal{I}$; $\mathcal{T}=\left\{1\ldots N\right\}$ is a set of transaction indices. The pattern language $\mathcal{L}$ also consists of sets of items, i.e., $\mathcal{L}=2^{\mathcal{I}}$. An itemset $p$ occurs in a transaction $t$, iff $p\subseteq t$. The frequency of $p$ is the proportion of transactions in which it occurs, i.e., $freq\left(p\right)={\left|\left\{t\in\mathcal{D}\mid p\subseteq t\right\}\right|}/N$. In labeled datasets, each transaction $t$ has a label from $\left\{-,+\right\}$; $freq^{-,+}$ are defined accordingly.

Given an (arbitrarily ordered) pattern set $\mathcal{P}$ of size $k$, its diversity can be measured using joint entropy $H_{J}$, which essentially quantifies the overlap of sets of transactions, in which the patterns in $\mathcal{P}$ occur. Let $\left[\cdot\right]$ denote the Iverson bracket, $b^{\prime}\in\left\{0,1\right\}^{k}$ a binary $k$-tuple, and $P\left(b^{\prime}\right)=\dfrac{1}{{\left|\mathcal{D}\right|}}\ \sum\limits_{t\in\mathcal{D}}{\prod\limits_{i\in\left[1,\ k\right]}{\left[b_{i}^{\prime}=1\Leftrightarrow\mathcal{P}_{i}\subseteq t\right]}}$ the fraction of transactions in $\mathcal{D}$ covered only by patterns in $\mathcal{P}$ that correspond to non-zero elements of $b^{\prime}$ (e.g., if $k=3$ and $b^{\prime}=101$, we only count the transactions covered by the 1st and the 3rd pattern and not covered by the 2nd pattern). Joint entropy $H_{J}$ is defined as $H_{J}\left(\mathcal{P}\right)=-\sum\limits_{b\in\left\{0,1\right\}^{k}}{P\left(b\right)\times\log{P\left(b\right)}}$. $H_{J}$ is measured in bits and bounded from above by $k$. The higher the joint entropy, the more diverse are the patterns in $\mathcal{P}$ in terms of their occurrences in $\mathcal{D}$.

The choice of constraints and a quality measure allows a user to express her analysis requirements. The most common constraint is minimal frequency $freq\left(p\right)\geq\theta$. In contrast to hard constraints, quality measures are used to describe soft preferences that allow to rank patterns; see Section 6 for examples.

While common mining algorithms return the top-$k$ patterns w.r.t. a measure $\varphi:\mathcal{L}\rightarrow\mathbb{R}^{+}$, pattern sampling is a randomized procedure that ‘mines’ a pattern with probability proportional to its quality, i.e., $P_{\varphi}\left(p\text{\small\ is sampled}\right)=\left.\varphi\left(p\right)\middle/Z_{\varphi}\right.$, if $p\in\mathcal{L}$ satisfies $\mathcal{C}$, and [math] otherwise, where $Z_{\varphi}$ is the (unknown) normalization constant. This is an instance of weighted constrained sampling.

Weighted constrained sampling. This problem has been extensively studied in the context of sampling solutions of a SAT problem [21]. WeightGen [12] is a recent algorithm for approximate weighted sampling in SAT. The core idea consists of partitioning the solution space into a number of “cells” and sampling a solution from a random cell. Partitioning with desired properties is obtained via augmenting the SAT problem with uniformly random XOR constraints (XORs).

To sample a solution, WeightGen dynamically estimates the number of XORs required to obtain a suitable cell, generates random XORs, stores the solutions of the augmented problem (i.e., a random cell), and returns a perfect weighted sample from the cell. Owing to the properties of partitioning with uniformly random XORs, WeightGen provides theoretical performance guarantees regarding quality of samples and efficiency of the sampling procedure.

For implementation purposes, WeightGen only requires an efficient oracle that enumerates solutions. Moreover, it treats the target sampling distribution as a black box: it requires neither a compact description thereof, nor the knowledge of the normalization constant. Both features are crucial in pattern sampling settings. Flexics [15], a recently proposed pattern sampler based on WeightGen, has been shown to be accurate and efficient. See Appendix 0.A for a more detailed description of these algorithms.

Preference learning. The problem of learning ranking functions is known as object ranking [19]. A common solving technique involves minimizing pairwise loss, e.g., the number of discordant pairs. For example, user feedback $U=\left\{p_{1}\succ p_{3}\succ p_{2},\ p_{4}\succ p_{2}\right\}$ is seen as $\left\{\left(p_{1}\succ p_{3}\right),\left(p_{1}\succ p_{2}\right),\left(p_{3}\succ p_{2}\right),\left(p_{4}\succ p_{2}\right)\right\}$. Given feature representations of objects $\vec{p_{i}}$, object ranking is equivalent to positive-only classification of difference vectors, i.e., a ranked pair example $p_{i}~{}\succ~{}p_{j}$ corresponds to a classification example $\left(\vec{p_{i}}-\vec{p_{j}},+\right)$. All pairs comprise a training dataset for a scoring classifier. Then, the predicted ranking of any set of objects can be obtained by sorting these objects by classifier score descending. For example, this formulation is adopted by SvmRank [18].

5 Algorithm

Key questions concerning instantiations of the Mine, interact, learn, repeat framework include

1. the feedback format,
2. learning quality measures from feedback,
3. mining with learned measures, and crucially,
4. selecting the patterns to show to the user. As pattern sampling has been shown to be effective in mining and learning, we present LetSIP, a sampling-based instantiation of the framework which employs Flexics. The sequel describes the mining and learning components of LetSIP. Algorithm 1 shows its pseudocode.

Mining patterns by sampling. Recall that the main goal is to discover patterns that are subjectively interesting to a particular user. We use parameterised logistic functions to measure the interestingness/quality of a given pattern $p$:

[TABLE]

where $\vec{p}$ is the vector of pattern features for $p$, $\vec{w}$ are feature weights, and $A$ is a parameter that controls the range of the interestingness measure, i.e. $\varphi_{\text{logistic}}\in\left(A,\ 1\right)$. Examples of pattern features include $Length\left(p\right)={\left|p\right|}/{\left|\mathcal{I}\right|}$, $Frequency\left(p\right)=freq\left(p\right)/{\left|\mathcal{D}\right|}$, $Items\left(i,p\right)=\left[i\in p\right]$; and $Transactions\left(t,p\right)=\left[p\subseteq t\right]$, where $\left[\cdot\right]$ denotes the Iverson bracket. Weights reflect feature contributions to pattern interestingness, e.g., a user might be interested in combinations of particular items or disinterested in particular transactions. The set of features would typically be chosen by the mining system designer rather than by the user herself. We empirically evaluate several feature combinations in Section 6.

Specifying feature weights manually is tedious and opaque, if at all possible. Below we present an algorithm that learns the weights based on easy-to-provide feedback with respect to patterns. This motivates our choice of logistic functions: they enable efficient learning. Furthermore, their bounded range $\left[A,\ 1\right]$ yields distributions that allow efficient sampling directly proportional to $\varphi_{\text{logistic}}$ with Flexics. Parameter $A$ essentially controls the tilt of the distribution [15].

User interaction & learning from feedback. Following previous research [13], we use ordered feedback, where a user is asked to provide a total order over a (small) number of patterns according to their subjective interestingness; see Figure 6 for an example. We assume that there exists an unknown, user-specific target ranking $R^{*}$, i.e., a total order over $\mathcal{L}$. The inductive bias is that there exists $\vec{w}^{*}$ such that $p\succ q\Rightarrow\varphi_{\text{logistic}}\left(p,\vec{w}^{*}\right)>\varphi_{\text{logistic}}\left(q,\vec{w}^{*}\right)$. We apply the reduction of object ranking to binary classification of difference vectors (see Section 4). Following Boley et al. [7], we use Stochastic Coordinate Descent (SCD) [23] for minimizing L1-regularized logistic loss. However, unlike Boley et al., we directly use the learned functions for sampling.

SCD is an anytime convex optimization algorithm, which makes it suitable for the interactive setting. Its runtime scales linearly with the number of training pairs and the dimensionality of feature vectors. It has two parameters:

1. the number of weight updates (per iteration of LetSIP) $T$ and
2. the regularization parameter $\lambda$. However, direct learning of $\varphi_{\text{logistic}}$ is infeasible, as it results in a non-convex loss function. We therefore use SCD to optimize the standard logistic loss, which is convex, and use the learned weights $\vec{w}$ in $\varphi_{\text{logistic}}$.

Selecting patterns to show to the user. An interactive system seeks to ensure faster learning of accurate models by targeted selection of patterns to show to the user; this is known as active learning or query selection. Randomized methods have been successfully applied to this task [13]. Furthermore, in large pattern spaces the probability that two redundant patterns are sampled in one (small) batch is typically low. Therefore, a sampler, which produces independent samples, typically ensures diversity within batches and thus sufficient exploration. We directly show $k$ patterns sampled by Flexics proportional to $\varphi_{\text{logistic}}$ to the user, for which she has to provide a total order as feedback.

We propose two modifications to Flexics, which aim at emphasising exploitation, i.e., biasing sampling towards higher-quality patterns. First, we employ alternative cell sampling strategies. Normally Flexics draws a perfect weighted random sample, once it obtains a suitable cell. We denote this strategy as $\varsigma=$ Random. We propose an alternative strategy $\varsigma=$ Top($m$), which picks the $m$ highest-quality patterns from a cell (Line 15 in Algorithm 1). We hypothesize that, owing to the properties of random XOR constraints, patterns in a cell as well as in consecutive cells are expected to be sufficiently diverse and thus the modified cell sampling does not disrupt exploration.

Rigorous analysis of (unweighted) uniform sampling by Chakraborty et al. shows that re-using samples from a cell still ensures broad coverage of the solution space, i.e., diversity of samples [11]. Although as a downside, consecutive samples are not i.i.d., the effects are bounded in theory and inconsequential in practice. We use these results to take license to modify the theoretically motivated cell sampling procedure. Although we do not present a similar theoretical analysis of our modifications, we evaluate them empirically.

Second, we propose to retain the top $l$ patterns from the previous query and only sample $k-l$ new patterns (Lines 9–10). This should help users to relate the queries to each other and possibly exploit the structure in the pattern space.

6 Experiments

The experimental evaluation focuses on 1) the accuracy of the learned user models and 2) the effectiveness of learning and sampling. Evaluating interactive algorithms is challenging, for domain experts are scarce and it is hard to gather enough experimental data to draw reliable conclusions. In order to perform extensive evaluation, we emulate users using (hidden) interest models, which the algorithm is supposed to learn from ordered feedback only.