Approaching Adaptation Guided Retrieval in Case-Based Reasoning through Inference in Undirected Graphical Models

TL;DR

This paper introduces a novel adaptation-guided retrieval method in case-based reasoning using Markov Random Fields to improve retrieval of adaptable cases, especially when similarity assumptions are weak.

Contribution

It proposes a new approach that models case adaptability within an MRF, enhancing retrieval by considering solution space proximity and adaptability levels.

Findings

Enlarges the set of potentially adaptable cases without losing accuracy

Uses MRF inference to improve retrieval in case-based reasoning

Combines simple kNN retrieval with MRF-based refinement

Abstract

In Case-Based Reasoning, when the similarity assumption does not hold, the retrieval of a set of cases structurally similar to the query does not guarantee to get a reusable or revisable solution. Knowledge about the adaptability of solutions has to be exploited, in order to define a method for adaptation-guided retrieval. We propose a novel approach to address this problem, where knowledge about the adaptability of the solutions is captured inside a metric Markov Random Field (MRF). Nodes of the MRF represent cases and edges connect nodes whose solutions are close in the solution space. States of the nodes represent different adaptation levels with respect to the potential query. Metric-based potentials enforce connected nodes to share the same state, since cases having similar solutions should have the same adaptability level with respect to the query. The main goal is to enlarge the set of potentially adaptable cases that are retrieved without significantly sacrificing the precision and accuracy of retrieval. We will report on some experiments concerning a retrieval architecture where a simple kNN retrieval (on the problem description) is followed by a further retrieval step based on MRF inference.

Tables

Table 1. Table 1: Features and domains of the Travel case base

Name	Domain
Duration	Numeric
#Persons	Numeric
Accommodation	Ordinal
Season	Ordinal
HolidayType	Categorical
Destination	Categorical
Transport	Categorical

Equations

$$\mathcal{P}(X_1\dots X_n)=\frac{1}{Z}\prod _{i,j}{\Phi }_{i,j}(X_i,X_j)$$

$${\Phi }_{i,j}(x_i,x_j)=\exp (-w_{ij}d(x_i,x_j))$$

$$s(i,j)=\frac{1}{1+d(i,j)}$$

Full text

1 Introduction

In Case-Based Reasoning (CBR), the similarity assumption states that similar problems have similar solution(s). The similarity defined on the case description is often called structural similarity, in contrast to the solution similarity defined over the solution space. Because of that, the CBR problem solving process is based on the well-known 4R steps: Retrieve, Reuse, Revise and Retain [2]. The more valid the similarity assumption, the more efficient the CBR process is, since the retrieved solutions are more similar to the (unknown) solution to the query.

The most common retrieval strategy is based on k-Nearest Neighbor (kNN) algorithms, returning the solutions of the $k$ cases stored in the library that are most similar (from the structural point of view) to the query. However, the similarity assumption is not always guaranteed to hold, and it has been questioned several times [11, 12, 3]. Adaptation guided retrieval can be exploited when it is not possible to rely only on structural similarity. Different solutions have been devised in this context: the introduction of specific or task dependent adaptation knowledge into the retrieval step [11, 9, 5], the modeling of solution preferences in preference-based CBR [3], the learning of a utility-oriented similarity measure minimizing the discrepancies between the similarity values and the desired utility scores [14].

In this paper, we describe and experiment with a novel technique where standard kNN retrieval is complemented through an adaptation guided inference process; in particular, we propose to model adaptation knowledge through the construction of a particular type of undirected graphical model, a metric Markov Random Field (MRF) [7], and to exploit MRF inference to enhance the retrieval step in terms of more adaptable cases.

The paper is organized as follows: section 5 outlines some basic notions concerning MRFs and metric MRFs; section 3 describes the characterization of case adaptability we rely on; section 4 discusses a framework for case retrieval based on inference on an MRF capturing the relevant adaptation knowledge for the cases of interest; section 5 proposes a possible retrieval architecture integrating kNN retrieval with the MRF inference enhancement step; section 6 introduce the experimental framework for the evaluation of the proposed architecture, section 7 reports the results which are finally discussed in section 8.

2 Markov Random Fields

A Markov Random Field (MRF) is a undirected graphical model defined as the pair $\langle\cal{G},\cal{P}\rangle$ where $\cal{G}$ is an undirected graph whose nodes represent random variables (we assume here discrete random variables) and edges represent dependency relations among connected variables (i.e. an edge between $X_{i}$ and $X_{j}$ means that $X_{i}$ and $X_{j}$ are dependent variables); $\cal{P}$ is a probabilistic distribution over the variables represented in $\cal{G}$. We restrict our attention to pairwise MRFs: each edge $(X_{i}$ — $X_{j})$ is associated with a potential $\Phi_{i,j}:D(X_{i})\times D(X_{j})\rightarrow\mathbb{R}^{+}\cup\{0\}$; here $D(X)$ is the domain (i.e. the set of states or values) of the variable $X$.

In a MRF, the distribution $\cal{P}$ factorizes over $\cal{G}$; this means that

[TABLE]

where $Z=\sum_{X_{1}\ldots X_{n}}\prod_{i,j}\Phi_{i,j}(X_{i},X_{j})$ is a normalization constant called the partition function.

We consider a special case of pairwise MRF called metric MRF. In a metric MRF, all nodes take values in the same label space $V$, and a distance function $d:V\times V\rightarrow\mathbb{R}^{+}\cup\{0\}$ is defined over $V$; the edge potentials are defined as

[TABLE]

given that $x_{i}$ is a value or state of variable $X_{i}$, and $w_{ij}>0$ is a suitable weight stressing the importance of the distance function in determining the potential. The idea is that adjacent variables are more likely to have values that are close in distance.

Concerning inference, we are interested in the computation of the posterior probability distribution of each single unobserved variable, given the evidence. In this paper, we will resort to mean field inference, a variational approach where the target distribution is approximated by a completely factorized distribution [13]. This algorithm is implemented in the Matlab UGM toolbox [10] that we have exploited in our experimental analysis.

3 Case Solutions and Adaptation Knowledge

The standard CBR process requires the definition of a suitable distance metric over the case description (i.e. the case features) and it employs a kNN algorithm, in order to get the k most structurally similar cases. Given the similarity assumption, we expect that the solutions of similar cases are also similar. If this assumption is not valid, kNN retrieval can result is a set of useless cases, since their adaptation level with respect to the query is unsuitable (i.e., no adaptation mechanism can be either devised or adopted with a reasonable effort). If this is the case, some abstract notion of adaptation space should be defined, and adaptation knowledge has to be exploited during retrieval (see [6, 4]). A common approach is to consider the solution space equipped with a similarity or a distance metric [12, 3]. This metric can then be used to measure how close two potential solutions are: the closer the solutions, the more similar the adaptation effort needed to revise them for a specific query. When adapting a solution with respect to a query, we need to use the available adaptation knowledge; depending on the “complexity” of the inference process executed during adaptation, we can devise different adaptation levels corresponding to the effort (or cost) needed to perform this phase. For example, if no adaptation is needed (i.e., the query case is solved through the reuse step) we can map this situation to the minimum level of adaptation effort. On the contrary, suppose that the adaptation knowledge is provided through adaptation rules: the type and the number of applied rules can define a measure of the effort needed to revise the solution, and such a measure can be mapped into different adaptation levels (see below).

Example 1. Suppose we have a case library containing the description of some holiday packages (this corresponds to the case study Travel described in section 6); a customer requires a package by specifying some features as reported in Table 1, and the returned solution consists in a specific package characterized by such features, and completed with the name of a hotel and the package final price. Let each hotel be described by an object $h$ with properties $h.\mbox{Name}$ (the hotel name), $h.\mbox{Location}$ (the hotel place) and $h.\mbox{Category}$ (the hotel classification in number of stars).

Consider now a customer with a given budget $b$ and with some specific criteria $A(h)$ for accepting a proposed hotel $h$: if $A(h)={\tt true}$ then the hotel is accepted by the customer, otherwise it is rejected.

Given a query (with the specification of some of the features in Table 1), suppose we have the adaptation rules reported in Figure 1 (where $r$ refers to the retrieved case, $q$ to the query, $a$ to the adapted case and $h$ to a particular hotel) to be applied in sequence. Given the above adaptation rules, we can in principle define different adaptation levels by considering the type and the number of rules applied during revise. By way of example, we could devise the following levels:

• •

Level 1: if no rule is applied

• •

Level 2: if only rule R1 is applied

• •

Level 3: if one or more rules R2, R3, R4 are applied with success

• •

Level 4: if case is flagged as not adaptable

For instance, we can consider a solution with level 1 to be reusable, a solution with level 2 to be revisable with a small cost, a solution with level 3 revisable with a larger cost and a solution with level 4 to be unadaptable. In the following, we propose to characterize the principle “similar solutions imply similar adaptation levels” by means of a metric MRF built on a given case library; section 4 will discuss how MRF inference can be then exploited to provide an adaptation guided retrieval approach.

4 MRF Inference for Retrieval of Adaptable Cases

Given a case library of stored cases with solutions, we first construct a metric MRF. Let #adapt_levels be the number of different adaptation levels, $sim\_s$ be the similarity metric defined over the solution space and $st$ be a threshold of minimal similarity for solutions. Algorithm 1 shows the pseudo-code for the construction of the MRF.

The idea is to build a metric MRF where nodes have the possible case adaptation levels as states and they are connected only if the corresponding cases have a sufficiently large solution similarity (greater than the threshold $st$). Edge potentials $\Phi_{n,m}(i,j)$ are determined as in metric MRFs: the smaller the distance between the states $i$ and $j$ of nodes $n$ and $m$ respectively, the larger the related potential entry. When a given node assumes a specific state, connected nodes tend to assume close state values with a high probability (i.e., if a case has a given adaptation level, we expect the cases having similar solution to have a very close adaptation level). Moreover, the more similar the solutions of the cases, the stronger this effect should be; this is the reason why we use solution similarity $s=sim\_s(n,m)$ as a weight for the metric potential. Once we know the adaptation level of some of the stored cases, MRF inference can be used to propagate this information in the case library; this for identifying other cases as good candidates for solution reuse or revision, as well as rejecting some cases because they are likely to be useless for adaptation or reuse. The idea is then to start from a standard kNN retrieval, followed by the usual reuse and revise steps. The results of the reuse/revise phases are used as input for MRF inference. Algorithm 2 details this process.

Let $al(i)$ be the adaptation level of the $i$-th retrieved case (through kNN retrieval); for each retrieved case, its adaptation level is set as evidence in the corresponding node of the MRF. Inference is then performed and the posterior probability of each MRF node is computed into the multidimensional vector $Bel$. $Bel[i]$ is the posterior distribution or node belief of node $i$; $Bel[i]$ is a $l$-dimensional vector (where $l$ is the number of adaptation levels) such that $Bel[i,j]$ is the probability of node $i$ being in state $j$ given the evidence. Input parameter $cond()$ is a function testing a condition on the node belief; if this condition is not satisfied, it returns false, otherwise it returns the state of the node (i.e., the adaptation level) for which the condition is satisfied. Algorithm 2 finally outputs a set of cases with their adaptability level.

Example 2. Suppose we want to determine for each (non retrieved) case the most probable adaptation level, then we will set $cond(Bel[i])=\arg\max_{j}Bel[i,j]$. In this case every case has a potential adaptation level and we can consider it for the next actions: for instance, we could be interested only in the most easily adaptable cases, and if $1$ is the minimum adaptation level, we will select only those nodes $i$ for which $cond(Bel[i])=1$

Consider now a more complex condition: suppose we consider as interesting any adaptation level from $1$ to $a$, and suppose that we want to be pretty sure about the adaptability of the case. We could set a probability threshold $pt$ and to require that

if $Bel[i,1]+\ldots Bel[i,a]>pt$ then

     $cond(Bel[i])\leftarrow a$

  else

     $cond(Bel[i])\leftarrow{\tt false}$

  end if

In this case we are collapsing all the adaptability levels from 1 to $a$ into a unique level (the choice of $a$ is completely arbitrary here, and any other label would be fine as we no longer need to distinguish them); in case the required confidence on adaptability is not reached, we will simply ignore the case. Of course, several other implementations of the $cond()$ function can be devised.

5 An Architecture for Adaptation Guided Retrieval

The problem of retrieving “useful” cases with respect to a given query is characterized by two different aspects: the structural similarity between the query and the retrieved case (addressed by kNN retrieval), and the adaptability to the query of the retrieved solution (addressed by MRF inference); this means that the cases of interest are those which are sufficiently similar to the query, while having an adaptable solution. We call them positive cases. We would like the retrieval to return only positive cases, possibly with a large structural similarity and with a low adaptability cost (i.e., a low adaptability level). While cases retrieved through kNN do not have the guarantee of being adaptable, cases retrieved through MRF inference are more likely to be adaptable, but they do not have any guarantee of being sufficiently similar to the query. Moreover, the reason why retrieval is often restricted to a set of $k$ cases, is because it is in general unfeasible to take into consideration all the positive cases: considering all the cases returned by MRF inference may lead to an unreasonably large number of cases to be managed.

The proposed retrieval architecture starts with standard kNN retrieval; if all the $k$ retrieved cases are actually adaptable, then the process terminates with such $k$ cases as a result. On the contrary, let $0\leq k^{\prime}<k$ be the number of adaptable cases retrieved by kNN. Since there is still room for finding positive cases, MRF inference is performed as shown in Algorithm 2, then the top $k-k^{\prime}$ cases in descending order of structural similarity are returned, from the output of Algorithm 2. The main idea underlying this architecture is that $k$ is the desired output size. In case kNN retrieval tangles with the solution similarity problem, we complement the retrieval set with some cases that are likely to be adaptable. Since they are selected by considering their structural similarity with respect the query, we also maximize the probability of such cases being positive. In order to evaluate the effectiveness of such architecture, we set up an experimental framework described in section 6, and which results are reported in section 7.

6 Experimental Testbed: the Travel dataset

As a testbed for the approach, we consider a dataset called Travel containing instances of about $1500$ holiday packages described through the features reported in Table 1. Such features are those used to query the case base; in addition, stored cases also contain the price of the package and the name of a hotel.

Local distances for features are defined as follows: for numeric features and for the ordinal attribute Accommodation (mapped into integers from [math] to $5$) we adopted the standardized Euclidean distance (Euclidean distance normalized by feature’s standard deviation); for categorical features we adopted the overlap distance ([math] if two values are equal and $1$ if they are different); finally for the ordinal attribute Season we adopted a cyclic distance, since the values are mapped into the ordinal numbers of the months. The cyclic distance on a feature $f$ is defined by the following formula $d_{f}(x,y)=\min(|x-y|,R_{f}-|x-y|)$ where $x,y$ are the values (from $1$ to $12$ in such a case) and $R_{f}=\mbox{range}(f)+1$ ($12$ in this case). In all the above cases, when there is a missing value, the maximum distance value for the feature is considered. We also have defined a vector of feature weights and the aggregation function producing the global distance $d(i,j)$ between two cases $i$ and $j$ is the weighted average.

The solution of a case is actually a complete description of a package (including a specific hotel and the computed price). Since adaptation knowledge considers the price and the hotel characteristics, in order to revise a potential solution, the characterization of the similarity of solutions only requires information about the price, the destination and the the hotel category. Distance between two solutions is again computed as a weighted average of the three local distances on Price, Accommodation and Destination.

Similarity measure for both case descriptions and solutions is computed as

[TABLE]

where $d()$ is the global distance ($0<s(i,j)\leq 1$).

We consider the example adaptation rules illustrated in Section 3; we set a budget $b$ on price equal to the average price of the packages in the case base; moreover, in order to simulate the acceptability criterion, we implemented a random acceptance with probability of $80\%$. Concerning the adaptability levels, we consider a binary characterization with only two levels: $1=\mbox{adaptable}$, $2=\mbox{not adaptable}$. In our experiments, we also set the threshold $pt=0.9$ (i.e., MRF inference considers a case as adaptable if the probability of the corresponding MRF node of assuming state $1$ is greater than $90\%$). Finally, we define a similarity threshold $thr$ on the structural similarity between a retrieved case and a query. A “positive” case is a case having a structural similarity with the query greater than or equal to $thr$, and such that its adaptability level is $1$. With the above characterization of positive cases, given a particular retrieval set, we consider the usual notions of accuracy, precision, recall (more precisely precision and recall at $k$ since we focus on retrieving $k$ cases) and F1-measure (the harmonic mean of precision and recall).

7 Experimental Results

If in a problem there are features that are predictive of the solution, then we expect that similar values of such features will corresponds to similar values of the solution. In order to stress the “similarity assumption” and to bring out problems related to it, we have considered queries with missing values for such “solution-correlated” features. We decided to evaluate the performance of the proposed architecture by selecting as potential missing features the attiributes Duration, Accommodation and HolidayType; taken together they have a correlation coefficient with the Price (the most relevant part of the solution) of $0.77$, thus it can be expected that, missing values on such features will reduce the validity of the similarity assumption.

For a given query, we set the probabilities of a missing value as follows: $p_{A}=0.15$ for Accommodation, $p_{D}=0.3$ for Duration and $p_{H}=0.6$ for HolydayType. In all the experiments we adopted the thresholds $st=0.9$ (see Algorithm 1) and $pt=0.9$ (see Example 2 in section 4). The construction of the metric MRF (Algorithm 1) and the MRF inference (Algorithm 2) have been implemented in Matlab by exploiting the UGM toolbox [10]. In particular, we resort to mean field variational inference as previously mentioned111 Comparable results have been obtained by using Loopy Belief Propagation [13].. We finally consider two other evaluation parameters, the structural similarity threshold $thr$ and the number of retrieved cases $k$, and we perform different runs by varying such parameters. In particular, we set $thr=\alpha\mu_{c}$ where $\mu_{c}$ is the average structural similarities among the cases in the case library, and $\alpha$ is a scale factor. Variation on $thr$ influences the set of positive cases222 Others parameters that can be used to vary the set of positive cases are those influencing adaptation, namely the customer budget $b$ and the hotel acceptability criterion $A(h)$; for the sake of brevity we do not consider them here. while variation on $k$ models different retrieval capabilities. We set values of $k$ from 1 to 15 with step of 1, and then from 20 to 100 with step of 10. Greater is the threshold $thr$, smaller is the set of positive cases and vice-versa; values of $\alpha$ are chosen in such a way of setting the threshold at $75\%$ of the average similarity, at the median of the case similarities ($95\%$ of the average in this problem), at the average similarity, and finally setting the threshold $25\%$ and $50\%$ above the average similarity. In particular, given a case base of $1500$ cases, the average number of positive cases resulted as follows: $559$ for $\alpha=0.75$, $400$ for $\alpha=0.95$, $320$ for $\alpha=1$, $86$ for $\alpha=1.25$, $11$ for $\alpha=1.5$. A few words are needed to explain the choice of the considered $k$ values: small values are reasonable when the query should not return too many results to the user (as in the case of recommending holiday packages). However, our aim here is to use the Travel dataset as an evaluation framework, not tied to the specific application task of recommending travels; in situations where the scale factor $\alpha\leq 1$, the number of relevant cases (i.e., positive cases) is rather large, producing a very low recall for small values of $k$. For this reason we considered large values of the $k$ parameter (from 20 to 100) as well.

We have performed a 10-fold cross validation for every considered value of $k$, by measuring every time the mean values for accuracy, precision, recall and F1-measure, for both simple kNN retrieval (label kNN in the figures) and kNN+MRF inference (label MRF in the figures).

Figure 2 shows the Precision/Recall (PR) curves obtained for the different values of the scale factor $\alpha$, by varying the value of $k$: increasing the value of $k$ will increase recall, by decreasing precision. The curves are actually an approximation of the real PR curves, since in the experiments we almost never reached a situation with a recall close to $1$333 To obtain this result we should have considered a very large value for parameter $k$; indeed, when the set of positive cases is large ($\alpha\leq 1$ in our experiments), recall is necessarily low, and can be increased only with large values of $k$.; as usual in these cases, we set the last point of the PR curve with the pessimistic estimate for precision corresponding to $\frac{P}{N}$, where $P$ is the number of positive cases, and $N$ the total number of cases. An exception to this situation is the case $\alpha=1.5$, where with $k=90$ we get an average recall very close to $1$, producing the PR curve shown in the bottom right of Figure 2. We also computed the Area Under the Curve (AUC); for the reasons outlined above, the computed value is a pessimistic estimate (smaller than the actual one), but in the case of $\alpha=1.5$ where it has been possible to compute it exactly. The graphics of Figure 2 with $\alpha\leq 1$ only show a part of the PR curve, since the estimated last point $(1,\frac{P}{N})$ would result really far from the last measured point (the reported values for AUC are however computed by taking into account the whole curve).

Figure 3 shows the behavior of accuracy and F1 measure, in dependence of $k$, for different values of $\alpha$. Values for the accuracy are plotted on the right axis, values for F1 on the left axis.

Accuracy and F1 measure for the specific case of $\alpha=1.5$ are reported in Figure 4 (again, values for the accuracy are plotted on the right axis, values for F1 on the left one).

In particular, we report the whole plot ($k=1\ldots 90$) on the left part of the figure, and we “magnify” the plot for $k=1\ldots 10$ on the right part.

8 Discussion and Conclusions

From the experimental results we notice that the benefits of the MRF approach with respect to simple kNN retrieval (in terms of balance between precision and recall) are all the more evident how large is the size of the set of positive cases (i.e., for small values of $\alpha$). This is noticeable from both PR curves (and the corresponding values for AUC), and F1 measure. When the are a a lot of cases sufficiently similar to the query and potentially adaptable, kNN alone has difficulty in retrieving positive cases: simple structural similarity is not sufficient and the integration with MRF inference is fruitful.

The MRF integration also provide benefits in terms of accuracy as shown in Figure 3, since some false negatives are actually moved into true positives with respect to simple kNN. In general, accuracy (for both kNN and MRF) turns out to be negatively correlated with the size of the set of positive cases. Moreover, when the number of retrieved cases $k$ is too large with respect to the number of positives, accuracy shows a decreasing pattern as we can expect, since too many false positives can be potentially retrieved (this is evident in the plots relative to $\alpha=1.25,1.5$).

A better behavior of kNN is apparent in case of $\alpha=1.5$, but it is worth noting that in this situation we have very few positive cases ($11$ on average in each run), making not very significant the results for large values of $k$. This is the reason why in Figure 4 we considered also the situation restricted to $1\leq k\leq 10$; by considering reasonable values for $k$, even in the case of $\alpha=1.5$, both accuracy and F1 measure show a small advantage in adopting the MRF integration to kNN retrieval. In conclusions, the evaluation in terms of accuracy, precision and recall of the considered testbed suggests that the proposed integrated architecture can provide advantages, when simple structural similarity is not able to suitably capture the actual effort in adapting the retrieved solutions to the current query. A final remark is worth on the characteristics of the MRF models that have been obtained in the study; they are undirected graphs that tend to have multiple connected components, since only cases having close solutions are connected, naturally resulting in different independent groups of nodes (see Figure 5 for a typical example).

This means that even if the number of cases becomes large, the approach is likely to scale-up; inference on such groups can be performed independently, and if they involve a limited number of nodes, even exact inference may be attempted.

The integration of CBR and graphical models for retrieval has been usually investigated by concentrating on directed models like Bayesian Networks (BN). In [1], a BN model is coupled with a semantic network to adress case indexing and retrieval. BN-based retrieval is triggered by the introduction of the observed features as evidence, and cases can be set in a particular on state and retrieved if the posterior probability of such a state exceeds a given threshold. Recent advances in this setting are presented in [8] within the BNCreek system which applies a Bayesian analysis aimed at increasing the accuracy of the similarity assessment. These approaches focus only on structural similarity and there is no attempt to address the problem of adaptation-guided retrieval. Our approach can then be seen as the first attempt of exploiting probabilistic inference on a graphical model to build a strategy for adaptation guided retrieval.