Relevance Feedback with Latent Variables in Riemann spaces

TL;DR

This paper introduces two novel relevance feedback methods using Riemannian geometry in semantic query spaces, employing Gaussian and latent semantic models, and provides a new evaluation methodology for comparison.

Contribution

It develops and evaluates two new relevance feedback techniques based on Riemannian metrics and probabilistic models, with a novel experimental evaluation approach.

Findings

Gaussian-based relevance feedback improves retrieval accuracy.

Latent semantic variable model enhances relevance estimation.

New evaluation methodology offers unbiased comparison of feedback methods.

Abstract

In this paper we develop and evaluate two methods for relevance feedback based on endowing a suitable "semantic query space" with a Riemann metric derived from the probability distribution of the positive samples of the feedback. The first method uses a Gaussian distribution to model the data, while the second uses a more complex Latent Semantic variable model. A mixed (discrete-continuous) version of the Expectation-Maximization algorithm is developed for this model. We motivate the need for the semantic query space by analyzing in some depth three well-known relevance feedback methods, and we develop a new experimental methodology to evaluate these methods and compare their performance in a neutral way, that is, without making assumptions on the system in which they will be embedded.

Tables

Table 1. Table 1:

Symbol	Range	meaning
$D$	$\mathbb{N}$	Number of elements in the data base
$𝒟$	$\{d_1,\dots ,d_D\}$	Data base of documents
$N$	$\mathbb{N}$	Number of elements selected for feedback
$R$	$\mathbb{N}$	Number of dimension in the reduced representation of Rui and Huang
$W$	$\mathbb{N}$	Number of word spaces in the latent model.
$K$	$\mathbb{N}$	Number of latent variables.
$M$	$\mathbb{N}$	Dimension of a generic vector.
$M_{\nu }$	$\mathbb{N}$	Dimension of the $\nu$th word space.
$𝔽$	$\equiv {ℝ}^M$	Complete feature space.
${𝔽}^{\nu }$	$\equiv {ℝ}^{M_{\nu }}$	Feature space of the $\nu$th word.
$u_n$	${ℝ}^M$	Feature vector of the $n$th sample
$u_{n\nu }$	${ℝ}^{M_{\nu }}$	Feature vector of the $\nu$th word if the $n$th sample
$u_{n\nu ,p}$	${ℝ}^{M_{\nu }}$	$p$th dimension of the $\nu$th word if the $n$th sample
indices
Index	span	elements indexed
$n,m$	$1,\dots ,D$	elements in a data base (used rarely, will not conflict with the other use of the same symbols)
$n,m$	$1,\dots ,N$	elements in a feedback set or an image set
$k,h$	$1,\dots ,K$	latent variables
$\nu ,\mu ,\zeta ,\xi$	$1,\dots ,W$	word spaces
$p,v$	$1,\dots ,M$	dimensions of the feature spaces

Table 2. Table 2:

Variables $𝒟$ the data base $M$ the set of images of the target category ($M\subseteq 𝒟$) $ℛ$ the positive samples selected for relevance feedback ($ℛ\subseteq M$) $Q$ the result list after relevance feedback		Paramters $D$ size of $𝒟$ $m$ size of $M$ $r$ size of $ℛ$ ($r\le m$) $q$ size of $Q$ ($q\le m$) $k$ number of elements of $Q$ that belong to $M$ (number of hits)
(a)		(b)

Equations

$$q^{t+1}=q^{t+1}(q^0,q^t,{\mathcal{R}}^t,{\mathcal{R}}_{+}^t,{\mathcal{R}}_{-}^t).$$

$$q^{t+1}$$

$${\Omega }^{t+1}$$

$$u_n=[u_{n1},\dots ,u_{nM}{]}^{\prime }$$

$$Q(t+1)=(q(t+1),\delta(t+1))=\Phi\bigl{(}q(t),\delta(t),[{\mathcal{R}}_{+}^{0},\ldots,{\mathcal{R}}_{+}(t)],[{\mathcal{R}}_{-}^{0},\ldots,{\mathcal{R}}_{-}(t)]\bigr{)}$$

$$(q(t),\delta(t))=\Phi\bigl{(}q(t),\delta(t),[{\mathcal{R}}_{+}^{0},\ldots,{\mathcal{R}}_{+}(t-1),\emptyset],[{\mathcal{R}}_{-}^{0},\ldots,{\mathcal{R}}_{-}(t-1),\emptyset]\bigr{)}$$

$$u_p=[u_{1,p},\dots ,u_{N,p}{]}^{\prime }$$

$${\sigma }_p=\text{mbox}var(u_p).$$

$$d^{2}(u_{n},q(t))=\Bigl{[}\prod_{p=1}^{M}\sigma_{p}\Bigr{]}^{\frac{1}{M}}\sum_{p=1}^{M}\frac{(u_{n,p}-q_{,p}(t))^{2}}{\sigma_{p}}$$

$$D^2(u_n,q)=(u_n-q{)}^{\prime }\mathbf{M}(u_n-q)$$

$$\sum _{n=1}^N{\pi }_n(r_n-q{)}^{\prime }\mathbf{M}(r_n-q)$$

$$q=\frac{1}{{\sum }_n{\pi }_n}\sum _n{\pi }_n{u}_n$$

$$c_{pv}=\sum _n{\pi }_n(r_{n,p}-q_p)(r_{n,v}-q_v)$$

$$\mathbf{M}=\text{mbox}det(\mathbf{C}{)}^{\frac{1}{M}}{\mathbf{C}}^{-1}$$

$$\mathbf{C}={\mathbf{U\Sigma U}}^{\prime }$$

$$\mathbf{\Sigma}=\mbox{diag}\bigl{(}\sigma_{1},\ldots,\sigma_{R},0,\ldots,0)$$

$$\mathbf{\Sigma}^{+}=\mbox{diag}\bigl{(}1/\sigma_{1},\ldots,1/\sigma_{R},0,\ldots,0)$$

$${\mathbf{C}}^{+}={\mathbf{U\Sigma }}^{+}{\mathbf{U}}^{\prime }$$

$$u_{n\nu}=\bigl{[}u_{n\nu,1},\ldots,u_{n\nu,M_{\nu}}\bigr{]}^{\prime}$$

$$d_n=d(u_n,q)=w^{\prime }{g}_n$$

$$g_n$$

$$g_{n\nu }$$

$$\underset{{\mathbf{M}}^{\nu },q_{\nu },w}{\min }{\pi }^{\prime }d$$

$$d$$

$$d_n$$

$$g_n$$

$$g_{n\nu }$$

$$\sum _{\nu }\frac{1}{w_{\nu }}=1$$

$$\text{mbox}det({\mathbf{M}}^{\nu })=1$$

$$u_{n\nu }=[u_{n\nu ,1},\dots ,u_{n\nu ,M_{\nu }}{]}^{\prime }$$

$$q_{\nu }=\frac{{\mathbf{R}}^{\nu }\pi }{{\sum }_n{\pi }_n}$$

$$c_{pv}^{\nu }=\sum _n{\pi }_n(u_{n\nu ,p}-q_{\nu ,p})(u_{n\nu ,v}-q_{\nu ,v})$$

$${\mathbf{M}}^{\nu }=(\text{mbox}det({\mathbf{C}}^{\nu }){)}^{1/M}(C^{\nu }{)}^{-1}$$

Full text

We motivate the need for the semantic query space by analyzing in some depth three well-known relevance feedback methods, and we develop a new experimental methodology to evaluate these methods and compare their performance in a neutral way, that is, without making assumptions on the system in which they will be embedded.

1 Introduction

Relevance Feedback is an important and widespread Information Retrieval technique for query modification that formulates a new query based on a previous one and on the response of the user to the answer to that query. It is, by now, a classic: it origin in Information Retrieval can be traced back to the early 1970s, with the work of Rocchio on the SMART retrieval system [37]. Like any true classic, relevance feedback has provided a nearly inexhaustible breeding ground for methods and algorithms whose development continue to this day.

In very general terms, let ${\mathcal{D}}=\{{u_{1},\ldots,u_{D}}\}$ be a data base containing $D$ items, and let $q^{0}$ be a query. As a result of the query, the data base proposes a set of $N$ results, ${\mathcal{R}}^{0}\subseteq{\mathcal{D}}$. Out of this set, the user selects two subsets: the set of ${r_{+}}$ positive (relevant) documents, ${\mathcal{R}}_{+}^{0}\subseteq{\mathcal{R}}$, and the set of ${r_{-}}$ negative (counter-exemplar) ones ${\mathcal{R}}_{-}^{0}\subseteq{\mathcal{R}}$ with, in general, ${\mathcal{R}}_{+}^{0}\cap{\mathcal{R}}_{-}^{0}=\emptyset$. This information is used by the system to compute a new query $q^{1}=q^{1}(q^{0},{\mathcal{R}}^{0},{\mathcal{R}}_{+}^{0},{\mathcal{R}}_{-}^{0})$, which is then used to produce a new set of results ${\mathcal{R}}^{1}$. The process can be iterated ad libitum to obtain results sets ${\mathcal{R}}^{t}$ and queries

[TABLE]

Note that in document information retrieval the query $q^{0}$ is expressed directly by the user and is often considered more significant and stable than those which are automatically generated. For this reason, the parameter $q^{0}$ is always present in the calculation of $q^{t}$. The situation is different in image retrieval, as we shall see in the following.

Relevance feedback has been applied to many types of query systems, including systems based on Boolean queries [41, 17]. However, its most common embodiment is in similarity-based query systems, in which it offers a viable solution for expressing example-based queries [49]. This is especially useful in a field like image retrieval, in which Boolean queries are seldom used. In this case, the items $u_{i}\in{\mathcal{D}}$ are points in a metric space ${\mathbb{F}}(\Omega^{0})$, where $\Omega^{0}$ is the metric of the space. Given a query $q^{0}$, each item of the data base receives a score which depends on the distance between the item and the query as given by the metric $\Omega^{0}$, $s_{i}=s(u_{i},q^{0},\Omega^{0})$. In this case, it is possible not only to change the query based on the feedback, but to change the metric of the query space as well. Iteration $t$ is now characterized by the pair $(q^{t},\Omega^{t})$, where $q^{t}$ is the query (typically, $q^{t}$ is a point in ${\mathbb{F}}(\Omega^{t})$ or a set of such points) and $\Omega^{t}$ is the metric on which the distance from the query and, consequently, the score of each image in ${\mathcal{D}}$ will be computed. Given the feedback, the two are updated as

[TABLE]

Not all relevance feedback models implement the full scheme. Rocchio’s algorithm [37], for example, is a query rewriting technique that doesn’t change the metric of the space, while MARS [38] changes the metric space but doesn’t do explicit query rewriting. Algorithms that do metric modification usually use statistical methods to fit a parametric model to the observed relevance data. Many of these methods use only the set ${\mathcal{R}}_{+}$ of positive examples, ignoring ${\mathcal{R}}_{-}$. The reason for this is that ${\mathcal{R}}_{+}$ is usually a reasonably reliable statistical sample of what the user wants (except for the cases detailed in section 2.3). Not so ${\mathcal{R}}_{-}$, since there are often many and contrasting criteria under which an element can be deemed irrelevant. Paraphrasing Tolstoy, one could say that positive samples are all alike; every negative sample is negative in its own way. Rocchio’s algorithm, on the other hand, uses both positive and negative examples.

In this paper we shall present and evaluate two methods of relevance feedback. We shall build them in two stages. First, we shall endow a reduced-dimensionality, ”semantic” feature space with a Gaussian similarity field and with the Riemann metric induced by this field. The extension of this model to a mixture of Gaussians leads quite naturally to our second model, with latent variables. This, in turn, will result in an extension to mixed (discrete/continuous) observations of what in Information Retrieval is known as Probabilistic Latent Semantic analysis [25].

The paper is organized as follows. In section 2, we shall analyze in some detail two methods for metric change: MARS [38] and MindReader [26]. The purpose of this background is twofold. Firstly, it provides a general introduction to metric modification, making the paper more self-contained. Secondly, it serves the purpose of introducing the problematic of dimensionality reduction, which we present in section 2.3 following the work of Rui and Huang [39].

In section 3, we shall analyze the dimensionality problem from an alternative point of view, leading to the definition of semantic spaces composed of groups of semantically homogeneous features, and the definition of a higher level, reduced dimensionality, feature space, in which the dimensions are given by distance measures in the two semantic spaces.

In section 4, we present the first of our relevance feedback schemes. We shall model the set of positive examples in a semantic space using a Normal distribution, and use this distribution to endow the feature space with a Riemann metric that we shall then use to ”score” the images in the data base.

In section 5, we introduce the latent variable model. A series of binary stochastic variables is used to model abstract topics that are expressed in the positive samples. Each topic endows each semantic space with a Normal distribution. A mixed (discrete-continuous) version of Expectation Maximization (EM, [13]) is developed here to determine the parameters of the model, and then the results of section 4 are used to define a distance-like function that is used to score the data base.

In section 6, a system-neutral testing methodology is developed to evaluate the algorithms abstracting from the system of which they will be part, and is applied to evaluating the two schemes presented here vis-à-vis the three algorithms presented in section 2. Conclusions are given in section 7.

2 Relevant Background

In this section we shall describe in detail a limited number of approaches that bear direct relevance on the work presented here. We shall not try to present a bibliography of related work. Instead, we shall use this section to build a collection of techniques and open problems that we shall use in the following sections in relation to our work.

In all the following discussion we shall assume that we have a data base of $D$ elements, ${\mathcal{D}}=\{{d_{1},\ldots,d_{D}}\}$. We shall use the indices $n,m=1\ldots,D$ to span the elements of the data base. In its simplest representation each document $d_{n}$ is represented as a point $u_{n}$ is a point in a smooth manifold (in later sections we shall extend this representation to the Cartesian product of a finite number of manifolds). Depending on the model, this manifold will be either ${\mathbb{R}}^{M}$ (the $M$-dimensional Euclidean space) or ${\mathbb{S}}^{M-1}$ (the unit sphere in ${\mathbb{R}}^{M}$). We shall indicate this space as ${\mathbb{F}}$, specifying which manifold it represent whenever necessary. The individual coördinates will be identified using the indices $p,v=1,\ldots,M$. The $n$th item of the database, $d_{n}$ is represented as the vector

[TABLE]

The initial query will be denoted by $q$ (or $q^{0}$), with $q\in{\mathbb{F}}$. Note that with this choice we assume that the query is a point in the feature space. The extension of all our considerations to queries represented as sets of points in the feature space is not hard, but would complicate our presentation considerably, so we shall not consider such case. The iterations of the relevance feedback will be indicated using the indices $t,v\in{\mathbb{N}}$; these indices will be used in a functional notation ($q(t)$). The query resulting after the $t$th iteration of relevance feedback will be indicated as $q(t)$, and the relative result set as ${\mathcal{R}}(t)$. After the $t$th iteration, the user feedback will produce two sets of items: a set ${\mathcal{R}}_{+}(t)\subseteq{\mathcal{R}}(t)$ of positive items, and a set ${\mathcal{R}}_{-}(t)\subseteq{\mathcal{R}}(t)$ of negative items. The manifold ${\mathbb{F}}$ is endowed with a metric that, in general, will vary as the relevance feedback progresses. We indicate with $\Omega^{t}$ the metric and with $\delta(t)$ the associated distance function at step $t$. The query at step $t$ is the pair $Q(t)=(q(t),\delta(t))$. In its most general formulation, a relevance feedback scheme is a function $\Phi$ such that

[TABLE]

A relevance feedback scheme is stable if, whenever ${\mathcal{R}}_{+}(t)={\mathcal{R}}_{-}(t)=\emptyset$, it is $Q(t_{1})=Q(t)$, that is, if

[TABLE]

Stability entails that if an iteration doesn’t provide any new information, the query will not change. Note that in practice stability is not necessarily a desirable property: users tend to select positive examples more readily than negative ones and, if at any time the result set doesn’t provide any useful sample, the user might not select anything, leaving the system stuck in an unsatisfactory answer.

The various models presented in this paper requires us to use a fairly extended apparatus of symbols. For the convenience of the reader, we have tried to keep the meaning of the symbols and the span of the indices as consistent as possible throughout the various models that we present. The most important symbols and indices used in the rest of the paper are available at a glance in Table 1.

2.1 Life on MARS

There are compelling reasons to believe that the metric of the feature space should indeed be affected by relevance feedback. Let us consider the following idealized case, represented in Figure 1: we have two kinds of images: checkerboards and vertical-striped, of different densities (total number of stripes) and different colors (not represented in the figure). We have a feature system with three axes: one measures color, the second measures line density, and the third the ratio between the number of horizontal and vertical lines.

Our goal is to find an image of a regular checkerboard. A typical user, when shown a sample from the data base, will select images with approximately the right density, rather regular, and of pretty much any color. That is, the selected items will be rather concentrated around the correct value, and the set of positive examples will have a small variance on the ratio axis (all the positive samples will have a ratio of approximately 1:1), probably a larger one, but not too large, on the density axis (images of very low density don’t have enough lines to qualify as a ”checkerboard”) and very high variance on the color axis (color is irrelevant for the query, so the positive samples will be of many different colors). The idea of the MARS system is to use the inverse of the variance of the positive samples along an axis to measure the ”importance” of that axis, and to use a weighted Euclidean distance in which each feature is weighted by the inverse of the variance of the positive examples along that axis. Let ${\mathcal{R}}_{+}(t)=[r_{1},\ldots,r_{N}]$ be the set of positive examples at iteration $t$. Build the projection of all the results on the $p$th feature axis as

[TABLE]

and compute the variance

[TABLE]

The query point of iteration $t$, $q(t)$ is determined using Rocchio’s algorithm, and the items in the data base are given scores that depend on the following distance from the query

[TABLE]

As we mentioned above, only the positive examples are used for the determination of the metric.

2.2 MindReader and optimal affine rotations

The idea of modifying the distance of the feature space to account for the relevance of each feature has proven to be a good one, but its execution in MARS has been criticized on two grounds: first, it doesn’t take into account that what is relevant (and therefore has low variance) might not be the individual features, but some linear combination of them; second, the weighting criterion looks ad hoc, and not rigorously justified.

In [26] an example of the first problem is given. The items in a data base are people represented by two features: their height and their weight. The query asks for ”mildly overweight” people. The condition of being mildly overweight is not given by any specific value of any individual feature. If we consider, with a certain approximation, that being mildly overweight depends on one’s body mass index, then the relation is $W/H^{2}=\mbox{const}$, where $W$ is the weight in kilograms and $H$ the height in meters. So, a typical user might give a series of positive examples characterized as in figure 2.

If we consider the features individually, on each one of them the selected items have high variance, so we would conclude that the response carries little information. On the other hand, if we rotate the coördinate system as shown, the variance along the $b$ axis will be small, indicating that the corresponding weight/height ratio is relevant.

MindReader [26] takes this into account by considering a more general distance function between an item and a query, one of the form

[TABLE]

where $\mathbf{M}$ is an $M\times M$ symmetric matrix such that $\mbox{det}(\mathbf{M})=1$ (viz., a rotation). These matrices generate distance functions in which the iso-distance curves are ellipsoids centered in $q$ and whose axes can be rotated by varying the coefficients of $\mathbf{M}$. The matrix $\mathbf{M}$ and the query point $q$ are determined so as to minimize the sum of the weighted distances of the positive samples from the query. That is, given the $N$ weights $\pi_{n}$ ($n=1,\ldots,N$, $0\leq\pi_{n}\leq 1$), the matrix $M$ and the vector $q$ are sought that minimize

[TABLE]

subject to $\mbox{det}(\mathbf{M})=1$. The weights $\pi_{n}$ are introduced to handle multiple-level scores: the user, during the interaction, may decide that some of the positive examples are more significant than others, and assign to them correspondingly higher weights $\pi_{n}$.

The problem can be solved separately for $q$ and $\mathbf{M}$. The optimal $q$, independently of $\mathbf{M}$, is the weighted average of positive samples

[TABLE]

with $u_{n}\in{\mathbb{R}}^{M}$.

In order to find the optimal $M$, define the weighted correlation matrix $\mathbf{C}=\{c_{pv}\}$ as

[TABLE]

It can be shown that $C=\lambda{\mathbf{M}^{-1}}$ and $\mbox{det}(\mathbf{C})=\lambda^{M}\mbox{det}(\mathbf{M}^{-1})=\lambda^{M}$, where $\lambda$ is the parameter of the Lagrangian optimization. The optimal $\mathbf{M}$ is [26]:

[TABLE]

Note that the matrix $\mathbf{C}$ depends on the query point. The optimal solution is obtained when $C$ is computed with respect to the optimal query point (11).

2.3 Dimensionality Problems

The adaptation of the metric works well when the number of positive examples is at least of the same order of magnitude as the dimensionality of the feature space. A good example of this is given by MindReader. The affine matrix $\mathbf{M}$ can be determined using (13) only if $\mathbf{C}$ is non-singular, and this is not the case whenever $N<M$. This is an important case in image search, as $N$ is in general of the order of 10 or, in very special cases, of 100 images, while $M$ may easily be of the order of 10.000. When $N<M$ the inverse $\mathbf{C}^{-1}$ doesn’t exist, and [26] uses in its stead a pseudo-inverse based on singular value decomposition. $\mathbf{C}$ being symmetric, it can be decomposed as

[TABLE]

where $\mathbf{U}$ is an $M\times{M}$ orthogonal matrix and

[TABLE]

where $R\leq{N}$ is the rank of $\mathbf{C}$. The pseudo-inverse of $\mathbf{\Sigma}$ is defined as

[TABLE]

and that of $\mathbf{C}$ as

[TABLE]

The metric matrix $\mathbf{M}$ is then defined as $\mathbf{M}=\alpha{\mathbf{C}^{+}}$, where $\alpha$ is chosen in such a way that $\mbox{det}(\mathbf{M})=1$.

The whole procedure depends only on $R<M$ parameters. To see what this entails, consider that $\mathbf{U}$ is orthogonal, so the transformation $x\mapsto{\mathbf{U}x}$ is an isometry. In this coördinate system, $\mathbf{C}^{+}=\mathbf{\Sigma}^{+}$ and \mathbf{M}^{+}=\mathbf{\Sigma}^{+}=\mbox{diag}\bigl{(}\sigma_{1},\ldots,\sigma_{R},0,\ldots,0). That is, the distance depends on the value of only $R$ axes, a small fraction of those of the feature space.

In the general case, this situation is unavoidable: the matrix $\mathbf{C}$ has $M(M-1)/2$ coefficients, and we only have $MN$ coefficients of the positive samples that we can use to estimate them. In image data bases, feature spaces have high dimensionality, and $M$ can be of the order of $10^{4}$. The number of selected images is limited by practical consideration to an order $10$. So, we are trying to estimate $\sim 10^{8}$ coefficients using $\sim 10^{5}$ samples—an obviously under-determined problem.

A system like MARS, on the other hand, only requires the estimation of $M$ coefficients, making the estimation using the $MN$ feature values of the positive example stable even for reasonably low values of $N$. The price one pays is that the MARS metric matrix can weight the different axes of the feature space but it can’t rotate them, preventing the method from exploiting statistical regularities on axes other than the coördinate ones.

In order to alleviate this problem, Rui and Huang [39] propose dividing the feature space into $W$ separate components, and determine the distance between an item and the query by first computing thee distance for each component and then combining the distances thus obtained.

More precisely, consider the feature space ${\mathbb{F}}$ as the Cartesian product of $W$ manifolds: ${\mathbb{F}}\sim{\mathbb{F}}^{1}\times\cdots\times{\mathbb{F}}^{W}$ and let $M_{\nu}$ be the dimensionality of the manifold ${\mathbb{F}}^{\nu}$. In the following, the indices $\nu,\mu$ will span $1,\ldots,W$, while $p,v$ will span $1,\ldots,M_{\nu}$111The construction that we are presenting could be called ”top-down”: we have an overall feature space and we break it down into smaller, mutually orthogonal pieces. A different, ”bottom-up”, point of view would simply ignore the overall feature space ${\mathbb{F}}$ and consider that our items are described by $W$ feature vectors ${\mathbb{F}}^{1},\ldots,{\mathbb{F}}^{W}$. We compute distances separately in these spaces and then stitch them together. The top-down point of view has the advantage of highlighting certain limitations of this decomposition. We are assuming here that the ${\mathbb{F}}^{k}$ are independent of each other, and that the corresponding feature vectors can vary freely and independently. However, if ${\mathbb{F}}=S^{M-1}$, there is no guarantee that the Cartesian combination of independently varying vectors will have unit length, that is, there is no guarantee that ${\mathbb{F}}$ is decomposable in this way. As we shall see, this is not a problem in the model of Rui and Huang, as the model of distance combination is very simple, and ${\mathbb{F}}$ will not (per se) be a metric space–it will be one only qua combination of metric spaces. The problem has to be taken into account, however, for other combination models..

Each item $u_{n}$ will be described by $W$ feature vectors $u_{n\nu}$ with

[TABLE]

In Rui and Huang’s model the user, in addition to selecting the set of $N$ positive samples, ${\mathcal{R}}_{+}$, can give a relevance $\pi>0$ to each one of them. Relevance is modeled as a vector $\pi=[\pi_{1},\ldots,\pi_{N}]^{\prime}$. The overall distance between an item $u_{n}$ and the query $q$ is the weighted sum of the distances, in the component spaces ${\mathbb{F}}^{\nu}$, between the $\nu$th feature vector of $u_{n}$ and the projection of $q$ on ${\mathbb{F}}^{\nu}$, $q_{\nu}$. That is, if $w=[w^{1},\ldots,w^{W}]$ is the weight vector, then

[TABLE]

where

[TABLE]

where $\mathbf{M}^{\nu}$ is the symmetric $M_{\nu}\times M_{\nu}$-dimensional metric matrix of ${\mathbb{F}}^{\nu}$. With these definitions the metric optimization problem is the following:

[TABLE]

where

[TABLE]

subject to

[TABLE]

Defining $\mathbf{R}^{\nu}$ as the $M_{\nu}\times N$ matrix whose $n$th column is

[TABLE]

we have the optimal query point

[TABLE]

which is the one given by Rocchio’s algorithm without negative examples applied to the $\nu$th feature.

The matrices $\mathbf{M}^{\nu}$ are determined as in the previous section, where all the quantities are now limited now to feature $\nu$. The solution is similar: defining the matrix $\mathbf{C}^{\nu}=\{c_{pv}^{\nu}\}$ as

[TABLE]

we have

[TABLE]

The optimal weight vector is given by

[TABLE]

where $a^{\nu}=\sum_{n}\pi_{n}g_{n\nu}$ is the weighted average of the distance between the positive samples and the query in the subspace that defines feature $\nu$.

Once the axes have ben rotated, we find here the same general idea that we find in MARS: the subspace in which the positive samples are far away from the query are less informative, and will receive a smaller weight; the average of the square distance along a direction is related to the variance of the coördinates along that direction, viz. to (7).

*          *          *

In this section we have limited our considerations to a handful of systems that are needed to provide the background for our discussion. We have, in other words, preferred depth of analysis to breadth of coverage. We should like, however, to give the briefest of mentions to a few examples more, as a recognition to the pervasiveness of these techniques.

In the introduction, in (1) we have mentioned that the original query $q^{0}$ is kept fixed and contributes to the expression of $q^{t}$. The extent to which this is done was left unspecified, as will be in the rest of the paper. The problem is analyzed in [31].

Some of the ideas presented in [38] have been extended in [40], while ideas along the lines of the semantic spaces presented below were presented in [30].

This paper focuses on the use of relevance feedback in image search but, of course, the general ideas have been applied to many areas, from information retrieval in collection of documents [22, 5], web systems [47] and other heterogeneous data [28]. Relevance feedback is present in a number of methods and algorithms; in [14], user feedback is used in order to set system parameters, in [34] in order to understand user behavior in faceted searches. User expectation and feedback has also been used in order to measure the effeciveness of systems [33].

For more general information, the reader should consult the many excellent books [1, 6, 36, 10, 32] and reviews [4, 41, 49, 35] on Information Retrieval. Information on the application of Relevance Feedback to image retrieval can be found in general texts and review on this areas [42, 45, 27, 15, 17].

3 Semantic Spaces

Rui & Huang try to solve the problems deriving from the extremely high dimensionality of the feature space by breaking the space in a two-level hierarchy. At the lower level, each feature defines a separate space, upon which one operates as in the usual case and in which one computes a distance from the query. At the higher level, these distances are linearly combined to provide the final distance. One useful point of view, one that isn’t explored in Rui & Huang’s paper, is to consider the latter as a higher order feature space, one in which the coördinates of the images are given by their distances from the query in each of the low level spaces. Since all coördinates are positive (and therefore equal to their absolute value), we can consider their linear combination as a weighted $L_{1}$ distance. Note that this choice of $L_{1}$ distance is essential to Rui & Huang’s method, since the combination function of the different feature spaces must be linear. Consequently, it is impossible to use such method to endow the high level space with any other metric.

In addition to this, Rui & Huang’s method doesn’t always succeeds in reducing the size of the problem to a manageable size, as the matrices $\mathbf{M}^{\nu}$ can still be very large. Figure 3 shows the pre-computed features of the SUN data set [46] together with their size.

The metric matrix for the whole feature space (of size $25,751$) contains a whopping $10^{9}$ coefficients. Breaking up the feature space using Rui & Huang’s method alleviates the dimensionality problem, but doesn’t quite solve it, as the total size of the matrices $\mathbf{M}^{\nu}$ is $10^{8}$ coefficients: an order-of-magnitude improvement but, still, a problem too large for many applications.

3.1 Semantic partition

In this section, we shall take the essential idea of Rui & Huang, but we shall apply it to the point of view that we just expressed, that is, to define a higher order feature space. As in Rui & Huang’s work, we shall assume that the total feature space is the Cartesian composition of $W$ feature spaces

[TABLE]

but, in this case, we shall assume that the spaces ${\mathbb{F}}_{1},\ldots,{\mathbb{F}}_{W}$ are not simply a partition dictated by technical matters, that is, they don’t simply segregate the components of different features, but have semantic relevance: the spaces ${\mathbb{F}}^{\nu}$ must somehow correlate with the semantic characteristics that a person would use when doing a relevance judgment [23, 2].

The division of the feature space into sub-spaces entails a semantic choice of the designer, as each subspace should correlate with an aspect of the ”meaning” of the image.

How are these groups to be selected? The general idea is that they be ”meaningful”, and meaning is often assigned through language [20, 7, 19, 24, 43]. The psychology of feedback selection is still somewhat unexplored, but it is certain that if we ask somebody why he chose certain images as positive example, we shall receive a linguistic answer [16]. For lack of a better theory, we can assume that this answer is a reflection of the perception that made a person choose these images. Consequently, a dimension in the feature space should be something that we can easily describe in words (in practice: in a simple and direct sentence) without making reference to the underlying technical feature.

One possibility, which we shall not analyze in this paper, is that of a prosemantic space, in which each dimension in the reduced dimensionality space is the output of a classifier, trained to recognize a specific category of images [8, 9]. Here, we shall consider the individual feature spaces as given, and use the query to transform each one into a dimension of the semantic space.

3.2 The query space

Assume, according to our model, that we have a feature space defined as the Cartesian composition of $W$ feature spaces, as in (29), and let $h_{\nu}$ be the dimensionality of the $\nu$th of such spaces. Each item $u_{n}$ will be described by the $W$ feature vectors

[TABLE]

The query $q$ will also be defined by $W$ vectors

[TABLE]

Each feature space ${\mathbb{F}}^{\nu}$ is a metric space, endowed with a distance function $d_{\nu}$. We consider the distance between the $\nu$th feature of item $u_{n}$ and the $\nu$th component of the query $q$ as the coördinate of $u_{n}$ along the $\nu$th dimension of the query space, that is, we represent $u_{n}$ with the $W$-dimensional feature vector

[TABLE]

The space of these vectors is the query space ${\mathbb{Q}}$ of dimensionality $W$. The query space itself can be given a metric structure defining a distance function in it. If the distance is a weighted Minkowski distance, we have

[TABLE]

Note that in this space the query is always the origin of the coördinate system, so that the score of an image is a function of its distance from the origin.

In this space, all the coördinates are positive and, depending on the characteristics of the distance functions, they can be bounded. In later sections, we shall use suitable probability densities to model the distribution of images in this space. One reasonable model for many situation in which the coördinates are positive is the logonormal distribution, that is, a normal distribution of the logarithm of the coördinates [12, 18]. To this end, sometimes we shall use the transformed query space $\bar{{\mathbb{Q}}}$, in which the coördinates of the $n$th image are

[TABLE]

The distance in this space is defined as in (33).

Several arguments have been brought forth to argue that spaces of this kind are more ”semantic” than normal feature spaces, in the sense that they correlate better with the linguistic descriptions of images or, at the very least, they are more amenable to a linguistic description than the usual feature spaces.

We shall appropriate these arguments and assume that the query space is the most suitable space in which relevance feedback should be implemented in the sense that, its reduced dimensionality notwithstanding, the query space contains the essential (semantic) information on which Relevance Feedback is based. In our tests section we shall validate this assumption by comparing the performance of the MARS algorithm in the query space with that of the same algorithm in the feature space.

Note that in all our tests the query vectors $q_{\nu}$ are obtained by applying Rocchio’s algorithm to the individual feature spaces. For this reason, it is not possible to implement Rocchio’s algorithm in the query space, as the algorithm is necessary in order to build it.

4 Riemann Relevance Feedback

Relevance feedback begins by placing a number of positive examples in a metric space which, in this case, is the query space of the previous section. We can consider these images as samples from a probability distribution that determines the probability that a certain region of the spaces contain semantically interesting images.

To be more precise, consider the problem of using relevance feedback to identify a target image $u$ in the query space, and let $p$ be a probability density on ${\mathbb{F}}$. Then $p$ models the semantics of $u$ if, given a volume $\Delta{V}$ around a point $x\in{\mathbb{F}}$, the probability that $u\in\Delta{V}$ is $p(x)\Delta{V}$.

The idea of our method is to use this distribution to model the feedback process as a deformation in the metric of the query space. In particular, we shall use this distribution to determine a Riemann metric in ${\mathbb{F}}$ such that images that differ in a significant area of the space will be fairly different, while images that differ in a non-significant area of the space won’t be as different. To clarify things, consider a one-dimensional query space and a distribution like that of figure 4.

Qualitatively, the area $\Delta$ is the ”interesting” area for the user, the area where most of the relevant examples are found. Two images placed in this area will be equally relevant, that is, the distance between them will be small. On the other hand, two images placed in the area $\Delta^{\prime}$ will not be affected by relevance feedback, and the distance between them will be given by the normal Euclidean distance. Note that in this section we are assuming a unimodal distribution; we shall consider a more general case in the next section.

So, given the same difference in the coördinates of two points, their distance will be small in the area of high density, and will be (approximately) Euclidean where the density is close to zero. Consider the elementary distance element in a given position $x$ of the axis. We can write it as

[TABLE]

In a uniform Euclidean space, $g(x)\equiv{1}$. In the space that we have devised, $g(x)$ should have a behavior qualitatively similar to that of figure 5.

Let us now apply these considerations to our relevance feedback problem. We have obtained, from the user, a set of $N$ positive examples, each one being a vector in ${\mathbb{F}}_{n}\equiv{\mathbb{R}}^{W}$:

[TABLE]

We arrange them into a matrix:

[TABLE]

This matrix is a sample from our unknown probability distribution. If we assume that we are in the transformed feature space (34), we can model the unimodal distribution as a Gaussian

[TABLE]

where $\mu$ and $\Sigma$ are the sampled average and covariance. For the sake of simplicity, we translate the coördinate system so that $\mu=0$. We model the space as a Riemann space in which the distance elements at position $x$ for a displacement $dx=[dx_{1},\ldots,dx_{W}]^{\prime}$ of the coördinate is222In differential geometry it is customary to apply Einstein’s summation convention: whenever an index appears twice in a monomial expression, once as a contravariant index (viz. as a superscript) and once as a covariant index (viz. as a subscript), a summation over that index is implied. The components of the differentials $dx$ are contravariant, while $g$ is a doubly covariant tensor. The distance element would therefore be written as

$ds^{2}=g_{\nu\mu}(x)dx^{\nu}dx^{\mu}$

This convention is not common in Computer Science and, for the sake of clarity, we shall not follow it.

[TABLE]

Based on our qualitative considerations, we shall have

[TABLE]

with $0\leq\alpha\leq{1}$. The factor $\alpha$ is necessary in order to avoid that the Riemann tensor become degenerate in 0, and its necessity will be apparent in the following.

Working in a space with this Riemann tensor is a very complex problem, but it can be simplified if, before we define the tensor $g$, we decouple the directions making them (approximately) independent. We apply singular value decomposition to write $\mathbf{T}$ as

[TABLE]

then, if we represent the images in the rotated coördinate system $Y$, where, for image $n$,

[TABLE]

the covariance matrix is diagonal. Consequently, the Riemann tensor will also be diagonal:

[TABLE]

with

[TABLE]

The distance between two points in a Riemann space is given by the length of the geodesic that joins them, the geodesic being a curve of minimal length between two points (in an Euclidean space geodesics are straight lines, on a sphere they are maximal circles, and so on). Let $\gamma(t)$ a geodesic curve in the query space parameterized by $t\in{\mathbb{R}}$. Then, its coördinate expressions $[\gamma_{1}(t),\ldots,\gamma_{W}(t)]$ satisfy

[TABLE]

(as customary, the dot indicates a derivative), where $\Gamma_{\nu\mu}^{\zeta}$ are the Christoffel symbols

[TABLE]

and $g^{\nu\mu}$ are the components of the inverse of $g_{\nu\mu}$. In our case, the only non-zero symbols are

[TABLE]

The geodesic is therefore the solution of

[TABLE]

Define the auxiliary variables $u_{\zeta}(\gamma)=\dot{\gamma}_{\zeta}$. Then

[TABLE]

With this change of variable we have

[TABLE]

or

[TABLE]

Defining $\beta_{\zeta}=\left(\gamma_{\zeta}/\sigma_{\zeta}\right)^{2}$ we have

[TABLE]

and defining $\theta_{\zeta}=\alpha\exp(-\beta_{\zeta})$ we obtain

[TABLE]

Integration yields

[TABLE]

where $C_{\zeta}$ is a constant, that is

[TABLE]

Rolling back the variable changes, we have

[TABLE]

The constants $C_{\zeta}$ determine the direction of the geodesic. Let $[\tau_{1},\ldots,\tau_{W}]$ be the tangent vector that we want for the geodesic in 0, then

[TABLE]

Note that if $\alpha\rightarrow{1}$ the geodesic degenerates, as $\dot{\gamma}(0)\rightarrow\infty$ (this is the reason why we introduced the constant $\alpha$). If $0<\alpha<1$ we can choose

[TABLE]

This leads to

[TABLE]

that is

[TABLE]

Defining the function

[TABLE]

we have

[TABLE]

These equations define (implicitly) the geodesic

[TABLE]

The geodesics are curves of constant velocity and, in this case, we have

[TABLE]

Let $\beta(t)$ be any curve such that $\beta(t_{0})=y_{0}$ and $\beta(t_{1})=y_{1}$. The length of the segment $y_{0}\!-\!y_{1}$ of the curve $\beta$ is

[TABLE]

Given an image in $y$-coördinates (42)—$y=[y_{1},\ldots,y_{W}]^{\prime}$—its distance from the origin (remember that in the query space the query is always placed at the origin) is the length of a segment of geodesic that joins the origin with the point $y$. All geodesics of the form (62) go through the origin, so we only have to find one that, for a given $t_{y}$, has $\gamma(t_{y})=y$. We can take, without loss of generality, $t_{y}=1$: any geodesics through $y$ can be re-parameterized so that $\gamma(1)=y$. That is, we must have

[TABLE]

which entails an initial velocity vector

[TABLE]

Since the geodesics are of constant speed, and due to the parameterization that we have chosen, we have

[TABLE]

where

[TABLE]

therefore

[TABLE]

This is the distance function that we shall use to re-score the data base in response to the feedback of the user.

The computation of the function $\Xi$ entails the calculation of the integral in (61) which, for want of a closed form solution, must be integrated numerically. Fortunately, the integrand is well-behaved, and the integral can be approximated with a linear interpolation on a non-uniform grid.

5 Relevance feedback with latent variables

In Information Retrieval one common and useful way to model sets of documents is through the use of latent variables, probabilistically related to the observed data, resulting in a method known as Probabilistic Latent Semantic Analysis [25]. This method builds a semantic, low-dimensional representation of the data based on a collection of binary stochastic variables that are assumed to model the different aspects or topics [3, 48] of the data in which one might be interested. It should be noted that these topics are assigned no a priori linguistic characterization: they are simply binary variables whose significance is statistical, deriving from the analysis of the data. It has nevertheless been observed that they often do correlate with linguistic concepts in the data.

In information retrieval, we have a collection of documents ${\mathcal{D}}=\{d_{1},\ldots,d_{D}\}$ and a collection of words ${\mathcal{W}}=\{w_{1},\ldots,w_{W}\}$. The observation $X$ is a set of pairs, $X=\{(d_{n},w_{\nu})\}$, where $d_{n}$ is a document and $w_{\nu}$ a word that appears in it. From the observations we can estimate $P(w_{n},w_{\nu})$, that is, the probability that document $d_{n}$ and word $w_{\nu}$ be randomly selected from the corpus.

The model associates an unobserved variable $z_{k}\in{\mathcal{Z}}=\{z_{1},\ldots,z_{K}\}$ to each observation $(d_{n},w_{\nu})$. The unobserved variables $z_{k}$ are assumed to represent topics present in the collection of documents. Let $P(d)$ be the probability that a document $d$ be selected, $P(z|d)$ the probability that variable $z$ be active for $d$ (viz., the probability that document $d$ be about topic $z$), and $P(w|z)$ the class-conditioned probability of a word $w$ given $z$ (viz., the probability that the topic $z$ produce word $w$). Using these probabilities, we define a generative model for the pair $(d,w)$ as follows (see also Figure 6):

i)

select a document $d$ with probability $P(d)$;

ii)

pick a latent variable $z$ with probability $P(z|d)$;

iii)

generate a word $w$ with probability $P(w|z)$.

The model can be used to predict the observation probabilities of a pair $(d,w)$:

[TABLE]

This model is asymmetric in $d$ and $w$, and undesirable characteristic. One can use Bayes’s theorem,

[TABLE]

to write

[TABLE]

A model that is symmetric in $d$ and $w$ and whose interpretation is (Figure 7):

i)

select a topic $z$ with probability $P(z)$;

ii)

generate a document $d$ containing that topic with probability $P(d|z)$

iii)

generate a word $w$ associated to the topic with probability $P(w|z)$.

The parameters of this model are $P(z_{k})$, $P(d_{n}|z_{k})$, and $P(w_{\nu}|z_{k})$, where $n=1,\ldots,D$, $k=1,\ldots,K$, and $\nu=1,\ldots,W$. The probability distributions depend in general on some parameters $\theta$, which we have to determine. We do this by maximizing the log-probability of the observations:

[TABLE]

If we had the complete observations (viz., if we had observed $X$ and the latent variables $Z$ and had therefore triplets $(d_{a}i,w_{\nu},z_{k})$), the maximization, especially in the case of exponential and Gaussian distribution, would be relatively easy. Unfortunately, we do not observe $Z$, but only pairs $(d_{n},w_{\nu})$. With only these data, we have to estimate, in addition to $\theta$, the unknown parameters $P(z_{k})$, $P(w_{\nu}|z_{k})$, and $P(d_{n}|z_{k})$, making the problem much harder. A common algorithm for solving this estimation problem is Expectation Maximization (henceforth: EM). Define the function $Q(\theta,\theta^{0})$ as

[TABLE]

Then $\theta^{0}$ and the other unknown parameters are initialized to suitable random values and the following iteration is applied

E:

compute $P(Z|X,\theta^{0})$

M:

$\displaystyle\theta^{0}\leftarrow\mbox{arg}\max_{\theta}Q(\theta,\theta^{0})$

The first step (Expectation) uses the previous parameters to find the values $P(Z|X,\theta^{0})$ that determine the expected value necessary to compute $Q$, the second (Maximization) uses the probabilities computed in E to determine new parameters $\theta$ by maximizing $Q$. It can be shown that this leads to the maximization of $Q(\theta,\theta)$ and this, in turn, of (74); for the details, we refer the reader to [13].

*          *          *

In order to apply this method to our problem, we consider again the semantic feature space ${\mathbb{F}}\equiv{\mathbb{F}}^{1}\times\dots\times{\mathbb{F}}^{W}$. We consider the space ${\mathbb{F}}^{\nu}$ as the space that defines the $\nu$th visual word $\Phi^{\nu}$. Assume that each space ${\mathbb{F}}^{\nu}$ is endowed with a probability density $p_{\nu}$. The feedback provides us with $N$ observations, $X=\{x_{1},\ldots,x_{N}\}$, each of them being a tuple:

[TABLE]

with $u_{n\nu}\in{\mathbb{F}}^{\nu}\equiv{\mathbb{R}}^{M_{\nu}}$ being the $\nu$th feature of the item $d_{n}\in{\mathcal{D}}$. The value $p_{\nu}(u_{n\nu})$ is the probability (density) that $x_{n}$ express the word $\Phi^{\nu}$. Note that word expression is continuous: each observation contains all the words, a word in a volume $dV$ around $\Phi^{\nu}$ being expressed with a probability $p_{\nu}(u_{n\nu})dV$, while in the discrete case the probability that any word be expressed in a given document is either $1$ or [math] (viz., each document contains some of the words). The document identities, $d_{n}$, on the other hand, are discrete (their probabilities are either $1$ or [math]), making our model a mixed one.

We now introduce a vector of $K$ ”topical” latent random variables, $Z=[z_{1},\ldots,z_{K}]$ in which one element is equal to one and all the others are zero, that is,

[TABLE]

There are $K$ possible states for $Z$. We interpret the fact that $z_{k}=1$ as the presence of topic $z_{k}$ in the item that we are considering. Let $\pi_{k}=P\{z_{k}=1\}$,with $0\leq\pi_{k}\leq{1}$ and $\sum_{k}\pi_{k}=1$. Each topic is associated with a probability distribution in each word space, $p_{k\nu}=p_{\nu}(u|z_{\nu}=1)$, which gives the probability of observing the word $\Phi^{\nu}$ when topic $z_{k}$ is expressed. We assume that there distributions are Gaussians333Given that we have a mixed model, we have to work both with probability distribution (for the discrete variables) and densities (for the continuous). We shall use a lowercase letter, $p$, to indicate densities, and an uppercase, $P$, to indicate probabilities. with means $\mu_{k\nu}\in{\mathbb{R}}^{M_{\nu}}$ and covariance matrices $\Sigma_{k\nu}\in{\mathbb{R}}^{M_{\nu}\times{M_{\nu}}}$

[TABLE]

Given an observation $x_{n}$, we have

[TABLE]

The parameters that we have to determine for the model are:

[TABLE]

We determine them by maximizing the log probability (74), which we write, in this case, as

[TABLE]

with $\pi$, $\mu$, $\Sigma$ being structures that collect the respective parameters. In the following, in order to simplify the notation, we shall often omit the conditioning on $\pi$, $\mu$, and $\Sigma$. We can factor $p(x)$ using $Z$ as

[TABLE]

so that

[TABLE]

In order to apply EM, we need to define the function $Q(\theta,\theta^{0})$ as in (75), where $\theta=[\pi,\mu,\Sigma]$. We begin by determining $P(Z|X)$:

[TABLE]

where, in the last equation, we have equality if we add a normalization value so that $\sum_{k}{P(z_{k}|x_{n})}=1$. Set $\gamma_{nk}=P(z_{k}|x_{n})$. Then

[TABLE]

Normalizing we have

[TABLE]

so that

[TABLE]

These values are computed in the E step using the parameters of the previous iteration. For the M step, we need to compute

[TABLE]

We maximize $Q$ with respect to $\pi_{k}$, $P(d_{n}|z_{k})$, $\mu_{k\nu}$, $\Sigma_{k\nu}$, keeping $\gamma_{nk}$ fixed, subject to the conditions

[TABLE]

and

[TABLE]

Also, define

[TABLE]

$N_{k}$ is the ”effective” (viz., weighted by probability) number of items that express topic $z_{k}$. We introduce the Lagrange multipliers $\lambda$, $\zeta_{k}$ and maximize the function

[TABLE]

In order to maximize ${\mathcal{F}}$, we set the derivative with respect to the parameters to zero. For the $\pi_{k}$, we have

[TABLE]

Multiplying by $\pi_{h}$, summing over $h$ and using (89), we have

[TABLE]

That is, $\lambda=\sum_{h}N_{h}=N$. Applying this back to (93), we have

[TABLE]

Similarly, for $P(d_{n}|z_{k})$, we have

[TABLE]

Multiplying by $P(d_{n}|z_{h})$, summing and applying (90), we have $\zeta_{k}=N_{k}$ and

[TABLE]

The optimization with respect to $\mu_{k\nu}$ and $\Sigma_{k\nu}$ is unconstrained, and no Lagrange multipliers are present. We have

[TABLE]

Multiplying by $\Sigma_{k\nu}^{-1}$ and rearranging, we have

[TABLE]

and similarly

[TABLE]

The iteration of the EM algorithm is shown in Figure 8. The stopping criterion is either the stabilization of ${\mathcal{L}}$ or of the parameter values.

In order to reduce the number of parameters, we can use the same technique that we have used in section 4: we apply SVD to transform the coördinates in a rotated space in which they are essentially uncorrelated. At the same time, we can keep only the dimensions corresponding to the largest eigenvalues to reduce dimensionality. This is what we called the $Y$-space in section 4. In this section, to avoid complicating further the notation, we shall still indicate the vectors in this space as $u_{n\nu}$, keeping in mind that they are uncorrelated and therefore that

[TABLE]

In this space, the last step of the algorithm is replaced by

[TABLE]

With the application of the EM algorithm, we have obtained the topic probabilities $\pi_{k}$ and, for each topic $z_{k}$ and word $\Phi^{\nu}$, a probability distribution with parameters $\mu_{k\nu}$ and $\sigma^{2}_{k\nu}$.

We now use these parameters and the results of section 4 to endow the feature space ${\mathbb{F}}={\mathbb{F}}^{1}\times\cdots\times{\mathbb{F}}^{W}$ with a suitable metric: the items of the data base closest to the query according to this metric will be those returned by the relevance feedback algorithm.

Assume $z_{k}=1$. Then each ${\mathbb{F}}^{\nu}$ is endowed with a Gaussian distribution. The distance between the feature vector $u_{\nu}$ and the center of the distribution is given by (70):

[TABLE]

The word spaces are orthogonal, so the squares of the distances are additive. The distance between a feature vector $u\in{\mathbb{F}}$ and the origin, given that $z_{k}=1$, is

[TABLE]

To determine the complete distance, we simply average over all $z_{k}$:

[TABLE]

6 Tests

Our experimental method is based on the observation that, in order to be useful, relevance feedback must have semantic value, that is, it must correlate positively with the linguistic categorization that people are likely to use when interacting with the data base. We shall consider categories as sets of images and assume that the linguistic categorization on which the taxonomy is based is a representative sample of what a person interacting with a data base might use. Consequently, we shall look for positive correlations between the results of relevance feedback and the taxonomy of our data set.

This observation places us in the middle of the so-called semantic gap [21]: the essential lack of correlation between computable visual features and linguistic categorization. We aim at checking to what extent relevance feedback allows us to bridge this gap.

Relevance feedback is an algorithm, one of the many components of a complete retrieval system. The performance that a user ”sees” depends on all these components, from feature extraction to indexing, to user interface. Here we are not interested in the evaluation of a complete system, nor are we interested in evaluating relevance feedback in the context of a specific system. We are looking for an evaluation as neutral as possible of the algorithms independently of the system in which they work. This consideration problematizes, if not outright excludes, the recourse to user studies, as these are always evaluation of a whole system in a specific context.

In order to evaluate the semantic value of our methods in a system-neutral way, we use a categorized data set and use relevance feedback as an example-based category predictor. Consider a data base ${\mathcal{D}}$ of $D$ images, and a target subset $M\subseteq{{\mathcal{D}}}$ of images that belong to a given linguistic category. To this data base we apply relevance feedback by selecting $r$ images from the set $M$, that is, by assuming that the user has selected $r$ images representative of the desired category. We use these $r$ samples to query the data base using the relevance feedback schemas that we are evaluating and we collect a result list $Q$ containing $q$ elements. We determine the number $k$ of elements of $Q$ that belong to the target set $M$ (that is, $k=|Q\cap{M}|$): this is the number of hits of the method that we are evaluating for that trial. If we repeat the test for constant value of $r$ and different sets $M$, the average value of $k$, $\bar{k}$ is a measure of the performance of the method for those values of $r$ and $N$. The variables and the parameters for the tests are summarized in table 2

The tests were carried out using test partition 1 of the SUN data set, consisting of 19,850 images, divided in 397 categories of 50 images each [46]. In each trial we use one of these categories, selected at random, as our target set $M$. This entails that in all our tests we have $m=50$. The result set $Q$ should be quite smaller than $M$ to avoid running out of ”good” images: this will allow us the theoretical possibility of a perfect score, that is, of having all results from the target category. In order to have plenty of good images to spare, we set $q=20$444During some preliminary tests, we checked for stability with respect to the choice of $q$; the result is that, unless $q$ is very small or very close to $m$, the results are consistent and independent of its value.. This leaves us with the problem of selecting $D$, the size of the data base. We determine $D$ so as to be able to compare the algorithms with the results obtained by random selection, so that we can use random selection as a control group. We show in Appendix A that if we have a set ${\mathcal{D}}$ of $D$ elements, a target set $M$ with $m$ elements and a set $Q$ with $q$ elements chosen at random from $D$ ($q\leq{m}$), the average size of $Q\cap{M}$ is

[TABLE]

So we can choose the desired values of $\bar{k}$ and use (106) to derive $D$.

We carry out tests with $\bar{k}=[0.1,0.5,1,5,10]$, which correspond to $D=[10,000;\,2,000;\,1,000;\,200;\,100]$. A control test will be carried out to check out when is the number of images returned by the various methods significantly better than the random average.

Finally, we chose the set of images in which retrieval is based: the set ${\mathcal{R}}$ is drawn at random from the set $M$ of target images and its size is chosen to cover a wide range of values: $r=[2,5,10,20,30]$.

All the tests are repeated 20 times, each time with a different target set $M$ and feedback set ${\mathcal{R}}$ picked at random. The comparison with the random selection was done determining the variance of the results and using ANOVA ($p<0.01$). For the comparison between the methods, the data were generated at random but for each trial all the methods were executed with the same data. This allowed us to determine significance using a simple signs rule [11], which has the advantage of not resting on an assumption of normality.

We represent the images using 10 out of the 12 standard features of Table 3. We do not consider the two largest features, as their size makes them unmanageable for the Rui & Huang algorithm. Between eliminating two features and giving up comparison with Rui & Huang, we have chosen the first option. It should be noticed, however, that even so the Rui & Huang algorithm took a time at least an order of magnitude larger than that of the other methods, up to ten minutes per trial on a standard personal computer when implemented using MATLAB555This long execution time is not a consequence of our implementation: virtually the whole time is spent in computing SVD, which is very efficient in MATLAB..

A treatment consists in a given average number of random hits ($\bar{k}=10,5,1,0.5,0.1$) and a given number of relevance feedback Images ($r=2,5,10,20,30$), which are chosen randomly in the target category. For each treatment we report in Figure 9 the average number of hits and its variance. Numbers in boldface correspond to statistically significant differences ($p<0.01$).

All methods, with the exception of the straight Rocchio algorithm, are significantly better than random in most cases. Apart from this and the Rui & Huang method with $\bar{k}=0.1$, there are only five cases in which the methods are not significantly better than random, all happening when $r=2$. This is hardly surprising: relevance feedback works on information from user’s inputs, and with $r=2$ there is very little information to work with. Moreover, we must take into account that the feedback images are taken at random from a linguistic category (the target); some images are good informative examples of the visual content of their categories, others are poor examples and using them for feedback is more deceiving than helpful. With $r=2$ there are not enough samples to ”average out” the effects of poor choices. This is confirmed by the generally high variance that we obtain for $r=2$. In view of these observations, we shall not use the case $r=2$ when comparing the methods, nor shall we compare with Rocchio’s algorithm.

Figure 10 contains the results of the comparison of five methods (we do not consider Rocchio here). The numerical values are the differences between the average number of hits (for given $\bar{k}$ and $r$) of the method in the column and that for the method in the row. Positive values mean that the method in the column performs better than that in the row; boldface numbers indicate statistically significant differences. So, for example, the boldface 5.20 in the top left corner of the ”Riemann/Rocchio” intersection indicates that for $\bar{k}=10$ and $r=5$, the Riemann method shows on average 5.20 images from the target set more than Rocchio’s algorithm, and that the difference is statistically significant.

From these results, we can draw a series of general conclusions.

i)

All the methods do work to some degree, that is, they perform almost always significantly better than chance. One exception is Rocchio’s algorithm, which is indistinguishable from chance for small target categories ($\bar{k}=0.1$) or large feedback sets ($r=30$). The reasons for this poor performance in the latter case is not clear: it appears that the excess of input information confuses the algorithm, but further analysis would be needed, which is beyond the scope of this paper.

ii)

The Rui & Huang method performs rather poorly on the largest data base ($\bar{k}=0.1$); the reasons for this behavior are also not clear. The reduced dimensionality space does have enough information, since the other methods that use it (”MARS on ${\mathbb{Q}}$”, ”Riemann” and ”Latent”) do perform well (see points iii) and iv) below).

iii)

The similar performance of ”MARS” and ”MARS on ${\mathbb{Q}}$” shows that the dimensionality reduction of the semantic query space doesn’t result in a loss of information, so that performance is maintained. The advantage of the query space is execution time: ”MARS on ${\mathbb{Q}}$” runs roughly one order of magnitude faster than ”MARS”666Measuring execution time was not part of our experimental design, and we took no particular care to control the noise variables and to separate them from the measured variables. Because of this, we can’t provide quantitative results, but only qualitative observations..

iv)

”MARS” and ”MARS on ${\mathbb{Q}}$” outperform ”Rui & Huang” (which, we remind the reader, is executed on the query space ${\mathbb{Q}}$). Here, again, the reasons are not clear and should be investigated further, but some considerations similar to those in ii) can be made. ”Rui & Huang” has been shown to work well in the original feature space, while our results show that its performance degrades in the query space. MARS, on the other hand, performs similarly in the whole feature space and on the query space. The main difference between MARS and the Rui & Huang algorithm is the rotation of the components of the input space, so it is possible that the degradation of ”Rui & Huang” be due to this rotation. These considerations might point to a characteristic of the space ${\mathbb{Q}}$, namely that its canonical axes are preferential and that rotating ${\mathbb{Q}}$ leads to information loss. The fact that ${\mathbb{Q}}$ is obtained from the feature space by applying non-linear operators, which are, as such, not necessarily rotation invariant, supports this interpretation, but further work is necessary to confirm this characteristic of ${\mathbb{Q}}$.

v)

The ”Riemann” and ”Latent” methods perform better than the others. The difference between the two, as well as with the two versions os MARS, is evident mainly for high values of $r$ and small values of $\bar{k}$. This is not surprising: these models, especially ”Latent”, are quite complex with a relatively high number of free parameters, and need a rather large number of samples to settle on a good solution.

A final consideration might be of use to the designers of information systems. The results that we obtained point at the opportunity of a mixed strategy, depending on the number of positive answers available. If the feedback consists of a few samples (e.g., in the first iterations) then there is not enough information available to justify the use of a complex method. In this case, simpler methods such as ”MARS on ${\mathbb{Q}}$” might be a good choice. As the number of positive examples increases, the additional information can be better exploited by a more complex method with more free parameters, leading to better performance and thereby justifying the larger computational effort of the more complex methods such as ”Riemann” or ”Latent”. In some cases, simple methods might actually get confused by too much information, as witnessed in Figure 9 by the poor performance of Rocchio with $r=30$.

7 Conclusions

In this paper, we have developed two methods for relevance feedback based on the data-directed manipulation of the geometry of suitable feature spaces. The first method assumes a reduced-dimensionality query space, ${\mathbb{Q}}$, models the distribution of positive samples with a Gaussian, and transforms the Gaussian into a Riemann metric of a modified query space. The distance from the query along the geodesics of this space is then used to re-score the data base.

The second method uses latent variables, and divides the query space in a collection of ”visual words”. In each word space, the distribution of samples is modeled as a mixture of Gaussians controlled by the latent variables. The distribution is obtained by applying a modification of the EM algorithm to adapt it to this case, in which the probability distribution is discrete in the set of items and continuous in the word spaces. Once the distribution is obtained, we operate as in the first model to obtain a Riemann metric that is then used to re-score the data base. In this paper, we have used the discrete variables only to model the identity of the images, but the method can easily be extended to model associations between words and images, possible associated to semantic model of short texts [29] or to n-gram models [44], techniques often used in mixed models of multimedia data.

We have developed a system-neutral, semantic-based testing methodology, and we have applied it to compare the performance of up to six different methods. The results that we have obtained led to some guidelines for the design of practical relevance feedback systems. They also hint at a possible information loss when the query space is rotated, possibly due to the non-linearity in the derivation of the space. This analysis could not be pursued here, but it opens up interesting perspectives, certainly worth exploring.

One point that this paper has left open is the use of negative samples. The heterogeneity of criteria that may lead one to mark an image as negative has hitherto hampered their statistical modeling. Latent variables, which naturally model a multiplicity of criteria offer a possibility in this sense, but the amount of negative examples necessary makes this approach problematic.

A relevant contribution of this paper has been the development of the mixed EM algorithm. Space limitations did not allow us to present a detail analysis of this algorithm, in particular we could not prove convergence, but this results can easily be derived from the similar result for the discrete version. The most interesting aspect of this algorithm is its potential applicability as a general representation method in all cases in which discrete and continuous features are used at the same time to characterize images, such as is the case, for example, when an image is represented by visual features (continuous) and associated words (discrete). We believe that this version of EM can provide a powerful tool for these situations. But this, of course, will have to be proved in work yet to come.

Appendix A The average number of hits in the random case

Consider the following problem: we have a set $D$ with $N$ elements. In $D$, we mark a target set $M\subseteq{D}$ with $m$ elements. Now we pick at random $q$ elements of $D$; how many of these elements will come from $M$, on average?

For the sake of convenience, we indicate with $p=N-m$ the number of elements not in $M$. To fix the ideas, imagine an urn or a jar containing $N$ balls; $m$ of these balls are black, the rest ($p$ of them) are white. We extract $q$ balls at random: what is the probability that $k$ of these balls will be black?

The balls will come out, white or colored, in a specific sequence and, in principle, different sequences have different probability of occurring. To see how things can be worked out, let us consider an example. Let $q=7$, $m=4$, and assume that we extract the balls in the following order:

When we extract the first ball, the probability of it being white is $p_{1}=p/N$. Now we extract the second: there are $m$ black balls and $N-1$ balls remaining in the urn, so the probability of the second ball being black is $p_{2}=m/(N-1)$; for the third ball, there are $m-1$ black balls left out of $N-2$, so $p_{3}=(m-1)/(N-2)$. Similarly

[TABLE]

that is, the probability of extracting this particular sequence of white and black balls is

[TABLE]

we can rearrange the term as

[TABLE]

It is evident that any sequence of extractions can be rearranged in this way, which leads us to conclude that the probability of extracting the $k$ black balls in a specific sequence is independent of the specific sequence, and equal to the last equation in (109).

The number of possible combinations is $\binom{q}{k}$, so the probability of extracting $k$ marked elements out of $q$ is

[TABLE]

The average value that we are looking for is therefore

[TABLE]

Theorem A.1**.**

[TABLE]

Proof.

We prove the result by induction over $q$.

For $q=1$ the only possible values of $k$ are [math] and $1$, therefore

[TABLE]

For the general case, we express $p[k,q]$ in terms of $p[k-1,q-1]$ as

[TABLE]

With this definition we can write

[TABLE]

We have

[TABLE]

Therefore

[TABLE]

∎