Criteria Sliders: Learning Continuous Database Criteria via Interactive Ranking

TL;DR

This paper introduces a method for learning low-dimensional continuous criteria in large databases through interactive ranking, enabling users to efficiently organize thousands of data points with minimal labeling effort.

Contribution

It presents a novel semi-supervised, active learning approach for interactive ranking that simplifies organizing large datasets along continuous criteria.

Findings

Effective organization of thousands of data points using 1D and 2D sliders.

Active data point suggestion improves ranking efficiency.

Applicable to image and geometry datasets.

Abstract

Large databases are often organized by hand-labeled metadata, or criteria, which are expensive to collect. We can use unsupervised learning to model database variation, but these models are often high dimensional, complex to parameterize, or require expert knowledge. We learn low-dimensional continuous criteria via interactive ranking, so that the novice user need only describe the relative ordering of examples. This is formed as semi-supervised label propagation in which we maximize the information gained from a limited number of examples. Further, we actively suggest data points to the user to rank in a more informative way than existing work. Our efficient approach allows users to interactively organize thousands of data points along 1D and 2D continuous sliders. We experiment with datasets of imagery and geometry to demonstrate that our tool is useful for quickly assessing and organizing the content of large databases.

Equations

$$E(\mathbf{f})=(\mathbf{f}-\mathbf{y}{)}^{\top }L(\mathbf{f}-\mathbf{y})+\lambda {\mathbf{f}}^{\top }H\mathbf{f},$$

$${\mathbf{f}}^{\ast }=(L+\lambda H{)}^{-1}L\mathbf{y}.$$

$$-\log p(\mathbf{f}|\mathbf{y})=(\mathbf{m}-\mathbf{f}{)}^{\top }{C}^{-1}(\mathbf{m}-\mathbf{f})+Z,$$

$$\mathcal{I}(X_i)=\sum _{j=1,\dots ,u}[C(t-1)-C(t{)}^i{]}_{jj}.$$

$$\text{diag}[C(t{)}^i]=\text{diag}[C(t-1)]-\frac{\text{squ}[C(t-1{)}_{[:,i]}]}{1+\text{diag}[C(t-1){]}_i},$$

$$C^{-1}$$

$$C\approx \overline{C}=\sum _{i=1}^r\frac{1}{{\lambda }_i}{\mathbf{e}}_i{\mathbf{e}}_i^{\top }=E{\Lambda }^{-1}{E}^{\top },$$

$$C(1{)}^i$$

$$=E\left({\Lambda }^{-1}-\frac{{\Lambda }^{-1}{E}_{[i,:]}^{\top }{E}_{[i,:]}{\Lambda }^{-1}}{1+H_{ii}^{-1}}\right)E^{\top }.$$

$$\text{diag}[C-C^i{]}_k$$

$$=\left(\frac{1}{1+H_{ii}^{-1}}\right)\left(E_{[k,:]}{\Lambda }^{-1}{E}_{[i,:]}^{\top }{E}_{[i,:]}{\Lambda }^{-1}{E}_{[k,:]}^{\top }\right).$$

$$[E{F}_i{E}^{\top }{]}_{k,k}$$

$${\mathbf{r}}_i=\left(\frac{1}{1+{\overline{C}}_{ii}^{-1}}\right)E_{[i,:]}{\Lambda }^{-1}.$$

$$f=\frac{Var(i|S)}{Var(i|U)},$$

$${\mathcal{I}}^{\prime }(X_i)=\frac{\text{diag}[C(t-1){]}_i}{\text{diag}[U(t-1){]}_i},$$

Full text

1 Introduction

Database organization often requires laborious metadata annotation. For visual data, computer vision helps automatically model the database with high-dimensional 2D image and 3D shape features. From these, we parameterize criteria for easy organization by embedding the high-dimensional representations into a low-dimensional space, e.g., for face expression, ascribing ‘amount of smile’ from the high-dimensional features. This task is complicated, because most desired criteria do not map to individual features, and instead map to complicated paths lying on a manifold in the high-dimensional feature space. As such, this parameterization is accomplished either by an expert, or by supervised learning from labeled examples [27, 9].

It is hard to survey a visual database of even a few thousand items, and even an expert would need time to assess the database for parameterization. This is especially time consuming for new databases, e.g., scraping the Internet and recovering unknown contents. Even after parameterization, organization tools are still restricted to only the expert-defined criteria, with no easy way for users to define new criteria for their interests, especially if they are abstract or cross typical boundaries. The ability to interactively parameterize a database is needed: to quickly describe database variation without prior knowledge or labels; to intuitively discover low-dimensional criteria from high-dimensional models.

We present an interactive system to generate continuous criteria from high-dimensional models. Interactive labeling requires efficient computation, and ideally extracts the most information from the fewest user-provided labels. Thus, we adapt a state-of-the-art semi-supervised learning algorithm for interactive use, to exploit the rich structure of unlabeled data points. To best exploit limited user interaction, our system actively suggests data points to label such that the information gain is maximized. Further, we propose a new sparse large-scale active learning strategy that enables interactive label suggestion for large datasets.

To simplify parameterization, we ask the user to rank examples. This removes much of the burden when describing potentially abstract criteria on continuous scales. For instance, deciding that item 1 is more than, equal to, or less than item 2 is easier than quantifying that item 1 is 0.2 criteria units away from item 2. From this ranking, our semi-supervised approach generates continuous criteria, which become sliders in our user interface. For 2D criteria, users provide example embeddings directly such that the underlying relative locations are automatically meterized.

Our problem is not conventional ranking for data retrieval applications, where the goal is to classify all data instances that match the given query. Instead, we wish to regress all database items into continuous 1D and 2D criteria. We present two motivating scenarios: 1) For criteria which have no well-defined answer, our system helps users define their opinion. 2) For new visual databases with no metadata, where the alternative may be to laboriously hand collect labels, our system helps to quickly assess and organize the variation within the database.

We show the generality of our approach across databases of images of paintings and faces, and of geometries of human bodies and man-made objects (Fig. 1). We contribute:

The scenario of interactively defining continuous criteria from high-dimensional models, with a prototype implementation (please see our supplemental video). 2. 2.

A maximally-informative efficient semi-supervised active label suggestion algorithm.

2 Related work

Re-ranking.

This related problem takes the results of context- or text-based search and refines the query and/or retrieved result with user interaction. Personalized faceted search [20] exploits relevant meta-data and suggests new keywords to refine the current search. User behavior is modeled probabilistically and tuned to maximize the expected utility of the facet. A rich literature shows the importance of re-ranking in search [20]; however, existing algorithms in this context focus on maximizing search efficiency rather than organizing databases along criteria.

Jain and Varma [18] assume click behaviour relates to the interest query, and use a click count model to predict relevant rankings. Zha et al. [39] proposed a similar approach for visual query suggestion. The COPE system [2] interactively refines search queries by users stating whether the results match their information need, which then weights image features for future searches. These approaches re-rank imagery based on user confirmation, and so they typically focus on the discrete problem of whether the retrieved results is a match [20]. Our goal is to learn criteria outright from sparse user labels and a high-dimensional model.

Rank learning.

Existing rank learning algorithms typically use supervised learning from labeled data points (e.g., rank support vector machines; RankSVMs), whereas in our interactive setting, the user starts with no labels. As such, we exploit information in the unlabeled data with semi-supervised learning [6, 41], which has been used in rank aggregation [8] though not in our rank propagation case. Semi-supervised learning approaches often rely heavily on graph Laplacian regularization, which is not suitable for learning on high-dimensional manifolds [25, 19]. As our data lie in high-dimensional spaces, our approach overcomes this limitation. Szummer and Yilmaz [33] apply the graph Laplacian to the orthogonal problem of learning a preference criteria, and our learning approach could improve this application.

For data retrieval, Parikh and Grauman [27] learn discrete ranking functions via RankSVM from existing user labels. This restricts exploration to known criteria, whereas we discover criteria interactively. Murray et al. [24] presented a database for visual analysis that is characterized by abstract ‘aesthetic’ features, while Caicedo et al. [5] exploited user preference for image enhancement. Reinert et al. [30] use interaction to visually arrange a small image database into an aesthetic overview, which is orthogonal to the efficient exploration that we pursue.

Our interactive criteria definition on high-dimensional data does not compare directly to existing supervised criteria learning systems. CueFlick [13, 1] learns on binary labels, and WhittleSearch [27] attempts to re-rank data along existing criteria rather than generate criteria from scratch. We improve upon WhittleSearch’s underlying RankSVM techniques when adapted to our scenario (Sec. 3.2).

Active learning.

Chen et al. [9] apply active learning to remove inconsistency from existing crowdsourced labels. Their non-interactive approach is approximate in information gain, while our interactive approach is exact given model assumptions. Shen and Lin [31] essentially use RankSVM for bipartite ranking, with active learning based on single point and pair closeness. This is very similar to baseline predictive variance, which may accidentally pick uninformative outliers. Our new measure is unbiased by outliers. Our active learning approach is complementary to human-in-the-loop active learning approaches, e.g., Branson et al. [4]. Their task is to select features for a given data point which minimize class conditional distribution uncertainty (e.g., 20 questions game). Our problem is to suggest data points to label. Fogarty et al. [13] learn image retrieval criteria from binary labels, e.g., outdoor vs. indoor, by iterative refinement of distance measures between data points. Their active label suggestion was extended by Amershi et al. [1] by adopting a Gaussian process model on distance measures. We ask users to provide rank labels, and increase performance over Amershi et al. (Sec. 3.2).

Concept embedding.

Our approach can be interpreted as using interaction to embed data into a high-level concept space. Existing work in this area focuses on category- or cluster-level supervision. Wilber et al. [38] receive triplet constants from users ($(i,j,k)$: object $i$ should be closer to object $j$ than it is to $k$) to learn pair-wise similarity kernels that are used in $t$-SNE-type embedding [11]. We ask users to provide rank labels and emphasize continuous parameterization.

3 Semi-supervised criteria learning

From a large database of images or geometry, we compute offline a high-dimensional vector of 2D and 3D features (Sec. 4). We assume that some combination of these are sufficient to describe the desired user criteria, with our problem being to learn the criteria from a very small number of samples. We use semi-supervised learning to compensate for the lack of labeled data points by exploiting the rich structural information contained within unlabeled data points. Given this preprocess, the user produces a ranking (with equality) for a subset of the dataset items with our interface (Sec. 3.3, and our supplemental video). The labels for these ranked examples are assigned an internal continuous representation ranging from -1.0 to 1.0.

Formally, for the set of data points $\mathcal{X}=\{X_{1},\ldots,X_{u}\}$, plus the corresponding labels for the first $l$ data points, $\mathcal{Y}=\{Y_{1},\ldots,Y_{l}\}\subset\mathbb{R}$, where $l\ll u$, the goal of semi-supervised learning is to infer or propagate to the labels of the remaining $u-l$ data points in $\mathcal{X}$. We adopt the standard energy minimization approach [40, 42, 6]:

[TABLE]

where $L$ is a diagonal label indicator matrix and $\mathbf{y}$ is a vector of continuous label values: $L_{[i,i]}=1$ and $\mathbf{y}_{i}=Y_{i}$ if $i$-th data point is labeled, and $L_{[i,i]}=0$ and $\mathbf{y}_{i}=0$, otherwise. $H$ is the regularization matrix which quantifies how to smooth the propagated labels $\mathbf{f}$ within their local context and the regularization hyper-parameter $\lambda$ balances between the smoothness of $\mathbf{f}$ and the deviation from the labels $\mathbf{y}$. In general, for a symmetric non-negative definite matrix $H$, the energy functional $E$ is convex with respect to $\mathbf{f}$. The solution $\mathbf{f}^{*}$ is then explicitly given as:

[TABLE]

One of the best established regularizers $H$ for semi-supervised learning is the graph Laplacian matrix $S$ [23, 40, 42] which measures pair-wise deviations of $\{\mathbf{f}_{i}\}$ weighted by the similarity of the corresponding inputs. With predominant use of the graph Laplacian regularizer, semi-supervised learning has become established in classification and clustering tasks where the output is a discrete set of labels. However, estimating continuous outputs is still an area of active research due to degenerate behavior of the graph Laplacian regularizer: in high-dimensional spaces, minimizing $E$ with $H=S$ tends to produce an output $\mathbf{f}^{*}$ where most elements are close to zero [25]. This is fine for classification as output magnitudes are irrelevant, e.g., to classify data points into positive and negative classes, $1$ and $1^{-10}$ are both positive; however, having close to zero outputs is meaningless for continuous outputs, as very little regularization occurs.

This problem has been approached with the local Gaussian (LG) regularizer [19], which explicitly addresses this degeneracy and reduces the computational complexity required to remove it. However, since the LG regularizer was developed for general batch semi-supervised learning, i.e., all labeled data points are provided before the learning process, it cannot be directly applied to interactive ranking. We adapt the LG regularizer to our setting, and demonstrate that it is more suitable than the graph Laplacian regularizer for cases with few labels (Sec. 4).

3.1 Interactive ranking and active label suggestion

In interactive ranking, the user iteratively provides training labels until they are happy with the learned criteria. Naïvely using the LG regularizer is computationally expensive for large datasets since, at each iteration, obtaining the propagated labels $\mathbf{f}$ requires minimizing $E$ in Eq. 1, which is order $O(u^{3})$ complexity and requires solving a linear system of size $u\times u$. Further, we wish to aid the user by suggesting data points to label. At iteration $t$, we wish to estimate criteria uncertainty per data point from existing labels, and present the user with more informative samples to label at time $t+1$.

Following the analogy between regularized empirical risk minimization and maximum a posteriori (MAP) estimation [29], and by adopting Bayesian optimization [32], we reformulate $-1\times E$ (Eq. 1) as (the logarithm of) a product of the prior $p(\mathbf{f})$ and a Gaussian noise model $p(\mathbf{y}|\mathbf{f})$. This leads us to assess the minimizer of $E$ as the mean of the predictive distribution (the posterior):

[TABLE]

with mean $\mathbf{m}=CL\mathbf{y}$, covariance matrix $C=(L+\lambda H)^{-1}$, and the normalization constant $Z$. This perspective informs a predictive uncertainty for each data point: The $i$-th diagonal component $C_{ii}$ of the covariance matrix $C$ contains information on the uncertainty of the prediction on label $\mathbf{f}_{i}$, which is typically low when $X_{i}$ is labeled and is high otherwise. Active label suggestion presents results which minimize uncertainty.

This Bayesian reformulation is mathematically rigorous: By the construction of $H$ [19], $C^{-1}$ is positive definite and so $\mathbf{m}$ and $C$ are valid mean vector and covariance matrix. This shapes the posterior $p(\mathbf{f}|\mathbf{y})$ as a sample from an underlying Gaussian process that factorizes to a prior on $\mathbf{f}$ and the corresponding likelihood $p(\mathbf{f}|\mathbf{y})$. This re-interpretation enables us to exploit well-established Bayesian uncertainty modeling. In particular, the variance of the predicted distribution $p(\mathbf{f}|\mathbf{y})$ indicates how uncertain we are about each prediction we make on data points $X\in\mathcal{X}$.

One simple and well-established strategy to exploit these modeled uncertainties for active label selection is to predict at each iteration $t$, and choose point $X_{i}$ with the largest uncertainty. However, Figure 4 shows that naively choosing data points with maximum uncertainty leads to poor performance with a higher error rate than random selection, as isolated outlier data points—which are not broadly informative—receive high variances and are chosen.

Instead, we construct the candidate data points that minimize the predictive variance over the entire set of data points. At each time step $t$, we choose data points with the highest average information gain $\mathcal{I}$, defined as:

[TABLE]

where $\text{diag}[C]$ is a vector containing the diagonal components of the matrix $C$. The matrix $C(t)^{i}$ is constructed by adding 1 to the $i$-th diagonal element of $C(t-1)^{-1}$ and inverting it (i.e., $i$-th data point is regarded as labeled; see Eq. 1). As suggested by the form of the matrix $C$, this score can be calculated without knowing the corresponding label value $Y_{i}$. This is due to the Gaussian noise model; in general, different noise models lead to the predictive variance ($C$) depending on the labels.

Naïvely estimating the information gain for all data points requires quadratic computational complexity: One has to estimate the minimizer of $E(\mathbf{f})$ (Eq. 1), which is $O(u^{3})$ for each data point. However, in our iterative label suggestion scenario, $\mathcal{I}$ can be efficiently computed in linear time: Assuming that $C(t-1)$ is given from the previous step, calculating $\text{diag}[C(t)^{i}]$ does not require inverting the matrix $L(t)+\lambda H$: Using Sherman–Morrison–Woodbury formula, $C(t)^{i}$ can be efficiently calculated from $C(t-1)$:

[TABLE]

where $A_{[:,j]}$ denotes a vector formed from the $j$-th column of the matrix $A$, $\text{diag}[A]$ constructs a vector from the diagonal components of matrix $A$, and $\text{squ}[B]$ is a vector obtained by taking element-wise squares of the vector $B$. Accordingly, after explicitly calculating $C(0)$,111In practice, we limit $C(0)$ to 2,000 labels for faster interactivity. subsequent updates in $C(t)$ and $C(t)^{i}$ can be performed efficiently for each iteration $t$.

Large-scale extension.

The rank-one update model (Eqs. 4 and 5) relies on explicitly calculating the covariance matrix $C=(L+\lambda H)^{-1}$. Even though the original regularization matrix $L+\lambda H$ is sparse, its inverse $C$ is generally dense. Therefore, this strategy is not directly applicable for large-scale problems where calculating and storing the inverse of $L+\lambda H$ is infeasible. For this case, we adopt sparse eigen-decomposition-based approximation of $L+\lambda H$.

Assume that $L+\lambda H$ is eigen-decomposed:

[TABLE]

where $u\times u$-sized matrix $E^{F}$ stores eigenvectors ($\{\mathbf{e}_{i}\}_{i=1}^{u}$) column-wise, and $\Lambda^{F}$ is a diagonal matrix of the corresponding eigenvalues ($\{\lambda_{i}\}_{i=1}^{u}$). We assume that the eigenvalues and the corresponding eigenvectors are arranged in increasing eigenvalues. Given the prescribed rank parameter $r$, our approximate covariance matrix $\overline{C}$ is given as the best possible $L^{2}$-approximation of rank $r$ to $C$:

[TABLE]

$E$ is a $u\times r$ submatrix of $E^{F}$ containing the first $r$ eigenvectors, with $\Lambda$ as the corresponding $r\times r$ submatrix of $\Lambda^{F}$.

Now suppose that we add a label of the $i$-th data point at time $1$. Applying the Sherman–Morrison–Woodbury formula to $\overline{C}$, we can represent the update equation:

[TABLE]

The corresponding information gain is given as:

[TABLE]

At step $t$, adding a label to the $i(t)$-th data point leads the a new covariance estimate:

[TABLE]

where:

[TABLE]

Amershai et al. comparison.

One of the first algorithms to propose active learning in end-user interaction application is Amershi et al.’s approach [1]. Their algorithm was originally developed for active labeling in metric learning contexts, and so it is not directly applicable to our problem. However, the re-interpretation of their label selection criterion enables transfer to our setting. Their strategy is based on a Gaussian process (GP) model with Gaussian noise: For a given set of labeled data points $S$ and the unlabeled data points $U$, and for the $i$-th unlabeled point, the corresponding score $f(i)$ is defined as:

[TABLE]

where $Var(i|S)$ is the variance predicted for $i$-th data point based on the GP model trained based on $S$, and $Var(i|U)$ is the variance predicted based on the GP model trained on $U$, assuming that $U$ are all labeled. Details of GP prediction can be found in [29]. This interpretation enables us to apply the same principle to our setting: for the $i$-th data point, the corresponding score $\mathcal{I}^{\prime}(X_{i})$ is:

[TABLE]

where $U(t-1)$ is the variance predicted from all unlabeled data points. Maximizing only the predictive uncertainty $\text{diag}[C(t-1)]_{i}$ corresponds to active label selection based on the predictive uncertainty, which is likely to choose an outlier. The (reciprocal of the) denominator represents how well the point $X_{i}$ is connected to the unlabeled data points [1]. Accordingly, normalizing the uncertainty by this value rules out the possibility of choosing the outlier. However, $\mathcal{I}^{\prime}(X_{i})$ is based on how much information the unlabeled data points contains about $X_{i}$, which does not directly represent our final goal: our goal is to choose a data point which minimizes the overall uncertainty on all data points—this is directly addressed by $\mathcal{I}$.

Zhu et al. comparison.

Zhu et al. [42] proposed an active learning framework that selects the labels by minimizing an estimate of the expected risk. This framework relies on evaluating the prediction accuracy of the updated system for each hypothesized output values. Therefore, the application of this approach is limited to the case in which the output space is finite, e.g., $\{-1,1\}$ in the classification problem. Our approach does not have this limit.

3.2 Evaluation

1D criteria learning.

We evaluate our interactive LG adaptation on user rank data versus two techniques: discrete-only RankSVM approaches [27, 31], and a ‘forced binary’ version of our approach (and RankSVM) where labels are set either to $1$ or $-1$ to simulate a simpler yes/no interaction. We compare over increasing numbers of randomly chosen labels, with the remaining data points used as unlabeled examples (10 trials, averaged). Our approach improves performance over both baselines once the number of labels surpasses 20 (Fig. 2).222For quantitative evaluation of the batch LG regularizer on objective measures, please see Kim et al. [19].

2D criteria learning.

For expert users, we can relax the ranking interface convenience and allow 2D criteria via direct positioning. Training labels are positioned in a 2D unit domain, effectively defining a geometric slider space. The labels are 25 faces selected randomly from 2,000 face images of a single person [35], describing horizontal and vertical rotations (Fig. 3). Objectively assigning pose angle is difficult, and so the labels are perceptual approximations. Compared to RankSVM, our embedding better reproduces the user’s intention with a more even spread over the output space. Since RankSVM does not enable users to define a ‘geometry’ in parameter space, coordinating more than one parameter in this way can be challenging.

Active label suggestion.

We compare our performance to 1) random label selection, 2) predictive uncertainty, and 3) Amershi et al. [1] adapted to our semi-supervised setting (Fig. 4). Over 10 trials on the CAESAR dataset (Sec. 4), we randomly selected a set $\mathcal{A}$ of $2,000$ data points, with two intial labels $\mathcal{B}\subset\mathcal{A}$ and train on $\mathcal{B}$. We calculate information gain $\mathcal{I}$ (Eq. 4) for each data point in $\mathcal{A}\backslash\mathcal{B}$, assign the best data point a label, and iterate until $|\mathcal{B}|=50$.

Predictive variance only resulted in suggesting outliers which led to worse results than random selection. The adapted algorithm of Amershi et al. improves upon random selection, while our new algorithm shows further improvement, especially when the number of labels is low. The computational complexity of Amershi et al. and ours are equal. We also demonstrates that our proposed sparse eigen-decomposition-based approximation (Active label selection sparse (proposed)) produces comparable results to the full information-gain-based method (Eqs. 4 and 5). We used only the first 100 ($r=100$) eigenvectors. Calculating the selected eigenvectors of a sparse matrix does not require explicitly generating a dense matrix of size $u\times u$, as the eigen-decomposition can be efficiently updated after an initial pre-interaction computation (Eq. 3.1).

Computation complexity.

This depends on the number of data points $u$, the number of nearest neighbors $k$ in building the LG regularizer [19], the rank of the sparse approximation $r$ (Eq. 3.1), and the number of non-zeros entries in the resulting regularization matrix $H$. This lies in-between $O(uk)$ and $O(uk^{2})$, depending on the well-behavedness of neighborhoods (where $O(uk^{2})$ corresponds to random neighbors). $H$ is built once per dataset as a preprocess. The dimensionality of the data model affects only the construction of the regularizer. At interaction time, complexity depends only on the number of data points. For the incremental step, we randomly select $1,000$ candidate data points among $u$ points, and choose the one which maximizes the information gain (Eq. 7).

On CAESAR, with $u=4,258$ and $k=20$, the preprocess takes 14 seconds. Solving the system took $0.5$ seconds for standard batch LG approach. For active label suggestion, in non-incremental batch LG learning, estimating the predictive variance for each of the $4,258$ data points requires re-training the entire system per data point, which takes $\approx 35$ minutes. In contrast, our algorithm enables suggesting the best data point across all examples in $1.5$ seconds. All timings were on an Intel Xeon 3GHz CPU in MATLAB.

For a larger $60,000$ item database333We use MNIST purely for its size, though it would be unlikely to use our system on this database. and $k=10$, the preprocess takes $\approx 10$ minutes, with $5$ seconds per solve. A database of this size requires our sparse eigen-decomposition-based approach to suggest labels, and this takes about $2$ seconds per label. Generally, only a handful of active labels are needed for suggestion, e.g., top 5, and so this still allows (if slower) interaction. Naively calculating the information gain (Eq. 4) took around $4$ hours, while (dense) incremental selection (Eq. 5) is infeasible due to the prohibitively large memory requirement.

Parameters.

As default parameters, we set regularization weight $\lambda=10^{-6}$, nearest neighbors $k=20$, and estimated dimensionality of the underlying high-dimensional manifold $m$ in feature vector space varies between 5–20 with database size. Parameters can vary per database. While a hyper-parameter search against a test set is at odds with many motivating applications, e.g., initially assessing a just-collected database, in some applications these parameters could be set by a ‘database curator’, after which end users would define their own criteria.

Rank usefulness.

To probe the performance required for useful orderings, we asked 28 users to rate candidate orderings of 50 items against a hypothetical ideal ordering. Users assigned to each candidate rank a 7-point Likert scale score and a binary ‘useful/not’ classification. Candidate orderings were real results generated by our system, which spanned the space between random selection and ground truth. The ideal orderings were visible for comparison. First, the experiment was thoroughly explained and users completed a training example. Then, users ranked 24 candidate rankings in a random order, taken in equal portion from the three ‘Objects’ geometry databases.

We compare the Kendall’s tau rank correlation coefficient of the different candidate rank orderings with the average Likert score attributed by the participants (Fig. 5). Assuming our scale data to be approximately continuous, we fit a line to describe the relationship of usefulness ($y$) to KT ($x$) as $y=4.8x+1.3$. This allows us to approximate the number of labels required for useful criteria. We compute $\tau$ thresholds for 70%, 80%, and 90% acceptance rates. For 70% of participants to be satisfied, $\tau=0.61$; at this level, $4.3\times$ more pairwise orderings must agree than disagree. For 80% of participants to be satisfied, $\tau=0.70$, and $5.9\times$ more pairs must agree than disagree. For 90% of participants to be satisfied, $\tau=0.80$, and $9\times$ more pairs must agree than disagree.

Human variation.

Some criteria are inherently ambiguous. In one application, we organize paintings along the criteria “abstract to concrete”. This task has no well-defined answer: while there is a general trend that can be agreed by most people, the specific rank positions of the paintings are open to interpretation. As such, one measure for the performance that our system should aim to achieve to be useful is the level of agreement between humans.

To investigate the human variance of ranking paintings, we asked seven users to rank 201 images along the criteria “abstract to concrete”. The users were aged 20–40, 2 female and 5 male, with varying artistic experience from regular painters to casual knowledge. During this ‘ground truth’ generation task, many visual comparisons must be made across the rank. Completing this task on a computer is currently difficult because displays are too small to show all images at once with sufficient size for comparison. Thus, we printed each image on paper and asked participants to tape them to a very large wall. The wall acted as the ‘rank space’, and participants were free to use it as they wished. Most used some combination of bubble, insertion, and merge sorts (Fig. 6).

We compute Kendall’s tau rank correlation coefficient between each pair of user rankings, producing 21 coefficients (Tab. 1). The mean coefficient is (coincidentally also) 0.619, with standard deviation of 0.043, and range 0.530–0.684. There is only moderate agreement between participants. This shows the difficulty of the task and the need for an interactive ranking system which can more easily create personal criteria. However, it also provides us evidence, at least for this task, that a system should aim to achieve this level of performance to reach the ‘agreement’ among participants. To achieve approximately the ‘subjective average’ performance across participants in this task with our interface would require approximately 100 labels (Fig. 2, top right).

On average, participants spent 158 minutes to produce their rank—ordering a complex database can take time! We initially developed a mouse-based list drag and drop computer interface for this task, with quick buttons to move items to different sections of the 201-item list as a quick approximate pre-sort (e.g., ‘Move to start/10%/20%…/end’). However, this was slower than using paper and a large physical rank space (Fig. 6).

\newfloatcommand

tcapsidetable[\capbeside]

3.3 Interactive prototype

Our prototype interactive interface allows users to describe criteria by ranking examples, receiving active label suggestions, and viewing and manipulating the recovered sliders (Fig. 7). The database is shown to the south, while the user rank is shown to the north. Initially, the database is shown in a random order. The user can drag suggested items to the rank, or drag any database item from the south up to the rank. The items are not ‘removed’ from the database on dragging; merely, this is the interaction metaphor for labeling an example. Every time an item is inserted into the rank or whenever the rank is reordered, we update the criteria and order all items in the database display to the south. Efficient learning is key to this interactivity.

Criteria may be defined by ranking in 1D. These can be displayed as a linear rank or in western page order, similar to a Web image search where items are laid out by relevance. As our underlying representation is continuous, the criteria can also be displayed as a continuous scale where items are separated by their estimated distances along the scale. For databases which use underlying generative models as their high-dimensional representation, this allows new examples to be generated at specific points along the scale, e.g., at a specific point between two desirable examples (Fig.9).

Criteria may also be defined in pseudo-2D by ranking with two independent 1D rankings (Fig. 8). The criteria output is then viewed as a 2D scatter plot, either as a matrix-style rank ordering or as a continuous 2D space from the underlying continuous representation. For experts, we can relax the rank interaction convenience and allow criteria to be defined directly by placing examples on the continuous scale, e.g., Figure 3.

Our databases can be large with many thousands of items, and so we allow the user to reduce the number of presented items to show the broader trend. When viewing the database organized by rank, we allow the user to control a subsampling of the database items. When viewing the database organized along continuous scales, we allow the user to zoom out. Users can also jump quickly to a relevant point in the output criteria by right-clicking on the rank, and vice versa.

4 Datasets and example applications

Object geometry.

We use data from Hua et al. [17]: 5,000 chair (Fig. 10), 3,000 airplane (Fig. 1), and 1,700 car geometries (Fig. 11) are downloaded from Trimble 3D Warehouse, then oriented and scaled to align object features, then locally deformed to better align shared elements, e.g., seat heights for chairs. Once normalized, we voxelize these spaces and extract, for each voxel, the shortest distance to the nearest mesh point to create a distance field which captures the shape variation of each example. Figure 10 shows an example for chair style preference. Related applications: Shape exploration [17] and synthesis [26] scenarios, especially for large Web collections.

Human geometry.

The CAESAR database contains 4,258 human 3D scans, along with ground-truth caliper body measurements. We fit a statistical model to each of the scans [28], which describes body variations in an abstract 20-dimensional linear shape basis. Fig. 12 shows 2226 female body scans ordered by ‘height with priority, then girth’, which project two orthogonal but related physical attributes down to a single criteria (Fig. 12). Related applications: Finding useful criteria for statistical shape models instead of semantically-meaningless axes found by dimensionality reduction, e.g., body [16], face [36], or cloth  [15].

Painting images.

We collected 19,808 paintings covering periods until 1930 (www.zeno.org/kunst). As a feature vector, we follow Gatys et al. [14] and use the pre-trained VGG 19-layer convolutional neural network to estimate a set of style matrices, based on computing Gram matrices from the neural response in the first five layers. Figure 13 depicts a subset of the database after 201 labels were provided for the criteria “abstract to concrete”. Related applications: Arranging cultural heritage or historical artifacts by style/genre [10, 37].

Face images.

We fit an active appearance model to the Labeled Face Parts in the Wild image database [3, 34]. We extract shape and appearance features as the coefficients of their respective PCA spaces. Figure 1 shows a result where a user ranked 10 faces by expression, to recover an expression slider covering $417$ males in the database. Related applications: Face appearance ranking [22], such as personalized attractiveness scales [12], or in policing to aid suspect identification by a witness ranking database examples by ‘more/less like him/her’.

4.1 Discussion

Ranking vs. metric distances.

Interactive ranking makes the intuitive assumption that people find ranking easier than specifying explicit or relative distances between many objects on scales, where the difficulty of this task is proportional to the complexity of the criteria. This ranking choice assumes that scale metric distances depend linearly on the ordering. In principle, this does not incur any loss of generality: Once a criteria scale is generated, we can always non-linearly re-metrize it by performing an order-preserving regression on the final result.

Feature power.

We focused on designing an interactive semi-supervised learning system suitable for high-dimensional spaces. However, ultimately, the performance of our algorithm also depends on the data features. We use existing feature representations from corresponding related problems (e.g., distance fields for geometric objects); however, in general, the best features depend on the specifics of the database. Here, more sophisticated methods for feature extraction and selection are applicable, e.g., using automatic relevance determination [29].

Non-zero-knowledge vs. exploiting existing labels.

We intentionally designed our system for the general ‘zero-knowledge’ case where there are no existing labels, e.g., using criteria definition as queries to assess what is in a Web-scraped database. However, in the wild, some labels may exist. Taking inspiration from Chaudhuri et al. [7], who extend a supervised learning framework [27], it may be possible to create an interactive criteria exploration system where crowd-sourced labels ‘guide’ our regularizer. This may pull the solution towards the ‘average’ case, which removes absolute subjectivity, but it may also be possible for this guiding to be parameterized and under user control. Further, we only address single session labeling scenarios, but over multiple sessions it may be possible to apply structured labeling techniques to reinforce previous decisions or allow criteria evolution [21].

5 Conclusion

Interactive database exploration is a difficult problem, and a lack of labels requires efficient analysis via semi-supervised learning. To achieve this, we introduced an incremental version of the LG regularizer, which has superior performance to RankSVM. This enables us to guide the user via a new active label suggestion method, which estimates the information gain of labeling a particular data point. Our approach is sparse to enable LG active learning for large datasets. We show an interactive system which uses rank labels to quickly and easily create criteria sliders, and demonstrate its use across datasets of images and geometry.

Acknowledgements

We thank Qi-Xing Huang, Leonid Pishchulin, Thomas Helten, and all of our study participants, particularly Atsunobu Kotani, Frances Chen, Gary Chien, Numair Khan, and Eleanor Tursman. Kwang In Kim thanks EPSRC EP/M00533X/2, and James Tompkin and Hanspeter Pfister thank the DARPA Memex program.