Visual Feature Fusion and its Application to Support Unsupervised Clustering Tasks

TL;DR

This paper introduces a novel user-guided feature fusion method for unsupervised clustering that leverages small data samples to enhance interpretability and control, enabling real-time feedback and improved analytical scenarios.

Contribution

The paper proposes a new feature fusion approach that allows user control and interpretation without a cost function, specifically tailored for unsupervised clustering tasks.

Findings

Effective in providing real-time feedback

Enhances user interpretability of feature combinations

Improves clustering results through guided feature fusion

Abstract

On visual analytics applications, the concept of putting the user on the loop refers to the ability to replace heuristics by user knowledge on machine learning and data mining tasks. On supervised tasks, the user engagement occurs via the manipulation of the training data. However, on unsupervised tasks, the user involvement is limited to changes in the algorithm parametrization or the input data representation, also known as features. Depending on the application domain, different types of features can be extracted from the raw data. Therefore, the result of unsupervised algorithms heavily depends on the type of employed feature. Since there is no perfect feature extractor, combining different features have been explored in a process called feature fusion. The feature fusion is straightforward when the machine learning or data mining task has a cost function. However, when such a function does not exist, user support for combination needs to be provided otherwise the process is impractical. In this paper, we present a novel feature fusion approach that uses small data samples to allows users not only to effortless control the combination of different feature sets but also to interpret the attained results. The effectiveness of our approach is confirmed by a comprehensive set of qualitative and quantitative tests, opening up different possibilities of user-guided analytical scenarios not covered yet. The ability of our approach to providing real-time feedback for the feature fusion is exploited on the context of unsupervised clustering techniques, where the composed groups reflect the semantics of the feature combination.

Figures

Tables

Table 1. Table 1: Datasets employed in the evaluations. We report its size, number of classes, and intrinsic dimension.

Name	Size	Classes
STL-10	13,000	10
Animals	30,475	50
Zappos	50,025	4
CIFAR-10	60,000	10
Photographer	181,948	41

Equations

$$E_{st}(F_i)=\frac{1}{|F_i{|}^2}\sum _i^{|F_i|}\sum _j^{|F_i|}{\left(\delta (f_i^i,f_j^i)-||r_i^i-r_j^i||\right)}^2$$

$$E_{al}(F_i)=\frac{1}{|F_i{|}^2}\sum _i^{|F_i|}\sum _j^{|F_i|}{\left(d({\overline{r}}_i,{\overline{r}}_j)-||{\overline{r}}_i-r_j^i||\right)}^2$$

$$E(F_i)=\lambda \cdot E_{st}(F_i)+(1-\lambda )\cdot E_{al}(F_i)$$

$$\sum _k{\beta }_k\Vert T_j^i(s_k^i)-r_k^i{\Vert }^2$$

$$s_j^i-\frac{{\sum }_k{\beta }_k{s}_k^i}{{\sum }_k{\beta }_k}\text{and}{r}_j^i-\frac{{\sum }_k{\beta }_k{r}_k^i}{{\sum }_k{\beta }_k}\text{, respectively.}$$

$$v_j^i=\left(f_j^i-\frac{{\sum }_k{\alpha }_k{s}_k^i}{{\sum }_k{\alpha }_k}\right)M+\frac{{\sum }_k{\alpha }_k{r}_k^i}{{\sum }_k{\alpha }_k}$$

$${\alpha }_i=1/\left(\sum _j^p\frac{(1+\Vert \widetilde{f_i}-\tilde{d}\Vert {)}^2}{(1+\Vert \widetilde{f_j}-\tilde{d}\Vert {)}^2}\right)$$

$$NNM=1.0-\frac{{\sum }_i^N{D}_i}{N}$$

Full text

1 Introduction

Machine learning and data mining techniques are, in general, split into supervised and unsupervised approaches or a combination of both. On supervised approaches, user knowledge is added to the analytical process through sets of already analyzed instances. On unsupervised, knowledge can be added by changing algorithms’ parameters or the input data representation, also known as features. The challenge is, therefore, not only to define the most appropriate set of parameters but also to find the data representation that best expresses the user or analyst knowledge.

Depending on the application domain (e.g., text or image), there exist several approaches to construct features, each providing complementary information about the original or raw data. Since there is no perfect feature, the idea of joining different representations is straightforward. This process is called data or feature fusion Bostrom:2007 , and can occur through the combination of features (vector) or merging distances calculated from the features. When the machine learning or data mining task involves a cost function, for instance, classification accuracy, such a function can be used to guide the combination. However, for tasks, like clustering Xu:2005 ; Tan:2005 or multidimensional projection Nonato:2018 ; Sacha:2017 , where such a function does not exist, user support for combination needs to be provided. Otherwise, in practice, the data fusion is impossible or useless given the abundance of different combinations.

In this paper, we present a novel feature fusion approach that allows users to control and understand the fusion of different feature sets. Starting with a small sample, users employ a simple widget to define the weights for the combination and observe the outcome in real-time through a scatterplot-based visualization. Once the user finds the weighted combination that best matches his or her point of view of similarity, the same weights are used to combine the complete dataset. In this way, we not only allow users to effortless test different combinations but also enables the interpretation of the attained results.

In summary, the main contributions of this paper are:

• •

A novel feature fusion technique that allows users to explore and understand different combinations of features in real-time;

• •

An approach to input user knowledge into unsupervised tasks much more interpretable than parameter tweaking;

• •

An interactive visualization-assisted tool to explore large image collections that allows real-time tuning of the similarity between images to match users expectations.

2 Related Work

The process of integrating information from multiple sources to produce a unified enhanced data model is called data fusion Bostrom:2007 . The reason is to combine different data representations into a single model aiming at incorporating properties of the various sources. Data fusion can occur in different ways, including combining features, that is, the vectorial data representation, or merging distances calculated from the various sources.

The concept of merging features is called feature fusion. Feature fusion aims at generating a unified vectorial data representation based on different sets of features (vectorial representations) Utthara:2010 ; Sudha:2017 . The most straightforward approach is the feature concatenation Anne:2015 ; KuangCLY:2016 . In the concatenation, given the sets of features $F_{1},F_{2},\ldots,F_{p}$, the unified representation is given by $[F_{1},F_{2},\ldots,F_{p}]$. Despite its simplicity, the literature reports several examples. In Wang:2009 , Local Binary Pattern (LBP) Ahonen:2004 and Histograms of Oriented Gradients (HOG) Dalalhog features are concatenated to improve performance in pedestrian detection. In Manshor:2012 , Scale Invariant Transform Features (SIFT) lowe:1999 and boundary-based shape Gonzalez:2004 features are concatenated to improve object recognition. In Chu:2018 , high, low, and medium-layers features of a deep neural network are united to support object detection, and in Chun:2008 color and texture features are progressively concatenated aiming at reducing model complexity in a content-based retrieval framework. Feature concatenation was also used in the text domain. In Loni:2011b , the authors extract seven types of lexical, syntactical and semantic features and combine subsets of them to improve text classification.

Weights can be used in the concatenation process to control the influence of the different features. In this process, the unified representations is given by $[\alpha_{1}F_{1},\alpha_{2}F_{2},\ldots,\alpha_{p}F_{p}]$ Utthara:2010 , where $\alpha_{1},\alpha_{2},\ldots,\alpha_{p}$ are the weights. In Loni:2011 , the weighted concatenation was used to improve text classification by combining lexical, syntactic, and semantic features. In Ma:2016 , a neural network was used to learn the weights of a concatenation, combining different image features, such as color, shape, and texture, to improve classification accuracy. In You:2017 , the authors use a saliency detection model to fuse color and texture features through a weighting strategy. They first transform the color and texture features in saliency features and then linearly combine the saliency features. Different from the previous weighted techniques, in this case, they linearly combine the features instead of concatenating them, that is, the unified representation is given by $[\alpha_{1}F_{1}+\alpha_{2}F_{2}+\ldots+\alpha_{p}F_{p}]$. This is possible since the saliency representations have the same dimensionality.

In practice, the feature concatenation is not recommended since it may result in a huge feature vectors leading to the curse of dimensionality problem Utthara:2010 . One solution is to apply a dimensionality reduction after the concatenation Yu:2017 , or to perform a distance fusion. In the distance fusion, instead of combining the vectorial representations, the distances calculated from the representations are combined. If $\Delta(F_{i})$ represents the distance matrix calculated from $F_{i}$, the resulting distance matrix is given by $\alpha_{1}\Delta(F_{1})+\alpha_{2}\Delta(F_{2})+\ldots+\alpha_{p}\Delta(F_{p})$. In Degani:2013 , a simple normalized combination of distances computed from different types of features is used to cover song identification. The distance fusion can also be performed using weights. In Huang:2010 , weights are used to combine distances calculated from color and texture features to improve the results of a content-based image retrieval system. In Vadivel:2004 , distances calculated from color and texture features are also combined to support content-based image retrieval applications. Finally, in Liu:2017 and in Chu:2018 distances calculated from features extracted from different layers of a deep neural network are combined seeking to improve retrieval tasks.

Different from data fusion, model fusion combines computational models instead of data. Such combination can be performed in two different ways: by combining different models (parametrizations) processing a single feature set (data set), or by combining different models processing different feature sets Kim:2016 . The former is called ensemble learning and has been extensively used for classification tasks. The idea is to combine the prediction of different models using some voting strategy to improve model diversity and classification accuracy Mendes-Moreira:2012 ; Dietterich:2000 . Ensembles of classifiers typically outperform single classifiers Schneider:2017 and have been used in different domains, including remote sensing, computer security, financial risk assessment, fraud detection, recommender systems, medical computer-aided diagnosis, and others Woniak:2014 ; Kim:2016 . Similarly, the later also employs a (weighted) voting strategy to combine different models, but in this case, the models use as input different sets of features. Examples of applications include fruit classification KuangCLY:2016 and sentiment analysis Xia:2011 .

Common to all these data and model fusion approaches is that the combinations can only be appropriately performed when a loss function is available to guide the process, like in classification. If such a function does not exists, or there is a degree of subjectivity in the process, the combination without proper user support hampers its applicability in practice or real scenarios, and none of the mentioned approaches offer such support. In this paper we devise an approach to aid on the process of feature combination, allowing users to control the process to match individual expectations, enabling applications where the user judgment is crucial.

3 Proposed Methodology

Our approach for feature fusion employs a two phase strategy to support users on defining combinations that reflect a particular point-of-view regarding similarity relationships. On the first phase, samples $S_{1},S_{2},\ldots,S_{p}$ are extract from each different set of features $F_{1},F_{2},\ldots,F_{p}$ and merged so that each set $S_{i}$ presents the same objects but represented using the different types of feature. Each sample $S_{i}$ is then mapped to a vectorial representation $R_{i}\in\mathbb{R}^{m}$ preserving as much as possible the distance relationships between the instances. These vectorial representations are then combined to generate a single representation $\overline{R}=\alpha_{1}R_{1}+\alpha_{2}R2+\ldots+\alpha_{p}R_{p}$, which is visualized.

The user can then change the features weights and observe the outcome. Once the sample visualization reflects the user expectations, that is, once the proper weights $\alpha_{1},\alpha_{2},\ldots,\alpha_{p}$ are found, the second step takes place and the defined weights are used to combine the complete sets of features. In this process, the vectorial sample representations $R_{1},R_{2},\ldots,R_{p}$ and the samples $S_{1},S_{2},\ldots,S_{p}$ are used to construct models to map each set of feature $F_{i}$ to a vectorial representation $V_{i}\in\mathbb{R}^{m}$. Since these vectorial representations are embedded in the same space, they can be combined using the weights $\alpha_{1},\alpha_{2},\ldots,\alpha_{p}$, obtaining the final vectorial representation $\overline{V}$ that matches the users expectations defined by the sample visualization. Figure 1 outlines our approach showing the involved steps. Next we detail these steps, starting with the sampling and the dimensionality reduction.

3.1 Sampling and Mapping

The first step of our process is sampling. Since users employ the sample visualization to guide the feature fusion process, it is important to have all possible data structures of the different features represented. Therefore, we recover samples from each different set of features so as to faithfully represent the distribution of each individual set.

In this process, we extract samples from each set $F_{1},F_{2},\ldots,F_{p}$ separately using a cluster-based strategy. We employ the k-means algorithm to create $\sqrt{n}$ clusters, getting the medoid of each cluster as a sample, where $n$ is the number of instances in the raw dataset $D$. We set the number of cluster to $\sqrt{n}$ since this is considered a good heuristic for the upper-bound number of clusters in a data set 413225 . After extracting the sample sets $S_{1},S_{2},\ldots,S_{p}$, we merge their indexes defining a unified set of indexes. Then we recreate the sets $S_{1},S_{2},\ldots,S_{p}$ to have the instances with the indexes contained in the unified set of indexes. Therefore, all sample sets have the same instances, which is mandatory for the sample visualization given that we visualize the combination of all features $\overline{R}$. Also we guarantee that the structures defined by the different types of features are represented by the samples. Notice that, the combined sample features $\overline{R}$ will have at most $\sqrt{n}\times p$ instances, enhancing the probability of having samples that represent the distribution of each individual set of features while not hampering the computational complexity of the overall process since $p\ll\sqrt{n}$.

After recovering the samples, we map them to a common $m$-dimensional space, obtaining their vectorial representation $R_{1},R_{2},\ldots,R_{p}\in\mathbb{R}^{m}$ so that we can combine them to obtain $\overline{R}\in\mathbb{R}^{m}$ (for the sample visualization). In this process, each set of samples $F_{i}$ is mapped to $\mathbb{R}^{m}$ preserving as much as possible the distance relationships in $F_{i}$. We do this by minimizing

[TABLE]

where $f^{i}_{i}$ and $f^{i}_{j}$ are instances in $F_{i}$, $\delta(f^{i}_{i},f^{i}_{j})$ is the distance between them, and $r^{i}_{i}$ and $r^{i}_{j}$ are the vectorial representations in the $m$-dimensional space of $f^{i}_{i}$ and $f^{i}_{j}$, respectively.

Besides preserving distance relationships, our mapping process aim to align the vectorial representations so that $r^{i}_{i}$ is placed as close as possible to $r^{j}_{i}~{}\forall~{}j~{}\in[1,p]$ without affecting the distance preservation of the individual mappings. This is necessary since the unified sample representation is calculated as a convex combination of these representations, that is, $\overline{R}=\alpha_{1}R_{1},\alpha_{2}R_{2},\ldots,\alpha_{p}R_{p}$, with $\sum\alpha_{i}=1$, and misalignments could result in meaningless unified representations. We first calculate the normalized average distance matrix $\overline{\Delta}=\frac{1}{p}\sum_{i}^{p}\Delta_{F_{i}}$ combining the distance matrices of all sets of features, where $\Delta_{F_{i}}$ is the distance matrix calculated from $F_{i}$. Then we map $\overline{\Delta}$ to the $m$-dimensional space using the Equation (1). The idea is to use this average representation as a guide to align the vectorial representations $R_{1},R_{2},\ldots,R_{p}$ minimizing

[TABLE]

where $d(\overline{r}_{i},\overline{r}_{j})$ is the distance between two instances of the average vectorial representation.

Joining Equation (1) and (2) we define the function we optimize in our mapping process seeking to preserve, as much as possible, the distance relationships of the original features $F_{1},F_{2},\ldots,F_{p}$ in the vectorial representations $R_{1},R_{2},\ldots,R_{p}\in\mathbb{R}^{m}$ while aligning them. This function is given by

[TABLE]

where $\lambda$ is a used to control the importance of the distance preservation and the alignment to the produced vectorial representations. $\lambda$ is a a hyperparameter and can be changed to defined a good tradeoff between distance preservation and alignment.

To minimize Equation (3) we use a stochastic gradient descent approach with a polynomial decay learning rate. We set the initial learning rate to $\gamma_{0}=0.1$ and the decay power to $\kappa=0.95$ following common choices found in the literature Wilson:2001 . Algorithm 1 outlines our mapping process. Function random$(F_{i},n)$ select $n$ samples from $F_{i}$ randomly and function init() initialize the mapping also randomly. We have tested a deterministic initialization using Fastmap Faloutsos:1995:FFA:568271.223812 but the gain in quality does not justify the computational overhead. Notice that we normalize all features $f_{i}\in F_{j},~{}\forall~{}i,j$ before this process, so that the Euclidean norm $||f_{i}||=1$. Given the triangular inequality property ($||f_{i}-f_{j}||\leq||f_{i}||+||f_{j}||$), this guarantees a upper limit for the maximum pairwise distance between features. Therefore the distances are in the same range despite the type of feature or its dimensionality, avoiding biasing the process towards the type of feature with the largest maximum distance. In addition, we define the desire dimensionality $m$ of the resulting mappings $R_{1},R_{2},\ldots,R_{p}\in\mathbb{R}^{m}$ as the largest intrinsic dimensionality of $F_{1},F_{2},\ldots,F_{p}$, calculated using the maximum likelihood estimation Levina:2004 . Such dimensionality can also be defined by the user if the target dimensionality is known, such as, ${m=\{1,2,3\}}$ for visualization purposes.

3.2 Weighted Feature Combination

Given the samples vectorial representations $R_{1},R_{2},\ldots,R_{p}$ we build a set of functions using the process defined in Joia:2011:LAM:2068462.2068681 to map each feature set $F_{i}$ into its vectorial representation ${V_{i}\in\mathbb{R}^{m}}$ preserving as much as possible the distance relationships while obeying the geometry define in $R_{i}$. In this process, each instance $f^{i}_{j}\in F_{i}$ is mapped to the $m$-dimensional space trough a orthogonal local affine transformation $T^{i}_{j}:\mathbb{R}^{q^{i}}\rightarrow\mathbb{R}^{m}$, where $q^{i}$ is the dimensionality of $F_{i}$.

The affine transformation $T^{i}_{j}(f)=fM+t$ associated to $f^{i}_{j}$ is defined so as to minimize:

[TABLE]

where $\beta_{k}=\|s^{i}_{k}-f^{i}_{k}\|^{-2}$, with $s^{i}_{k}$ the original feature representation of the $k$-th sample in $S_{i}$.

Equation (4) can be re-written in the matrix form $\|D\left(AM-B\right)\|_{F}$, where $\|\cdot\|_{F}$ denotes the Frobenius norm, $D$ is a diagonal matrix with entries $D_{ii}=\sqrt{\beta_{i}}$, and $A$ and $B$ are matrices with the $j$-th row given by the vectors

[TABLE]

Based on that, $M$ is computed as $M=UV$ where $U$ and $V$ are obtained from the singular value decomposition of $A^{\!\top}DDB=USV^{\!\top}$. Then the vectorial representation $v^{i}_{j}$ of $f^{i}_{j}$ is given by

[TABLE]

Equation (4) is subject to $MM^{\!\top}=I$, which avoids scale and shearing effects, therefore preserving the distance relationships of the input features. Also, notice that the sample vectorial representations $R_{1},R_{2},\ldots,R_{p}$ dictates the geometry of the embeddings $V_{1},V_{2},\ldots,V_{p}$. Since they are aligned by the mapping process defined in the previous section, the linear combination $V=\alpha_{1}V1,\alpha_{2}V_{2},\ldots,\alpha_{p}V_{p}$ can be performed to obtain the final embedding $V$ that incorporates the structures defined by each set of features, weighted according to the user’s point-of-view. For more information about this affine transformation and how the sample vectorial representation controls the final results, please refer to Joia:2011:LAM:2068462.2068681 .

3.3 Feature Combination Widget

To visually support the feature sample combination, we create a widget inspired by the strategy presented in 10.1016/j.neucom.2014.07.072 . The idea is to position anchors (circles) representing each different set of features over a circumference, computing the weights $\alpha_{1},\alpha_{2},\ldots,\alpha_{p}$ according to their distances to a “dial” contained in the circumference. If $\tilde{f_{i}}$ are the coordinates of the anchor representing the feature $F_{i}$ on the plane and $\tilde{d}$ the coordinates of the “dial”, the weight $\alpha_{i}$ related to $F_{i}$ is calculated as

[TABLE]

To help the perception of the weights, we change the transparency level of the anchors and fonts according to $\alpha_{1},\alpha_{2},\ldots,\alpha_{p}$. Figure 2 shows the combination widget. In this example, the “dial”, in orange, is closer to the anchor representing the feature $F_{1}$, so the corresponding anchor is more opaque than the other anchors.

4 Results and Evaluation

In this section, we evaluate our mapping and feature combination processes using different datasets aiming at showing that the sample manipulation effectively controls the complete feature fusion. Next, we describe the employed datasets, detail how we extract features, and present our quantitative and qualitative evaluation.

4.1 Datasets

We use five datasets in our tests, named STL-10 Coates:2011 , Animals Lampert:2009 , Zappos Yu:2014 , CIFAR-10 Krizhevsky:2009 and Photographers Thomas:2016 . These datasets come from a variety of different domains. The STL-10 consists of $13,000$ images split into $10$ classes of different objects. Similarly, CIFAR-10 contains $60,000$ images of $10$ commonly seen object categories (e.g., animals, vehicles, and such) in lower resolution. The Animals dataset is more specific and it is composed of $30,475$ images of animals in $50$ categories. Zappos is a dataset for shoes with $50,025$ images from Zappos.com split into $4$ shoe categories. Finally, the Photographers consists of $181,948$ photos taken by $41$ well-known photographers. Table 1 summarizes the datasets, showing the number of instances and classes.

4.2 Features

We use $4$ distinct methods to extract features, representing low-level and high-level image components. Low-level means that the dimensions of the feature vector has no inherent meaning, but represent a basic understanding of the image such as edges or color. High-level features have semantic meaning. For example, they denote the presence of an object or not in the image.

For the low-level features, we represent (1) color with LAB color histogram; (2) texture with Gabor filters Chen:2004 with $8$ orientations and $4$ scales; and (3) shape with HoG technique Dalalhog with a window size of $8$. For the high-level, we extract deep-features from the pool5 layer using a pre-trained CNN CaffeNet Jia2014 . This network was trained on approximately $1.3M$ images to classify images into $1,000$ object categories.

We believe that these features are discriminative for our datasets. For example, we can differentiate a leopard from a panda using a texture extractor. Texture can identify spots in leopard, and differentiate them from other animals. Similarly, color features can be helpful to recognize pandas, where the more common colors are black and white. Also, HOG is helpful to differentiate the type of animals by their shape, e.g., quadrupeds from birds. Finally, object recognition can complement the HOG descriptor. These examples can be generalized to other datasets as well.

4.3 Quantitative Evaluation

To confirm the quality of our approach, we quantitatively evaluate our mapping and feature combination processes. For the mapping process evaluation, the five datasets of Table 1 are sampled $10$ times randomly reducing them to $5\%$ of their original sizes. We sample the data since we cannot execute the mapping process with large datasets since its memory footprint is O($n^{2}$). Due to the random initialization (see Algorithm 1), we repeat the mapping process test $15$ times. Each different feature from the dataset has its own dimensionality. To ensure a common dimensional space, we calculate the intrinsic dimensionality for each of them and choose the smallest value. This value is used to do the mapping. The minimum values of intrinsic dimensionality are $57$, $71$, $91$, $41$, and $83$ for STL-10, Animals, Zappos, CIFAR-10, and Photographer datasets, respectively.

We use stress and alignment error to evaluate the mapping process (see Equation (1) and Equation (2), respectively). We summarize our results in Figure 3 varying the value of $\lambda$. The stress boxplots (in orange) decrease as $\lambda$ increases. On the other hand, alignment boxplots (in blue) have the opposite behavior. This is the expected outcome since larger values of $\lambda$ preserve the distance relationships, whereas small values align the data.

Setting $\lambda=1$ preserves as much as possible the original distance relationships. This is reflected on a average stress of $E_{st}=0.0009$, but it does not ensure a good alignment (average alignment of $E_{al}=2.0343$). On the other hand, $\lambda=0$ delivered almost a perfect alignment (average alignment of $E_{al}=0.0001$), but it does not enforce the distance preservation (average stress of $E_{st}=0.0345$). In this paper, we are interested in the best trade-off between distance preservation and alignment so that the alignment is obtained without penalizing the overall distance preservation of the mappings. According to our experiments, we achieved this in the range $\lambda=[0.45,0.65]$, where both stress and alignment errors are nearly [math] for our experiments (see Figure 3).

For a qualitative evaluation, we generate two-dimensional representations of the samples using our mapping process setting the target dimensionality to two. We show the results for the STL-10, Zappos and CIFAR-10 datasets in Figures 4, 5, and 6, respectively. In these figures, the points are colored according to image classes. The stress and alignment error values are shown on the top-left corner of each scatterplot. To show the influence of different $\lambda$ in the mapping process, we vary it in the range $[1.0,0.2]$. The first column shows the result produced using $\lambda=1.0$, best preserving the original distance relationship. Notice that the visual representations of each different feature are misaligned among themselves. The second column depicts results with $\lambda=0.8$. Now, the 2D mappings start to align (points of representing images of the same class are placed in close positions). We observe a small increase of the stress error, but the alignment error decreases considerably compared to the first column (see the second measure on the top-left corner). The same behavior is verified in the remaining columns. The last column aligns almost completely all features. As expected, as lambda decreases, the distance preservation also decreases (stress increases), and the alignment improves (alignment decreases). However, the stress changes are minimal. Hence, our approach is capable of making a good alignment between features whereas preserving distance relationships. Similar behavior can be observed in Figure 5 and Figure 6.

For the feature combination, we assess the degree the distance relationships of the sample are preserved into the feature fusion of the whole dataset, intending to demonstrate the effectiveness of the user sample manipulation to the produced dataset. In this evaluation, we first generate $30$ different weight combinations randomly summing up to $1$ and apply it to sample data. Then, we reuse these weights for the whole data fusion and measure if the distance relationships induced by the weights on the sample are presented in the whole dataset. We use the Nearest Neighbor Measure (NNM) Cui:2006 in this analysis.

NNM quantifies the similarity of each instance in the whole data with its nearest neighbor in the sampled data. NNM is given by Equation 7, where $D_{i}$ is the smallest distance among the $i-th$ instance in the complete dataset and the instances in the sample, and $N$ denotes the number of instances. The authors normalized each dimension of the data to the range $[0,1]$. However, this results in the loss of the magnitude of the dimensions. So, we change the normalization per dimension by a unit vector normalization per instance to avoid such an effect. The output of NNM is in the interval $[0,1]$ with larger values indicating better results.

[TABLE]

We compare the NNM values of our feature fusion with two baselines: feature concatenation and distance fusion (see Section 2). Boxplots in Figure 7 show that our approach outperforms the other two baselines by at least $5\%$. The mean value for our method is $0.9365$, and the baselines achieve $0.8877$ and $0.8958$, respectively. Hence, our method preserves more accurately the data distribution of the sample in the whole dataset fusion.

4.4 Qualitative Evaluation

Besides de quantitative evaluation, we also present an example based on projections for qualitative evaluation. The reasoning is to project the complete combined dataset ($E$), showing that the patterns observed in the sample projection ($R$) are preserved on the complete projection. In this example, we use our approach to explore large photo collections considering different user perspectives about similarity among images. We use the photographers dataset. In addition to the features described on Section 4.1, we create a new set of features to describe each photographer. We use Wikipedia articles about each photographer and construct a bag-of-words vector to represent them. Photos of the same photographer share the same feature vector, and the similarity among photos is defined as the similarity between texts describing the photographers.

As explained before, based on a sample, using our approach users can combine different features employing the combination widget (see Figure 2) until the sample visualization reflects a particular understanding regarding the similarity among photos. Figure 8 shows three different combinations. The first (Figure 8(a)) provides more importance to color and objects contained in photos and little importance to information about photographers. The second (Figure 8(b)) is defined taking the idea of photographic style from Thomas:2016 , fusing objects and Wikipedia features. Finally, the third (Figure 8(c)) shows the result of combining texture, borders and a little amount of color.

Once the feature combination has been defined reflecting the users’ point of view, a projection representing the complete photo collection is constructed. Figure 9 shows the produced layout using the weights defined on Figure 8(a). In this figure, since the color is an important feature, we observe a clear separation between back-and-white and colorful images. Also, given the weight assigned to the features representing objects, it is possible to notice a separation among photos of people, landscapes, houses in certain regions of the figure. We zoom in two small portion of the projection (at the top and at the right side) to show this effect. On the colored images (right), we observe images with sky and forest. On the gray images (top), we observe houses, sky, and forest.

Figure 10 depicts the final projection using the weights defined on Figure 8(b). In this figure, we zoom-in a region on the bottom-left. We mainly find portrait images in the zoomed region. Remember that in this weight combination, our goal was to represent the photographic style. The selected photos are from two well-known photographers, Van Vechten and Curtis, that mostly work with portraits, presenting similar styles Thomas:2016 . These examples qualitatively attest that the similarity patterns observed on the sample projection are presented on the complete projection, corroborating the quantitative results measured using the NNM index.

5 User-guided Clustering

One of the most appealing application scenarios for our approach is to assist non-supervised strategies, such as clustering techniques. Clustering techniques seek to split sets of data instances into groups so that instances belonging to the same group are more similar to each other than to those in other groups. Therefore, clustering is a subjective task that depends on the way similarity is computed, and the ability to explicitly control and understand similarity is the benefit our approach offers.

Following we present an example of using our approach to control clustering results of a sample of the photographers dataset containing $7,800$ instances. In this example, we define different weights for features and observe how this influence the composed groups. In Figure 11 we analyze a transition between color and Wikipedia features. Color starts with weight $1$, and decreases to weight [math] as Wikipedia weight increases from [math] to $1$. We generate new fused features in each intermediate state. In each combination state, we compute clusters using the Mean-shift Algorithm Comaniciu:2002 . We opt to use this algorithm because we do not need to provide the number of clusters as input, so the produced results directly reflect the provided similarity (or combination of features).

We display the different clustering configuration (for each combination) using the parallel sets Kosara:2006 . On the parallel sets, the vertical axes represent different clusterings $C_{k}$, where $k$ indicates a different weight combination of features. All axes contain a set of groups where different colors represent different groups. Curves between axis $k$ and $k+1$ are colored using the colors of the groups in $C_{k}$. This coloring scheme improves the perception of membership changes between different clusterings results. To reduce cluttering, we implement a simple filtering strategy to remove non-relevant curves. For each group in $C_{k}$, we evaluate how many instances from this group are redistributed in the groups in $C_{k+1}$. If the quantity is less than a percentage threshold, the curves representing these instances are removed. This threshold is a user parameter and can be adjusted accordingly.

Figure 11 shows parallel sets with $9$ axis representing clusterings results $(C_{0},C_{1},...,C_{8})$ with a filtering threshold equals to $0.1$. $C_{0}$ axis represents the results for the color feature only (no Wikipedia feature is considered). It has two groups, one presenting colorful and the other black-and-white photos. $C_{1}$ shows fused features with $0.79$ weight to color and $0.21$ weight to Wikipedia. Most of the two groups presented in $C_{0}$ remain in $C_{1}$, but some instances change their membership.

From $C_{4}$, more groups are composed, and the colorful and black-and-white photo division is lost. Finally, $C_{8}$ represents the clustering for Wikipedia feature only (no color features). Note that from clusterings $C_{6}$ to $C_{8}$, the groups are more stable, that is, most of the items in a certain group tend to be assigned to the same group as $k$ increases. In order to analyze the semantic meaning of the groups, we select the purple group ($g_{7}$) from $C_{8}$, and we check its correspondent instances backward. Photos of that group were taken by Brumfield, Gottscho, and Horydczak, which are three iconic American photographers. We map the data from $C_{8},g_{7}$ to the visual space using the force-scheme technique tejada:2003 . Figure  12(d) shows the result where each photo border is colored with its group color. As can be observed, photos are similar in content and appearance. Brumfield, Gottscho, and Horydczak work is focused on architectural photography We also observe that there is a mixture of colorful and black-and-white photos in this group. However, clustering $C_{0}$ shows a clear separation between these two types of photos (Figure 12(a)). Looking at the sequence of curves from clustering $C_{8}$ to $C_{0}$, it is possible to analyze the $g_{7}$ group, and when these photos are merged backward. We highlighted the path in the parallel set with darker colors for easy navigation. From $C_{8}$ to $C_{4}$, the groups are stable. Instances of that groups are also projected and depicted in Figures 12(d) and 12(c). In $C_{4}$, $g_{1}$ is formed by instances from $C_{3}$, $g_{1}$ and $g_{3}$ groups. Corresponding instances from $C_{3}$ are mapped in Figure 12(b). Note that in $C_{3}$, colorful and black-and-white photos are mixed.

Parallel sets are useful tools to show the difference between clustering results. However, they do not show the similarity relationships between instances. In order to explore clusters and the relationships between instances, we also visualize the pairwise dissimilarity matrix produce from a given feature combination as a heatmap. In our representation, similar items are rendered in brown colors, whereas dissimilar ones are rendered in pale orange colors. The order (rows and columns) of our representation is obtained by the position of the leaves in a dendrogram generated by the average linkage hierarchical clustering Sokal:1958 ; Day:1984 ; Sander:2003 .

Figure 13 shows dissimilarity matrices using the same weight combinations that generate $C_{0}$, $C_{3}$, $C_{4}$ and $C_{8}$ on parallel sets. In Figure 13(a) we can spot two groups (two brown areas on the main diagonal). The colored margins indicate the groups of the instances given by the clustering algorithm. Note there are some instances from the green group in the other group, denoting a potential problem with the clustering algorithm. Similar behavior can also be observed in Figure 12(a). In Figure 13(b), the two major groups remain, but sub-groups can now be noticed inside the larger ones. Figure 13(c) also shows two big brown areas on the diagonal. However, these groups have the same size. In the previous matrix, one group is bigger than the other because the color feature has more weight and the dataset has more black-and-white than colorful photos. In Figure 13(c), Wikipedia feature begins to have more contribution in the combination process forming groups that groups photos according to style and color. Finally, in Figure 13(d), there are several groups on the main diagonal and two major groups that intersect. A possible explanation is that some photographers tend to shoot similar object categories, but they are from different schools of thought Thomas:2016 . Looking at the purple part, we can see some sub-groups, each sub-group representing a photographer with a similar style. These sub-groups are also shown in Figure 12(d).

6 Conclusion

In this paper, we proposed a novel approach for feature fusion that successfully allows users to incorporate knowledge into the fusion process. It is a two-step strategy where, starting from a small sample of the input data, users can easily test different feature combinations and check in real-time the resulting similarity relationships. Once a combination that matches the user expectation is defined, it is propagated to the whole dataset through an affine transformation. Our experiments show that the complete dataset combination preserves the similarities from the sample configuration, providing our approach as a very flexible mechanism to assist the feature fusion process.

We have applied the proposed feature fusion approach to allow users to control and understand the results of clustering techniques. Clustering is one of the most attractive application scenarios for our approach given the subjectiveness involved in unsupervised tasks. Currently, visualization assisted clustering techniques only allow to add user knowledge by changing techniques parameters Kwon:2018 ; kern:2017 ; bruneau:2015 ; Wolfgang:2018 . Enabling users to guide the input feature configuration renders a much flexible control since users can explicitly steer the semantics of the input data and the similarity relationships (e.g., images are similar due to the color vs. images are similar due to the presence of objects), consequently controlling the reason for the cluster formation while allows an easy interpretation of the composed groups.