Learning Style Compatibility for Furniture

TL;DR

This paper explores using Siamese networks and joint image-text embeddings to assess furniture style compatibility, enabling style matching and attribute-based querying with a large dataset.

Contribution

It introduces a novel approach combining pretrained CNN features and joint embeddings for furniture style compatibility assessment.

Findings

Middle CNN layers effectively capture furniture style.

Joint image-text embeddings enable style-based querying.

Large dataset supports robust evaluation.

Abstract

When judging style, a key question that often arises is whether or not a pair of objects are compatible with each other. In this paper we investigate how Siamese networks can be used efficiently for assessing the style compatibility between images of furniture items. We show that the middle layers of pretrained CNNs can capture essential information about furniture style, which allows for efficient applications of such networks for this task. We also use a joint image-text embedding method that allows for the querying of stylistically compatible furniture items, along with additional attribute constraints based on text. To evaluate our methods, we collect and present a large scale dataset of images of furniture of different style categories accompanied by text attributes.

Tables

Table 1. Table 1 : Evaluation of short Siamese models along with the total number of network parameters. For the short Siamese networks, we consider GoogLeNet  [ 29 ] layers up to Inception 3a, 3b, 4b and 4c.

	Inception 3a	Inception 3b	Inception 4b	Inception 4c	canonical Siamese
AUC	0.6419	0.6938	0.7201	0.7214	0.7501
params	$\approx 0.3M$	$\approx 0.7M$	$\approx 1.5M$	$\approx 1.9M$	$\approx 6M$

Table 2. Table 2 : First two columns: comparison of style classification accuracy for a fine-tuned GoogLeNet versus a linear SVM trained on embedding vectors produced by categorical Siamese network. Second two columns: comparison of AUC scores for compatibility assessment between a fine-tuned GoogLeNet versus the categorical Siamese network.

	classification		compatibility
Model	Bonn Styles	Singapore [12]	Bonn Styles	Singapore [12]
Fine-tuned GoogLeNet	0.4141	0.6580	0.5710	0.6184
Categorical Siamese	0.4920	0.6362	0.7546	0.8289

Table 3. Table 3 : Stylistic-based retrieval results using Siamese networks. We report recall@K (high is good).

	R@1	R@5	R@10
Canonical	33.7	64.1	75.6
Categorical	34.5	64.8	74.9
Short	29.8	63.2	76.7

Table 4. Table 4 : Retrieval with text constraints. We report recall@K (high is good)

	style+category				exact match
	R@1	R@3	R@5	R@10	R@1	R@3	R@5	R@10
text query	0.8	1.9	4.9	6.9	0.3	0.3	0.6	0.8
image query	0.1	1.2	3.0	5.9	0	0	0.2	0.7

Equations

$$L_{\text{can}}(I_i,I_j,Y,\theta )=Y\cdot \frac{1}{2}{D}_{\theta }(I_i,I_j{)}^2+(1-Y)\cdot \frac{1}{2}\max [0,m-D_{\theta }(I_i,I_j){]}^2$$

$$L_{\text{cat}}(I_i,C_i,I_j,C_j,Y,\theta )=L_{\text{can}}(I_i,I_j,Y,\theta )+L_C(I_i,C_i,\theta )+L_C(I_j,C_j,\theta ),$$

$$L_C=-\frac{1}{N}\sum _{n=1}^N\sum _i{y}_i\log {\hat{y}}_i,$$

$$L_{\text{joint}}(I_i,T_i,\theta )=\sum _{j\ne i}\max [0,m-S_{\theta }(I_i,T_i)+S_{\theta }(I_i,T_j)]$$

Full text

1 Introduction

Understanding visual styles is important for application domains such as art, cinematography, advertising, and so on. Previous research in this domain focused primarily on recognition, e.g. of photo aesthetic quality [3], building styles [9], illustration and painting styles [7, 26, 16], city architectures [5], clothing styles  [18]. Beyond recognition, however, style becomes a difficult concept to quantify, since it is a subjective and fine-grained problem. In this paper, we address the problem of style compatibility. We ask whether two objects – furniture specifically, are stylistically congruent, e.g.“How well does this chair match that table?”. Answering such a question would be greatly useful for supporting interior design, but is highly challenging, as it requires the ability to recognize features and attributes characteristic of specific styles across multiple object categories.

Recently, a few works have addressed style compatibility, though their focus is on clothing [23, 31, 10]. Our work is inspired by [31], which uses a Siamese network to measure compatibility between different types of clothing. Siamese networks have shown great success in measuring fine-grained visual similarities [32, 21, 1]. In this work, we investigate the use of different Siamese architectures in assessing furniture style compatibility.

Style, as a high-level semantic concept, can be conveyed not only through the visual appearance, but also through text descriptions. As style data is often compiled from retail resources on the web [1, 10], the text tags and meta-data that often accompany images can also serve as an additional source of information. As such, we also use a joint visual-text embedding for learning stylistic compatibility. Such a model can make stylistically compatible recommendations in the presence of constraints such as the type of furniture, material and color.

To evaluate our methods, we collect and present the Bonn Furniture Styles Dataset. This is a large-scale dataset of approximately 90,000 images of furniture, drawn from six types and 17 different styles. To the best of our knowledge, the only dataset of furniture to date is the Singapore Furniture Dataset from Hu et al. [12] and is too small ($\sim$3000 images) to effectively learn deep models. Our dataset is more challenging and in addition, each image is accompanied by text descriptions of the furniture, with key attributes such as material, color, and so on.

We experiment using the Singapore Furniture Dataset [12] as well as our own large-scale dataset. Our findings show that Siamese networks are not only very successful at evaluating furniture style compatibility but also that compatibility can be assessed with only mid-level features. This opens up the possibility of deploying such a network efficiently on resource-constrained platforms and allows individuals to assess furniture compatibility in their own home with mobile devices. Our main contributions can be summarized as follows:

• •

We explore Siamese networks for learning the style compatibility of furniture and show that style, as a high-level semantic concept, is sufficiently encoded by the middle layers of convolutional networks. As such, style compatibility can already be assessed using relatively short CNNs without much loss in performance in comparison to full-depth CNNs.

• •

We use a joint visual-text embedding which can make style-aware furniture recommendations with additional text-based constraints.

• •

We collected a new large-scale dataset of furniture of different styles, along with associated text descriptions such as category, color, material etc. The dataset will be made publicly available and will be of interest for those working not only on style understanding but also on image attributes and fine-grained image understanding.

2 Related Work

Recognizing visual styles in computer vision has been addressed for classifying classic painting styles [26, 16], building architectures [9] and illustrations and photos [7, 15]. Style has also been explored within the context of predicting the aesthetic quality of photographs [3] and paintings [20], image memorability [13] and image “interestingness” [4]. Exploring stylistic similarity on clothing  [18, 27, 17, 31, 28] has drawn a lot attention due to the broad potential for commercial applications. For example, Veit et al. [31] showed that Siamese networks can assess clothing style compatibility. Likewise, in [10], a bidirectional LSTM and Inception-V3 [30] model was used for visual-semantic embedding of outfit images and their descriptions. The learned embedding are then used for outfit generation, compatibility prediction and finding matching item tasks. Note however, such a method requires sets or ensembles of fashion items for training compatibility.

Within the realm of furniture and home decoration a few works on style have been proposed [1, 12, 22]. Bell et al.[1] evaluated Siamese networks for learning visual similarity between iconic images of furniture depicted on white backgrounds versus in actual scenes. More recently, Hu et al. [12] presented a CNN for furniture style classification, testing on their own collected dataset with style labels from industry professionals. A work similar to ours in spirit [22], tries to assess compatibility of furniture, but is based on learning a compatibility function for 3D models and requires computing geometric features for each 3D model. Additionally it doesn’t account for material and texture which can carry important information about the style.

3 Style Compatibility Models

Our interest in learning about style compatibility of furniture is manifested in two tasks. First and foremost, we would like to be able to assess the style compatibility of two furniture items. We refer to this as the compatibility task. Secondly, given an image of an item of furniture, we would like to be able to retrieve, without knowing its style, other items which are stylistically compatible. Furthermore, we would like to place constraints on the retrieval, e.g. the type of item, or its color, material etc.  We refer to the second task as the retrieval task.

To tackle these two tasks, we learn Siamese architectures (Sections 3.1 to  3.3) to serve as a stylistic similarity metric between pairs of furniture images. These architectures can directly address the compatibility task. They can also naturally be extended to the retrieval task by making pairwise computations of a query image with respect to a collection of images and returning those which have the highest compatibility score. To apply a constrained query, we use using joint visual-text embeddings (Section  3.4) for representing both text and images in the same semantic space.

3.1 Canonical Siamese Networks

The Siamese architecture joins two sub-networks at their output [2] to make some inference about a pair of input data and are commonly used for comparison. Typically, the base of the sub-networks have the same configuration and parameter set. For our Canonical Siamese configuration (see Fig. 2(a)), we use a pre-trained CNN as the base and add a fully connected layer $E$ with D-dimensional output on top, and apply a contrastive loss to learn an embedding that maps similarly styled furniture items close to each other and different styles further apart:

[TABLE]

where $(I_{i},I_{j},Y)$ is a sample pair of images $I_{i}$ and $I_{j}$ with compatibility label $Y\in\{0,1\}$ ($Y=1$ for a positive, compatible pair and $Y=0$ for a negative non-compatible pair). $D_{\theta}(I_{i},I_{j})$ is a distance measure between $I_{i}$ and $I_{j}$. The first term of the loss minimizes the distance between positive image pairs, while the second term of the loss encourages the distance between negative pairs to be as large as possible, up to some margin $m>0$. Such a margin $m$ is necessary, since it is possible to push negative pairs arbitrarily far from each other and thereby dominate the loss function. Siamese networks maps an image $I_{i}$ into a D-dimensional embedding location $x_{i}$. In our experiments, we use the Euclidean distance on these output locations of images $I_{i}$ and $I_{j}$ as the distance, i.e. $D_{\theta}(I_{i},I_{j})=\|x_{i}-x_{j}\|^{2}$. The margin $m$ is determined via cross-validation. Note that the canonical model does not directly receive any style labels during training; styles are defined implicitly from positive compatible and negative non-compatible pairs.

3.2 Short Siamese Network

Convolutional neural networks learn feature hierarchies, where features increase in semantic complexity as one progresses through the layers [33]. The mid-network layers have been shown to be important for preserving painting styles when transferring styles for photographs [8, 14]. Based on this observation, we hypothesize that the later layers of a pretrained CNN are not necessary for differentiating style and experiment with a Short Siamese network (see Fig. 2(b)) with only the initial layers of a pre-trained CNN as an alternative base.

3.3 Categorical Siamese Network

It has been shown that including a classification loss during the training of Siamese model can improve the performance of image retrieval [1, 25]. To leverage style labels, we combine similarity learning and classification into a multi-task setup and define a Categorical Siamese network. This network adds a soft-max layer to the base network so that it produces both an embedding and classification output $C$ (See Fig. 2(c)). The loss for the Categorical Siamese is defined as:

[TABLE]

where $L_{\text{can}}$ is the contrastive loss of the canonical network (see Eq.1), $C_{i}$ and $C_{j}$ are the style annotations, and $L_{C}$ is a categorical cross-entropy:

[TABLE]

where $y$ is the output of the soft-max layer and $\hat{y}$ is the ground truth.

3.4 Joint Visual-Text Embeddings

We can further enhance our understanding of style and style compatibility by leveraging the text information from the meta-data of the product images. In particular, we extract meta-data for a product’s coloring, material, and furniture type, in addition to its style. It is highly challenging to work with such meta-data because such information is not consistently available for all the images; furthermore it can be of variable length e.g.“cream, wood, dresser” or “ruby english tea, hardwood fabric, sofa”.

Our objective is to learn a joint embedding space to which both images and text meta-data can be projected (see Fig. 2(d)). Images and correctly associated meta-data should be kept close together and vice versa for non-matching images and meta-data. We do this in a similar way as DeViSE [6], by projecting both image and text features into a common D-dimensional embedding space, $E$, while minimizing a hinge rank loss:

[TABLE]

where $(I_{i},T_{i})$ is a sample pair of image $I_{i}$ and sequence of text $T_{i}$. Image and text features can be projected into the the embedding positions $x_{I}$ and $x_{T}$ in the D-dimensional embedding space, E. Then, $S_{\theta}(I_{i},T_{i})$ is the dot product similarity between the vectors $x_{I}$ and $x_{T}$ of image $I_{i}$ and text $T_{i}$. $T_{i}$ represents the correct text for image $I_{i}$, while $T_{j}$ is the text for images other than $I_{i}$. This loss encourages correct image and text pairs $(I_{i},T_{i})$ to have a higher similarity score than the wrong pairs $(I_{i},T_{j})$ by a margin $m$.

To represent the meta-data for each product as a fixed-length feature vector (300 in our case), we convert each word in the meta-data to its word2vec [24] representation and apply it as input to a 1-layer Long Short Term Memory network [11] (LSTM) with 300 hidden nodes. The corresponding images are passed through a base network and projected into the joint embedding space.

4 Bonn Furniture Styles Dataset

We collect a dataset of 90,298 furniture images with corresponding text meta-data from Houzz.com, a website specialized in architecture, decoration and interior design. We focus only on iconic images, depicting the furniture on a white background, so as to minimize crosstalk from distractors that may be present in images of the item within a scene. Text meta-data contain information as manufacturer, size, weight, materials, subcategories (e.g folding chair, dining chair, armchair) and, most importantly, the style.

Our dataset has the six most popular categories of furniture on Houzz.com: beds (6594), chairs (22247), dressers (16885), lamps (32403), sofas (4080) and tables (8183). The distribution of styles per category are shown in Fig. 4. Fig. 3 provides examples of the styles for the “bed“ and “chair” categories.

Because our dataset and corresponding labels are extracted from the web, it tends to be noisy with mis-labelled data and duplicates. We first remove outliers by fine-tuning GoogLeNet [29] for the six categories of furniture based on a dataset of 25k images retrieved from the web. Samples with the lowest soft-max scores are considered to be irrelevant and removed. We find duplicates by thresholding on the Hamming distance between computed Perceptual Hash [19] values. When detecting duplicates, we remove one of the two duplicates if the style annotations are the same; if they are annotated with different styles, we remove both. This step also removes different colored versions of the same product.

5 Experiments

5.1 Setup, Training Details & Evaluation

As a base model, we use a pre-trained GoogLeNet [29] with the last layer removed. The embedding layer linearly transforms the 1024 dimensional feature vector output from GoogLeNet [29] into a 256-dimensional embedding feature. For the short Siamese, we first consider the GoogLeNet [29] layers up to Inception 4c and apply an average pooling that produces a 512-dimensional vector, to which we then append the embedding layer. For further comparisons, we also experimented with short networks with layers up to Inception 4b, 3b, and 3a.

To generate training pairs, we use a strategic sampling [31] that creates positive pairs only from different categories, i.e.same style, different types of furniture. This pushes the embedding to focus solely on the style compatibility and not on the similarities between items from the same furniture types. Negative pairs, on the other hand, are pairs of different styles drawn randomly from all types of furniture, both same and different.

We split the dataset into train, validation and test sets according to a 68:12:20 ratio. Our final training set of pairs has 2.2M pairs, while the validation and test sets have 200k pairs; similar to [31], we keep a 1:16 ratio of positive and negative pairs. We also test on the Singapore Furniture Dataset [12], using provided splits. Due to the smaller size of this dataset, we are able to generate 2.2M pairs for training but only 11K pairs for validation and 65K pairs for testing.

All networks are trained in two stages; first we back-propagate only over the embedding / classification layer (with learning rate $\epsilon=0.01$, weight decay $\rho=0.9$) for 50 iterations and then do a fine-tuning over the entire network with a lower learning rate ($\epsilon=0.0001$). Overall training takes eight epochs. Embedding / classification soft-max layer weights are randomly initialized.

For training the visual-text embeddings, we create batches of a reference image and $16$ other randomly sampled images from different styles than the reference. These sets of 17 images, along with their concatenated textual data are then considered as one batch for training the embeddings. The optimizer used for training is RMSProp with a learning rate $\epsilon=0.001$. The base model for extracting the features from the image is kept fixed, so that only the visual embedding weights is learned. However, for the text part of the model, the entire LSTM as well as the textual embedding weights are learned from scratch.

To determine the margin parameter $m$ in Eq. 1, we cross-validate to set a value of $m=50$ for the canonical and short Siamese networks and $m=1000$ for the categorical network.

We evaluate the compatibility task performance with area under the ROC curve (AUC). The Siamese networks, when presented with a pair of images, return a compatibility score between 0 (not compatible at all) and 1 (fully compatible). An AUC value of 0.5 is equal to taking a random guess, while 1.0 indicates perfect discrimination. For the retrieval task, we report recall@K which measures the fraction of times the correct item was found among top K results.

5.2 Comparison of Siamese Network Architectures

First, we compare the performance of three different Siamese network architectures presented in Sections 3.1 to 3.3. As a baseline, we fine-tuned the base network GoogLeNet [29] model to explicitly classify furniture styles, and extracted the CNN features before the classification layer. Later on we compute the Euclidean distance between the CNN features of pairs of images as a measure of similarity. The ROC curves are shown in Figure 5. We find that the canonical and categorical Siamese networks perform very similarly on the compatibility task, with AUCs of 0.7501 and 0.7546 respectively. While the categorical Siamese has only a marginally higher AUC than the canonical, the real impact of the additional categorical cross-entropy loss terms is in the fact that it helps training to converge much faster. After one epoch, the AUC of the categorical Siamese is already 0.732, as opposed to 0.710 for the canonical Siamese.

More interesting, however, is the performance of the short Siamese network. With an AUC of 0.7214, it performs only a little bit worse than the much deeper canonical version. Such a small gap in performance can be explained by the nature of the stylistic features. Most of the stylistic cues can be visually recognized on a local level, rather than the overall view. For example, the “Tropic” style is characterized by woven fabric, whereas “Victorian” stands out from specific ornaments and curves. As such, the majority of the relevant features can be captured by the mid-network convolutional layers of a deep network. Given that the short Siamese has much less parameters ($\approx 1.9M$ versus $\approx 6M$ for the canonical Siamese), it would be more advantageous for resource-limited scenarios such as deployments on a mobile device. In Table 1, we present comparisons for different short Siamese networks along with the number of parameters required.

5.2.1 Style-based comparison of all models:

Figure 6 depicts the AUC scores of the categorical Siamese on the compatibility task over the 17 style classes. Among others, “Victorian” and “Midcentury” styles have the highest AUCs. We attribute this to “Victorian” furniture usually having dark wood color with distinct ornaments and sometimes a golden coating. Similarly, “Midcentury” furniture can easily be distinguished by inclined legs. In contrast, “Eclectic” and “Southwestern” styles have the lowest AUCs; the poor performance can be attributed to both a lower number of images in the dataset (see Fig. 4) and large visual variance across furniture types.

5.3 Style transferability

5.3.1 From Compatibility to Classification and Vice Versa:

We want to test how well our Siamese networks, trained for style compatibility are able to perform on classifying styles. To do so, we train a linear SVM on the features extracted from the embedding layer of the categorical Siamese model and report our classification results in Table 2. We compare this with a fine-tuned version of the base network, i.e.GoogLeNet. We find that our proposed dataset is very difficult; the fine-tuned baseline has a classification accuracy of only 0.41, in comparison to the Singapore dataset [12], which has a classification of 0.66. Using the features learned from the embedding of the categorical Siamese, we get better classification performance on our large-scale dataset (0.49), but slightly worse on the smaller Singapore dataset (0.64). In comparison, the best reported classification results on this dataset is is 0.70, achieved through an SVM classification of a combination of fine-tuned CNN features and hand-crafted features [12]. Given that the differences to our categorical Siamese is only marginal, we conclude that the features learned by the Siamese network are good representations of the style class.

The reverse case, however, of using features learned for classification for determining style compatibility (see Sec. 5.2), does not hold as well (see Table  2). As previously reported, the AUC for the categorical Siamese is significantly higher than the fine-tuned GoogLeNet (0.7546 vs 0.5710). We find a similar trend for the Singapore dataset (0.8289 vs 0.6184). Based on these two experiments, we conclude that features learned for style compatibility are well-suited for performing classification but not vice versa.

5.3.2 Cross-dataset testing:

We investigated the transferability of features learned on one dataset to another by training on our proposed dataset and testing on the Singapore Furniture Dataset [12]. This dataset has similar types of furniture distributed over 16 styles, 14 of which do not appear in our dataset (only Mediterranean and modern are common to both datasets). We find that the categorical Siamese model trained on our proposed dataset is able to achieve an AUC score of 0.7129 on the Singapore Furniture Dataset [12]. This suggests that our network is able to generalize to the notion of style similarity, even when encountering never-seen-before styles. When we try the reverse, however, i.e.training on the Singapore dataset and testing on ours, the AUC is a much lower 0.5534. We attribute the difference in the performance to the size difference of the two datasets (90K vs. 3K). Although both datasets have almost equal number of style classes, our dataset has more data for each class which allows the deep model to learn finer grained similarities from higher amount of variations within each style across furniture types.

5.4 Stylistic image retrieval

We evaluate the Siamese model’s ability to correctly retrieve stylistically similar images in Table 3. To evaluate the ranking we report recall@K, $K\in{1,5,10}$. The performance of the different models in the retrieval task mirror that of the compatibility task. The canonical and categorical models perform similarly, with the categorical being slightly better, while the canonical is a little bit lower. In Figure 1, we show example retrieval results for the categorical Siamese network.

5.5 Retrieval with text constraints

To retrieve stylistically compatible images with additional text constraints, we use the joint visual-text embeddings from Section 3.4. We project the query image features extracted from the GoogLeNet to the joint embedding space position $x_{I}$. Similarly, for the text, which is used as a constraint, we first extract the LSTM [11] features and then project them to the joint embedding space position $x_{T}$. The sum of these two vectors, $x_{sum}=x_{I}+x_{T}$, is another position in this space and we return the results based on a nearest-neighbour search in the joint embedding space using $x_{sum}$ as our query vector.

As a baseline, we use the style text of the query image along with text constraints. We measure the recall of the style and furniture type for such queries and report recall@K, $K\in{1,3,5,10}$ in Table 4 and example retrievals in Figure 7. Admittedly, our recall performance is not so impressive; however there is only a small drop from querying with images versus text and even the image query is still several times better than chance. One should consider the difficulty of this task, which is expressed qualitatively in Figure 7: It can be seen that retrieving exact matches is quite difficult, given the available text attributes. Nonetheless, we can still maintain style and category similar to the query image constrained with text.

6 Conclusion

In this work we tackle the problem of learning style compatibility for furniture and addressed two different style tasks – one assessing compatibility between pairs of images, and one querying for stylistically compatible images. To accomplish this, we propose and test several Siamese architectures. Our evaluations show that Siamese networks excel at capturing the stylistic compatibility; in addition, these networks can be reduced to only low- and mid-level layers of pretrained CNNs and still perform comparably with full CNNs, thereby enabling efficient deployments. In addition, we use a joint visual-text embedding model for making stylistically compatible recommendations, while constraining the queries by attributes like furniture type, color and material. For evaluation, we collected an extensive dataset of furniture images of different style categories, along with textual annotations which we will release publicly. We report comprehensive evaluations on our dataset and show that it is very beneficial for style understanding with and without text.