Efficient Codebook and Factorization for Second Order Representation Learning

TL;DR

This paper introduces a novel method combining codebook and factorization techniques to produce compact, high-performing second order representations for image retrieval, outperforming previous approaches.

Contribution

The proposed approach jointly integrates codebook and factorization schemes to maintain second order performance with fewer parameters, achieving state-of-the-art results.

Findings

State-of-the-art results on three image retrieval datasets

Compact representations with minimal additional parameters

Effective preservation of second order performance

Abstract

Learning rich and compact representations is an open topic in many fields such as object recognition or image retrieval. Deep neural networks have made a major breakthrough during the last few years for these tasks but their representations are not necessary as rich as needed nor as compact as expected. To build richer representations, high order statistics have been exploited and have shown excellent performances, but they produce higher dimensional features. While this drawback has been partially addressed with factorization schemes, the original compactness of first order models has never been retrieved, or at the cost of a strong performance decrease. Our method, by jointly integrating codebook strategy to factorization scheme, is able to produce compact representations while keeping the second order performances with few additional parameters. This formulation leads to state-of-the-art results on three image retrieval datasets.

Tables

Table 1. Table 1 : Comparison of codebook strategy in terms of parameters and performances between the Baseline , BP and our JCF ​ for different codebook size (N) and low-rank approximation (R) on Stanford Online Products.

Method	Baseline	BP		JCF ​-N-R
N	-	-	4	-	4	16			32
R	-	-	-	-	-	4	8	16	4	8	16	32
Parameters	1M	34M	135M	0.8M	1.6M	1.6M	2.6M	4.7M	1.6M	2.6M	4.7M	8.9M
R@1	63.8	65.9	67.1	65.0	65.5	68.2	68.3	69.8	68.1	69.4	69.7	70.6

Table 2. Table 2 : Comparison with the state-of-the-art on Stanford Online Products dataset. bold scores are the current state-of-the-art and underlined are the second ones.

r@	1	10	100	1000
LiftedStruct [16]	62.1	79.8	91.3	97.4
Binomial deviance [27]	65.5	82.3	92.3	97.6
N-pair loss [25]	67.7	83.8	93.0	97.8
HDC [28]	69.5	84.4	92.8	97.7
Margin [29]	72.7	86.2	93.8	98.0
BIER [2]	72.7	86.5	94.0	98.0
Proxy-NCA [30]	73.7	-	-	-
HTL [31]	74.8	88.3	94.8	98.4
JCF -32	77.4	89.9	95.8	98.6
JCF -32-8	76.6	90.0	95.8	98.7

Table 3. Table 3 : Comparison with the state-of-the-art on CUB-200-2011 and Cars-196 datasets. bold scores are the current state-of-the-art and underlined are second.

	CUB-200-2011				Cars-196
r@	1	2	4	8	1	2	4	8
[27]	52.8	64.4	74.7	83.9	-	-	-	-
[25]	51.0	63.3	74.3	83.2	71.1	79.7	86.5	91.6
[28]	53.6	65.7	77.0	85.6	73.7	83.2	89.5	93.8
[2]	55.3	67.2	76.9	85.1	78.0	85.8	91.1	95.1
[31]	57.1	68.8	78.7	86.5	81.4	88.0	92.7	95.7
JCF -32	60.1	72.1	81.7	88.3	82.6	89.2	93.5	96.0
JCF -32-8	58.1	70.4	80.3	87.6	74.2	83.4	89.7	93.9

Equations

$$\mathbf{y}=\sum _i{\mathbf{x}}_i{\mathbf{x}}_i^T=\mathbf{X}{\mathbf{X}}^T\in {\mathbb{R}}^{d\times d}$$

$$\mathbf{y}=\sum _i{\mathbf{x}}_i\otimes {\mathbf{x}}_i=\mathrm{vec}(\mathbf{X}{\mathbf{X}}^T)\in {\mathbb{R}}^{d^2}$$

$$\mathbf{z}=\sum _{\mathbf{x}}\mathbf{z}(\mathbf{x})=\sum _{\mathbf{x}}{\mathbf{W}}^T(\mathbf{x}\otimes \mathbf{x})$$

$$z_i=\sum _{\mathbf{x}}{z}_i(\mathbf{x})=\sum _{\mathbf{x}}{\mathbf{w}}_i^T(\mathbf{x}\otimes \mathbf{x})=\sum _{\mathbf{x}}⟨{\mathbf{w}}_i;\mathbf{x}\otimes \mathbf{x}⟩$$

$$z_i(\mathbf{x})=⟨{\mathbf{u}}_i;\mathbf{x}⟩⟨{\mathbf{v}}_i;\mathbf{x}⟩$$

$$z_i(\mathbf{x})=⟨{\mathbf{w}}_i;\mathbf{h}(\mathbf{x})\otimes \mathbf{x}\otimes \mathbf{h}(\mathbf{x})\otimes \mathbf{x}⟩$$

$$z_i(\mathbf{x})=⟨{\mathbf{p}}_i;\mathbf{h}(\mathbf{x})\otimes \mathbf{x}⟩⟨{\mathbf{q}}_i;\mathbf{h}(\mathbf{x})\otimes \mathbf{x}⟩$$

$$\begin{array}[]{ll}{\bm{p}}_{i}&=\sum_{j}{\bm{e}}^{(j)}\otimes{\bm{u}}_{i,j}\\ {\bm{q}}_{i}&=\sum_{j}{\bm{e}}^{(j)}\otimes{\bm{v}}_{i,j}\end{array}$$

$$z_i(\mathbf{x})=\left(\mathbf{h}(\mathbf{x}{)}^T{\mathbf{U}}_i^T\mathbf{x}\right)\left(\mathbf{h}(\mathbf{x}{)}^T{\mathbf{V}}_i^T\mathbf{x}\right)$$

$$\begin{array}[]{ll}{\bm{p}}_{i}({\bm{x}})&=\sum_{r}{f}_{p,r}\Big{(}{\bm{h}}({\bm{x}})\Big{)}{\bm{e}}^{(r)}\otimes\widetilde{{\bm{u}}}_{i,r}~{}~{}\text{and}\\ {\bm{q}}_{i}({\bm{x}})&=\sum_{r}{f}_{q,r}\Big{(}{\bm{h}}({\bm{x}})\Big{)}{\bm{e}}^{(r)}\otimes\widetilde{{\bm{v}}}_{i,r}\end{array}$$

$$z_{i}({\bm{x}})=\left({\bm{f}}_{p}\Big{(}{\bm{h}}({\bm{x}})\Big{)}^{T}\widetilde{{\bm{U}}}_{i}^{T}{\bm{x}}\right)\left({\bm{f}}_{q}\Big{(}{\bm{h}}({\bm{x}})\Big{)}^{T}\widetilde{{\bm{V}}}_{i}^{T}{\bm{x}}\right)$$

$$z_i(\mathbf{x})=\left(\mathbf{h}(\mathbf{x}{)}^T\mathbf{A}{\tilde{\mathbf{U}}}_i^T\mathbf{x}\right)\left(\mathbf{h}(\mathbf{x}{)}^T\mathbf{B}{\tilde{\mathbf{V}}}_i^T\mathbf{x}\right)$$

Full text

Index Terms—  deep learning, second-order representation, codebook strategy, metric learning, image retrieval

1 Introduction

Learning rich and compact representations is an open topic in many fields such as object recognition [1] or image retrieval [2, 3]. Recently, representations that compute first order statistics over input data have been outperformed by improved models that compute higher order statistics [4, 5, 6, 7]. This strategy generates richer representations and are the state-of-the-art methods on fine grained visual classification tasks [8].

However, even if the increase in performances is unquestionable, second order models suffer from a collection of drawbacks: quadratically increasing dimensionality, costly dimensionality reduction, difficulty to be trained, lack a proper adapted pooling.

The two main downsides, namely the high dimensional output representations and the sub-efficient pooling scheme, have been widely studied over the last decade. On the one hand, the dimensionality issue has been studied through factorization scheme, either representation oriented [9, 10] or task oriented [11]. While these factorization schemes are efficient in term of computation cost and number of parameters, the intermediate representation is still very large (typically 10k dimensions) and hinders the training process, while using lower dimension greatly deteriorates performances.

On the other hand, it is well-known that global average pooling schemes aggregate unrelated features. This problem has been tackled by the use of codebooks (e.g., VLAD [12] and Fisher Vectors [4]) and extended to be end-to-end trainable [13, 14]. However, using a codebook on second-order features leads to an unreasonably large model, since the already large feature has to be duplicated for each entry of the codebook. This is the case for example in MFAFVNet [15] for which the second order layer alone (i.e., without the CNN part) costs as much as an entire ResNet50.

In this paper, we tackle the intermediate representation cost and the lack of proper pooling shortcomings by exploring joint factorization and codebook strategies. Our main results are the following:

We first show that state-of-the-art factorization schemes can be improved by the use of a codebook pooling, albeit at a prohibitive cost.

• -

We then propose our main contribution, a joint codebook and factorization scheme that achieves similar results at a much reduced cost.

Since our approach focuses on representation learning and is task agnostic, we validate it in a retrieval context on several image datasets to show the relevance of the learned representations. We show our model achieves competitive results on these datasets at a very reasonable cost.

The remaining of this paper is organized as follows: in the next section, we present the related work on second order pooling, factorization schemes and codebook strategies. In section 3, we present our factorization with the codebook strategy and how we improve its integration. In section 4, we show an ablation study on the Stanford Online Products dataset [16]. Finally, we compare our approach to the state-of-the-art methods on three image retrieval datasets (Stanford Online Products, CUB-200-2001, Cars-196).

2 Related work

In this section, we focus on second-order information (sections 2.1 2.2) and on codebook strategies (section 2.3).

2.1 Second-Order Pooling

In this section, we briefly review end-to-end trainable Bilinear pooling (BP) [8]. This method extracts representations from the same image with two CNNs and computes the cross-covariance as representation. This representation outperforms its first-order version and other second-order representations such as Fisher Vectors [4] once the global architecture is fine-tuned. Most of recent works on bilinear pooling only focus on computing covariance of the extracted features with a single CNN, that is :

[TABLE]

where ${\bm{X}}\in{\mathbb{R}}^{d\times hw}$ is the matrix of the $h\times w$ extracted $d$-dimensional CNN features. Another formulation is the vectorized version of ${\bm{y}}$ obtained by computing the Kronecker product ($\otimes$) of ${\bm{x}}_{i}$ with itself:

[TABLE]

Due to the very high dimension of the above representation that is quadratic in the feature dimension, factorization schemes are mandatory.

2.2 Factorization schemes

Recent works on bilinear pooling proposed factorization schemes with two objectives: avoiding the direct computation of second order features and reducing the high dimensionality output representation. One of the main end-to-end trainable factorization is based on Tensor Sketch (CBP-TS) [9] which tackles the high dimensionality of second-order features using sketching functions. Their formulation allows to keep less than 4% of the components with nearly no loss in performances compared to the uncompressed model.

This rank-one factorization has been generalized to multi-rank by taking advantage of the SVM formulation to jointly train the network and the classifier [11]. Even if the second-order features are never directly computed, their factorization is limited to the SVM formulation and cannot be used for other tasks. Another task agnostic extensions are e.g., FBN [17] which also integrates the first order into the representation and HPBP [10] which improves the factorization with attention model and non-linearity and applies it to visual question answering. Grassmann BP [18] also improves second-order pooling by dealing with the ”burstiness” of features which may be predominant in high order representations by using Grassmann manifolds and providing an indirect computation of the representation. However, their method relies on Singular Value Decomposition (SVD) and they need to greatly reduce the input feature dimension due to the SVD computation complexity which is cubic in the feature dimension.

For image retrieval tasks, producing very compact representation is mandatory to tackle the indexing of very large datasets. E.g., current state-of-the-art method on the CUB dataset [19] named HTL [18] uses only 512 dimensions for the representation. Thus, all of the aforementioned methods have representations that are still too large to compete in this category. In this work, we start from a rank-one factorization detailed in section 3.1 which is extended by the introduction of a codebook strategy that allows smaller representation dimension, improves performances and makes them competitive to state-of-the-art methods in image retrieval.

2.3 Codebook strategies

An acknowledged drawback of pooling methods is that they pool unrelated features that may decrease performances. To cope with this observation, codebook strategies (e.g., Bag of Words) have been proposed and greatly improved performances by pooling only features that belong to the same codeword.

In the case of second order information, the first representations that take advantage of codebook strategies are VLAT [5, 20] and Fisher Vectors [4]. While in VLAT the high-dimensionality is handled by PCA on local features and intra-projections, Fisher Vectors (FVs) replace the hard assignment by a Gaussian Mixture Model (GMM) and supposes that covariance matrices to be diagonal which leads to smaller representations. However, FV ignores cross-dimension correlations. Strategies like STA [6, 7] extends the VLAT representation by computing cross correlation matrices of nearby features to integrate spatial information and takes advantage of a codebook strategy to avoid the computation of unrelated features. However, as the dimensionality is both quadratic in the codebook size and the feature dimension, factorization scheme is mandatory. In the case of ISTA [7], the proposed dimensionality reduction only allows to reduce the dimensionality to around 20k dimensions, which is twice higher than standard second-order pooling factorization.

In end-to-end trainable architectures, FisherNet [14] extends the FVs and outperforms non-trainable FV approaches but nonetheless has the high output dimension of the original FV. MFA-FV network [15], which extends MFA-FV of [21], generates an efficient representation of non-linear manifolds with a small latent space and is trainable in an end-to-end way. The main drawbacks of their method is the direct computation of second-order features for each codeword (computation cost), the raw projection of this covariance matrix into the latent space for each codeword (computation cost and number of parameters), and finally the representation dimension. In the original paper, the proposed representation reaches 500k dimensions, which is prohibitive for image retrieval as it may require more memory than whole images.

To our knowledge, no efficient factorization combined with codebook strategy has been proposed to exploit the richer representation of second order features combined with the codebook strategy. Our propositions combine the best of both worlds by providing a joint codebook and factorization optimization scheme with a similar number of parameters and computation cost to that of methods without codebook strategies.

3 Method overview

After a presentation of the initial factorization (section 3.1), we first propose an extension to a codebook strategy (section 3.2) and show the limitations of this architecture in terms of computation cost, low-rank approximation, number of parameters, *etc. *Finally, we present our shared projectors strategy (section 3.3) which leads to a joint codebook and factorization optimization.

3.1 Initial factorization scheme

In this section, we present the factorization of the projection matrix and highlight the advantages and limitations of this scheme. Using the same notation as in section 2.1, we want to find the optimal linear projection matrix ${\bm{W}}\in{\mathbb{R}}^{d^{2}\times D}$ to build the output feature ${\bm{z}}({\bm{x}})\in{\mathbb{R}}^{D}$. These output features are then pooled to build the output representation ${\bm{z}}$:

[TABLE]

In the rest of the paper, we use the notation ${z}_{i}$ that refers to the $i$-th dimension of the output representation ${\bm{z}}$ and ${z}_{i}({\bm{x}})$ the $i$-th dimension of the output feature ${\bm{z}}({\bm{x}})$, that is:

[TABLE]

where ${\bm{w}}_{i}\in{\mathbb{R}}^{d^{2}}$ is a column of ${\bm{W}}$. Due to the large number of parameters induced by this projection matrix, we enforce the rank one decomposition ${\bm{w}}_{i}={\bm{u}}_{i}\otimes{\bm{v}}_{i}$ where $({\bm{u}},{\bm{v}})\in({\mathbb{R}}^{d})^{2}$ for all projectors of ${\bm{W}}$. ${z}_{i}({\bm{x}})$ from Eq.​ (4) becomes:

[TABLE]

This factorization is efficient in term of parameters as it needs only $2dD$ parameters instead of $d^{2}D$ for the full projection matrix. However, even if this rank one decomposition allows efficient dimension reduction, it is not enough to keep all the richness of the second-order statistics due to the pooling of unrelated features. Consequently, we extend the second-order feature to a codebook strategy.

3.2 Codebook strategy

To avoid destructive averaging, we want to pool only similar features which belong to the same codeword. This codebook pooling is interesting because each projection to a sub-space should have only similar features. Thus, they lie on a simpler manifold and they could be encoded with fewer dimensions. For a codebook size of $N$, we compute an assignment function ${\bm{h}}(\cdot)\in{\mathbb{R}}^{N}$. This function could be a hard assignment (*e.g. *​, the $\operatorname*{arg\,min}{}$ over distance to each cluster) or a soft assignment (*e.g. *​, the softmax). Thus, output feature ${z}_{i}({\bm{x}})$ becomes:

[TABLE]

Remark that now ${\bm{W}}\in{\mathbb{R}}^{N^{2}d^{2}\times D}$ and ${\bm{w}}_{i}\in{\mathbb{R}}^{N^{2}d^{2}}$. Here, we duplicate ${\bm{h}}({\bm{x}})$ to keep the generalization of bilinear pooling (two codebooks can be learned, one per modality) or for STA based strategies (two nearby features may belong to different codewords). As in equation 5, we enforce the rank one decomposition of ${\bm{w}}_{i}={\bm{p}}_{i}\otimes{\bm{q}}_{i}$ where $({\bm{p}}_{i},{\bm{q}}_{i})\in({\mathbb{R}}^{Nd})^{2}$ to split the modalities. This first factorization leads to the following output feature ${z}_{i}({\bm{x}})$:

[TABLE]

However, this representation is still too large to be computed directly. Then, we enforce two supplementary factorizations:

[TABLE]

where ${\bm{e}}^{(j)}\in{\mathbb{R}}^{N}$ is the $j$-th vector from the natural basis of ${\mathbb{R}}^{N}$ and $({\bm{u}}_{i,j},{\bm{v}}_{i,j})\in({\mathbb{R}}^{d})^{2}$. The decompositions of ${\bm{p}}_{i}$ and ${\bm{q}}_{i}$ play the same roles as intra-projection in VLAD [22]. Indeed, if we consider ${\bm{h}}(\cdot)$ as a hard assignment function, the only computed projection is the one assigned to the corresponding codewords. Thus, this model learns a projection matrix for each codebook entry.

Furthermore, by exploiting the same property used in Eq.​ (7), the following equation can be compacted such as:

[TABLE]

where ${\bm{U}}_{i}\in{\mathbb{R}}^{d\times N}$ and ${\bm{V}}_{i}\in{\mathbb{R}}^{d\times N}$ are the matrices concatenating the projections of all entries of the codebook for the $i$-th output dimension. We call it Joint Codebook and Factorization, JCF ​-N.

This representation has multiple advantages: First, it computes second order features that leads to better performances compared to its first order counterpart. Second, our first factorization provides an efficient alternative in terms of number of parameters and computation despite the decreasing performances when it reaches small representation dimensions. This downside is addressed by the codebook strategy. It allows the pooling of only related features while their projections to a sub-space is more compressible. However, even if this codebook strategy improves the performances, the number of parameters is in ${\mathcal{O}}(dDN)$ As such, using large codebook may become intractable. In the next section, we extend this scheme by sharing a set of projectors and enhance the decompositions of ${\bm{p}}_{i}$ and ${\bm{q}}_{i}$.

3.3 Sharing projectors

In the previous section, one projector is learned to map all features that belong to a given codebook entry for each entry of the codebook. The proposed idea is, instead of using a one-to-one correspondence, we learn a set of projectors that is shared across the codebook. The reasoning behind is that projectors from different codebook entries are unlikely to be all orthogonal. By doing such hypothesis (i.e., the vector space spaned by the combination of all the projection matrices has a lower dimension than the codebook itself), we can have smaller models with nearly no loss in performances. To check this hypothesis, we extend the proposed factorization from section 3.2. We want to generate ${\bm{U}}_{i}$ from $\{\widetilde{{\bm{U}}}_{i}\}_{i\in\{1,...,R\}}$ and ${\bm{V}}_{i}$ from $\{\widetilde{{\bm{V}}}_{i}\}_{i\in\{1,...,R\}}$ where $R$ is the number of projections in the set. Then the two new enforced factorization of ${\bm{p}}_{i}$ and ${\bm{q}}_{i}$ are:

[TABLE]

where ${\bm{f}}_{p}~{}\text{and}~{}{\bm{f}}_{q}$ are two functions from ${\mathbb{R}}^{N}~{}\text{to}~{}{\mathbb{R}}^{R}$ that transform the codebook assignment into a set of coefficient which generate their respective projection matrices. Similarly to Eq.​ (9), we have:

[TABLE]

In this paper, we only study the case of a linear projection:

[TABLE]

where $({\bm{A}},{\bm{B}})\in({\mathbb{R}}^{N\times R})^{2}$. Eq.​ (12) is more efficient in terms of parameters than Eq.​ (9) as it requires ${}^{R}\!/_{N}$ times lesser parameters and computation. We call this approach JCF ​-N-R. In section 4, we provide an ablation study of the proposed method, comparing Eq.​ (9) and Eq.​ (12), demonstrating that learning recombination is both efficient and performing.

3.4 Implementation details

We build our model over pre-trained backbone network such as VGG16 [23] (on CUB and CARS datasets) and ResNet50 [24] (on Stanford Online Products). In both case, the features are reduced to 256d and $\ell_{2}$-normalized. The assignment function ${\bm{h}}$ is the softmax over cosine similarity between the features and the codebook. In metric, we use Recall@K which takes the value 1 if there is at least one element from the same instance in the top-K results else 0 and averages these scores over the test set. Images are resized to 224x224 and we do not use data augmentation. We use SGD with a learning rate of $10^{-5}$, a batch of 64 images, the N-pair triplet loss [25] with the margin set to $0.1$ and $N=2$. We also use semi-hard mining for the final comparison to the state-of-the-art.

4 Ablation studies

4.1 Bilinear pooling and codebook strategy

In this section, we demonstrate both the relevance of second-order information for retrieval tasks and the influence of the codebook on our method. We report recall@1 on Stanford Online Products in Table 1 for the different configuration detailed below.

First, as a reference, we train a Baseline network, i.e., which consists in the average of the features reduced to 512 dimensions (first order model). Then we re-implement BP and extend it naively to a codebook strategy. The objective is to demonstrate that such strategy performs well, but at an intractable cost. Results are reported in the left part of Table 1. This experiment confirms the interest of second-order information in image retrieval with a improvement of 2% over the baseline, while using a 512 dimension representation. Furthermore, using a codebook strategy with few codewords enhances bilinear pooling by 1% more. However, the number of parameters becomes intractable for codebook of size greater than 4: this naive strategy requires 270M parameters to extend this model to a codebook with a size of 8.

Using the factorization from Eq.​ (9) greatly reduces the required number of parameters and allows the exploration of larger codebook. In the case of the factorization alone, the small representation dimension leads to poor performances and are only slightly retrieved using a codebook. On the opposite, our factorization which exploits both the larger codebook and the low-rank approximation is able to reach higher performances (+4% between BP and JCF ​-32-32) with nearly four times less parameters.

4.2 Sharing projections

In this part, we study the impact of the sharing projection. We use the same training procedure as in the previous section. For each codebook size, we train architecture with a different number of projections, allowing to compare architectures without the sharing process to architectures with greater codebook size but with the same number of parameters by sharing projectors. Results are reported in the right part of Table 1. Sharing projectors leads to smaller models with few loss in performances, and using richer codebooks allows more compression with superior results. In the next section, we compare our best model JCF ​-32-32 and its shared version with four times less parameters JCF ​-32-8 against state-of-the-art methods.

5 Comparison to the state-of-the-art

In this section, we compare our method to the state-of-the-art on 3 retrieval datasets: Stanford Online Products [16], CUB-200-2011 [19] and Cars-196 [26]. For Stanford Online Products and CUB-200-2011, we use the same train/test split as [16]. For Cars-196, we use the same as [2]. We report the standard recall@K with $K\in\{1,10,100,1000\}$ for Stanford Online Products and with $K\in\{1,2,4,8\}$ for the other two. We implement the codebook factorization from Eq.​ (9) with a codebook size of 32 (denoted JCF ​-32). While JCF ​-32 outperforms state-of-the-art methods on the three dataset, our low-rank approximation JCF -32-8, which cost 4 times less also leads to state-of-the-art performances on two of them with a loss between 1-2% consistent with our ablation studies. In the case of Cars-196 however, the performances are much more lower than the full model. We argue that the variety introduced by the colors, the shapes, *etc. *in cars requires more projections to be estimated, as it is observed for the full model.

6 Conclusion

In this paper, we explore codebook based second order representations that are intractable in practice. We propose a two-step factorization and a low-rank approximation designed to keep the richness of the second-order representation but with the compactness of the first-order. We provide ablation studies to confirm the necessity of a codebook pooling strategy, the impact of the different factorizations and the benefit of the low-rank approximation to control the computation cost. This representation named JCF outperforms state-of-the-art methods on three image retrieval benchmarks and its low-rank approximation is state-of-the-art on two of them.