Learning with Batch-wise Optimal Transport Loss for 3D Shape Recognition

TL;DR

This paper introduces a batch-wise optimal transport loss for deep metric learning, which automatically emphasizes hard samples, accelerates convergence, and improves 3D shape recognition performance.

Contribution

It proposes a novel batch-wise optimal transport loss that enhances deep metric learning by focusing on hard examples and improving convergence speed.

Findings

Accelerates convergence significantly.

Achieves state-of-the-art recognition performance.

Outperforms existing methods in 3D shape recognition after fewer epochs.

Abstract

Deep metric learning is essential for visual recognition. The widely used pair-wise (or triplet) based loss objectives cannot make full use of semantical information in training samples or give enough attention to those hard samples during optimization. Thus, they often suffer from a slow convergence rate and inferior performance. In this paper, we show how to learn an importance-driven distance metric via optimal transport programming from batches of samples. It can automatically emphasize hard examples and lead to significant improvements in convergence. We propose a new batch-wise optimal transport loss and combine it in an end-to-end deep metric learning manner. We use it to learn the distance metric and deep feature representation jointly for recognition. Empirical results on visual retrieval and classification tasks with six benchmark datasets, i.e., MNIST, CIFAR10, SHREC13, SHREC14, ModelNet10, and ModelNet40, demonstrate the superiority of the proposed method. It can accelerate the convergence rate significantly while achieving a state-of-the-art recognition performance. For example, in 3D shape recognition experiments, we show that our method can achieve better recognition performance within only 5 epochs than what can be obtained by mainstream 3D shape recognition approaches after 200 epochs.

Figures

Tables

Table 1. Table 1: Retrieval results on the SHREC13 benchmark dataset

Method	NN	FT	ST	E	DCG	mAP
CDMR	0.279	0.203	0.296	0.166	0.458	0.250
SBR-VC	0.164	0.097	0.149	0.085	0.348	0.116
SP	0.017	0.016	0.031	0.018	0.240	0.026
FDC	0.110	0.069	0.107	0.061	0.307	0.086
Siamese	0.405	0.403	0.548	0.287	0.607	0.469
LWBR	0.712	0.725	0.725	0.369	0.814	0.752
Our Method	0.713	0.728	0.788	0.366	0.818	0.754

Table 2. Table 3: Comparisons of batch-wise optimal transport loss with other benchmark methods on the ModelNet40 dataset.

	Evaluation Criteria
	Methods	NN	FT	ST	DCG	E	mAP (%)	Accuracy (%)
Pair-wise	Individual	0.8287	0.6544	0.7891	0.8562	0.5668	69.3%	88.6%
	Triplets	0.8324	0.6968	0.8029	0.8629	0.5927	74.1%	89.1%
	Random	0.8688	0.7948	0.9048	0.9140	0.6601	83.1%	89.5%
Batch-wise	Mean Weighted	0.8750	0.7986	0.9032	0.9158	0.6589	83.3%	89.7%
	Random Reweighted	0.8688	0.7673	0.8846	0.9051	0.6445	83.1%	89.0%
	Optimal Reweighted	0.8762	0.8013	0.8991	0.9178	0.6560	83.8 %	90.3 %

Table 3. Table 4: Retrieval and classification results on the ModelNet10 and ModelNet40 datasets.

Methods	Shape Descriptor	ModelNet10		ModelNet40
		mAP (%)	Accuracy (%)	mAP (%)	Accuracy (%)
(1) MVCNN [52]	2D View-based Descriptor (#Views=12)	N/A	N/A	80.2%	89.5%
	2D View-based Descriptor (#Views=80)	N/A	N/A	79.5%	90.1%
(2) GIFT [4]	2D View-based Descriptor (#Views=64)	91.1 %	92.3%	81.9%	83.1%
(3) 3DShapeNets [62]	3D Voxel Grid ($30\times 30\times 30$)	68.3%	83.5%	49.2%	77.0%
(4) Geometry Image [48]	2D Geometry Image	74.9%	88.4%	51.3%	83.9%
(5) PANORAMA-NN [45]	2D Panoramic View	87.4%	91.1%	83.5%	90.7 %
(6) DeepPano [46]	2D Panoramic View	84.1%	85.4%	76.8%	77.6%
Our Method	2D View-based Descriptor (#Views=12)	87.5%	93.7 %	83.8 %	90.3 %

Equations

$$\mathcal{L}({\mathbf{x}}_i,{\mathbf{x}}_j;f)={\mathbf{y}}_{ij}{\mathbf{M}}_{ij}+(1-{\mathbf{y}}_{ij})\max \{0,\epsilon -{\mathbf{M}}_{ij}\},$$

$$\mathcal{L}({\mathbf{x}}_i,{\mathbf{x}}_j,{\mathbf{x}}_k;f)=\max \{0,{\mathbf{M}}_{ij}-{\mathbf{M}}_{ik}+\epsilon \}.$$

$$D_{\mathbf{M}}(\mathbf{r},\mathbf{c}):=\underset{\mathbf{T}\in U(\mathbf{r},\mathbf{c})}{\min }⟨\mathbf{T},\mathbf{M}⟩,$$

$$D_{\mathbf{M}}^{\lambda }(\mathbf{r},\mathbf{c}):=\underset{\mathbf{T}\in U(\mathbf{r},\mathbf{c})}{\min }⟨\mathbf{T},\mathbf{M}⟩-\frac{1}{\lambda }h(\mathbf{T}),$$

$${\mathbf{D}}_{\text{OT}}^{\lambda }(\mathbf{r},\mathbf{c})=\underset{\mathbf{T}\ge 0}{\min }{\sum }_{i.j=1}^n{\mathbf{T}}_{ij}{\mathbf{M}}_{ij}-\frac{1}{\lambda }h({\mathbf{T}}_{ij})$$

$$\text{s.t.}{\sum }_{j=1}^n{\mathbf{T}}_{ij}=\mathbf{r}\text{and}{\sum }_{i=1}^n{\mathbf{T}}_{ij}=\mathbf{c}\forall i,j.$$

$${\mathbf{M}}^{+}({\mathbf{x}}_i,{\mathbf{x}}_j;f)=e^{-\gamma ||f({\mathbf{x}}_i)-f({\mathbf{x}}_j)|{|}_2^2},$$

$${\mathbf{M}}^{-}({\mathbf{x}}_i,{\mathbf{x}}_j;f)=e^{-\gamma \text{max}\{0,\epsilon -||f({\mathbf{x}}_i)-f({\mathbf{x}}_j)|{|}_2^2\}}.$$

$$\mathcal{L}({\mathbf{x}}_i,{\mathbf{x}}_j;f)={\mathcal{L}}^{+}+{\mathcal{L}}^{-}$$

$$={\mathbf{Y}}_{ij}\frac{1}{2}{\sum }_{ij}^n{\mathbf{T}}_{ij}{\mathbf{M}}_{ij}^{+}+({\mathbf{I}}_{ij}-{\mathbf{Y}}_{ij})\frac{1}{2}{\sum }_{ij}^n{\mathbf{T}}_{ij}{\mathbf{M}}_{ij}^{-},$$

$$\frac{\partial \mathcal{L}}{\partial f({\mathbf{x}}_i)}=\sum _{j=1}^n{\mathbf{T}}_{ij}^{\ast }(f({\mathbf{x}}_i)-f({\mathbf{x}}_j))({\mathbf{Y}}_{ij}-({\mathbf{I}}_{ij}-{\mathbf{Y}}_{ij}){\mathbf{\delta }}_{ij})$$

$$\frac{\partial \mathcal{L}}{\partial f({\mathbf{x}}_j)}=-\sum _{i=1}^n{\mathbf{T}}_{ij}^{\ast }(f({\mathbf{x}}_i)-f({\mathbf{x}}_j))({\mathbf{Y}}_{ij}-({\mathbf{I}}_{ij}-{\mathbf{Y}}_{ij}){\mathbf{\delta }}_{ij}),$$

Full text

1 Introduction

Learning a semantical embedding metric to make similar positive samples cluster together, while dissimilar negative ones widen apart is an essential part for modern recognition tasks [24, 12]. With the flourish of deep learning technologies [31, 47, 53], deep metric learning has gained more attention in recent years [26, 5, 44, 15, 50]. By training deep neural networks discriminatively end-to-end, a more complex highly-nonlinear deep feature representation (from the input space to a lower dimensional semantical embedding metric space) can be learned. The jointly learned deep feature representation and embedding metric yield significant improvement for recognition applications, such as 2D image retrieval [59, 5, 37] or classification [60, 42], signature verification [6], face recognition [12, 60, 44], and sketch-based 3D shape cross-modality retrieval [33, 58, 63].

Despite the progress made, most of the pre-existing loss objectives [6, 12, 26, 44, 5] do have some limitations for metric learning. Commonly used contrastive loss [24, 12] or triplet loss [60, 10] only considers the semantical information within individual pairs or triplets of examples at each update, while the interactions with the rest ones are ignored. It would bias the learned embedding metric and feature representation. Moreover, they do not give enough attention to hard positive or negative examples, by cause of the fact that these samples are often sparsely distributed and expensive to seek out. These hard samples can strongly influence parameters during the network is learned to correct them. As a consequence, methods which neglect them often suffer from slow convergence rate and suboptimal performance. Occasionally, such methods require expensive sampling techniques to accelerate the training process and boost the learning performance [10, 44, 36, 15].

In this paper, we propose a novel batch-wise optimal transport loss objective for deep metric learning. It can learn an importance-driven distance metric via optimal transport programming from batches of samples simultaneously. As we know, the fundamental idea behind metric learning is minimizing the intra-category variations (or distances) while maximizing the inter-category variations (or distances). Thus, those semantically similar positive samples with large ground distances and dissimilar negative examples with small ground distances should be regarded as hard samples. Such samples should be emphasized correctly to accelerate the metric learning process. Figure 1 illustrates our main idea of proposing the new batch-wise optimal transport loss objective. As illustrated, learning with the proposed loss can utilize all available semantical information of training batches simultaneously. The introduced importance-driven distance metric is partly obtained as a solution to the optimal transport program [56, 16]. It can mine and emphasize those hard samples automatically. Thus, the convergence rate of distance metric learning process can be significantly improved. We further develop the new loss objective in a deep metric learning manner. The whole network can be trained discriminatively in an end-to-end fashion. The jointly learned semantical embedding metric and deep feature representation would be more robust to intra-class and inter-class variations. We finally verify the performance of our proposed method applying to various visual recognition tasks, including 2D image recognition, sketch-based 3D shape cross-modality retrieval, and 3D shape recognition. Experiment results on six widely used benchmark datasets, i.e., MNIST, CIFAR10, SHREC13, SHREC14, ModelNet10 and ModelNet40, demonstrate the superiority of the proposed method. Our method can achieve a state-of-the-art recognition performance with a notably fast convergence rate.

In a nutshell, our main contributions in the present work can be summarized as follows:

(1) We propose a novel batch-wise optimal transport loss objective for learning an importance-driven distance metric to improve the existing pair-wise based loss objectives.

(2) We develop a deep metric learning method based on the proposed loss objective, which learns the importance-driven metric and deep feature representation jointly.

(3) We verify the superiority of our proposed method on visual recognition tasks, including 2D image recognition, sketch-based 3D shape retrieval, and 3D shape recognition.

2 Related Work

Recognition of 3D shapes is becoming prevalent with the advancement of modeling, digitizing, and visualizing techniques for 3D objects. The increasing availability of 3D CAD models, both on the Internet, e.g., Google 3D Warehouse [1] and Turbosquid [2], and in the domain-specific field, e.g., ModelNet [3] and SHREC [34], has led to the development of several scalable and efficient methods to study and analyze them, as well as to facilitate practical applications. For 3D shape recognition, one fundamental issue is how to construct a determinative yet robust 3D shape descriptor and feature representation. Compared to 2D images, 3D shapes have more complex geometric structures. Their appearance can be affected significantly by innumerable variations such as viewpoint, scale, and deformation. These have brought great challenges into the recognition task. A natural method is to construct a shape descriptor based on the native 3D structures, e.g., point clouds, polygon meshes, and volumetric grid. Then, shapes can be represented with distances, angles, triangle areas, tetrahedra volumes, local shape diameters [38, 9], heat kernel signatures [7, 29], extensions of handcrafted SIFT, SURF [28], and learned 3D CNNs [62, 35] on 3D volumetric grids. An alternative way is describing a 3D shape by a collection of 2D view-based projections. It can make use of CNN models, which have been pre-trained on large 2D image datasets such as ImageNet [31] and gained a decent ability of generalization [20, 47, 23]. In this context, DeepPano [46] and PANORAMA-NN [45] are developed to convert 3D shapes into panoramic views, e.g., a cylinder projection around its principal axis. Multi-view CNN (MVCNN) [52] groups multiple CNNs with a view-pooling structure to process and learn from all available 2D projections of a 3D shape jointly.

3 Background

Loss objective for metric learning: Metric learning aims to learn a semantical metric from input samples. Let $\bm{x}\in\bm{X}$ be an input sample. The kernel function $f(\cdot;\bm{\theta}):\bm{X}\to\mathbb{R}^{d}$ takes input $\bm{x}$ and generates an feature representation or embedding $f(\bm{x})$. In deep metric learning [24, 12, 50], kernel $f(\cdot;\bm{\theta})$ is usually defined by a deep neural network, parameterized by a series of weights and bias $\bm{\theta}$. Metric learning optimizes a discriminative loss objective to minimize intra-class distances while maximizing inter-class distances. For example, the contrastive loss in seminal Siamese network [24, 12] takes pairs of samples as input and trains two identical networks to learn a deep metric $\bm{M}_{ij}$ as

[TABLE]

where the label $\bm{y}_{ij}\in\{0,1\}$ indicates whether a pair of $(\bm{x}_{i},\bm{x}_{j})$ is from the same class or not. The margin parameter $\varepsilon$ imposes a threshold of the distances among dissimilar samples. Conventionally, the Euclidian metric $\bm{M}_{ij}=||(f({\bm{x}_{i}})-f({\bm{x}_{j}})||_{2}^{2}$ in the feature embedding space is used to denote the distance of a pair of samples. Triplet loss [60, 10] shares a similar idea with contrastive loss, but extends a pair of samples to a triplet. For a given query $\bm{x}_{i}$, a similar sample $\bm{x}_{j}$ to the query $\bm{x}_{i}$, and a dissimilar one $\bm{x}_{k}$, the triplet loss can be formulated as

[TABLE]

Intuitively, it encourages the distance between the dissimilar pair $\bm{M}_{ik}=||(f({\bm{x}_{i}})-f({\bm{x}_{k}})||_{2}^{2}$ to be larger than the distance between the similar pair $\bm{M}_{ij}=||(f({\bm{x}_{i}})-f({\bm{x}_{j}})||_{2}^{2}$ by at least a margin $\varepsilon$.

Optimal transport distances: Optimal transport distances [16], also known as Wasserstein distances [56] or Earth Mover’s distances [43], define a distance between two probability distributions according to principles from optimal transport theory [57, 61]. Formally, let $\bm{r}$ and $\bm{c}$ be $n$-dimensional probability measures. The set of transportation plans between probability distributions $\bm{r}$ and $\bm{c}$ is defined as $U(\bm{r},\bm{c}):=\{\bm{T}\in{\mathbb{R}_{+}^{n\times n}}|\bm{T1}=\bm{r},\bm{T^{T}1}=\bm{c}\},$ where $\bm{1}$ is an all-ones vector. The set of transportation plans $U(\bm{r},\bm{c})$ contains all nonnegative $n\times n$ elements with row and column sums $\bm{r}$ and $\bm{c}$, respectively.

Give an $n\times n$ ground distance matrix $\bm{M}$, the cost of mapping $\bm{r}$ to $\bm{c}$ using a transport matrix $\bm{T}$ can be quantified as $\langle\bm{T,M}\rangle$, where $\langle.,.\rangle$ stands for the Frobenius dot-product. Then the problem defined in Equation (3)

[TABLE]

is called an optimal transport problem between $\bm{r}$ and $\bm{c}$ given ground cost $\bm{M}$. The optimal transport distance ${D_{M}}(\bm{r},\bm{c})$ measures the cheapest way to transport the mass in probability measure $\bm{r}$ to match that in $\bm{c}$.

Optimal transport distances define a more powerful cross-bin metric to measure probabilities compared with some commonly used bin-to-bin metrics, e.g., Euclidean, Hellinger, and Kullback-Leibler divergences. However, the cost of computing ${D_{M}}$ is at least $\mathcal{O}(n^{3}log(n))$ when comparing two $n$-dimensional probability distributions in a general metric space [39]. To alleviate it, Cuturi [16] formulated a regularized transport problem by adding an entropy regularizer to Equation (3). This makes the objective function strictly convex and allows it to be solved efficiently. Particularly, given a transport matrix $\bm{T}$, let $h(\bm{T})=-\sum\nolimits_{ij}{{\bm{T}_{ij}}\log{\bm{T}_{ij}}}$ be the entropy of $\bm{T}$. For any $\lambda>0$, the regularized transport problem can be defined as

[TABLE]

where the lager $\lambda$ is, the closer this relaxation ${D^{\lambda}_{M}}(\bm{r},\bm{c})$ is to original ${D_{\bm{M}}}(\bm{r},\bm{c})$. Cuturi [16] also proposed the Sinkhorn’s algorithm [49] to solve Equation (4) for the optimal transport $\bm{T}^{*}$. Specifically, let the matrix $\bm{K}=exp(-\lambda\bm{M})$ and solve it for the scaling vectors $\bm{u}$ and $\bm{v}$ to a fixed-point by computing $\bm{u}=\bm{r}./\bm{Kv}$, $\bm{v}=\bm{c}./\bm{K^{T}u}$ in an alternating way. These yield the optimal transportation plan $\bm{T}^{*}=diag(\bm{u})\bm{K}diag(\bm{v})$. This algorithm can be solved with complexity $\mathcal{O}(n^{2})$ [16], which is significantly faster than exactly solving the original optimal transport problem.

4 Our Method

In this section, we propose a deep metric learning scheme by using principles of the optimal transport theory [57]. Currently, research works with optimal transport distance [16, 18, 17] mainly focus on theoretical analysis and simulation verification. Thus, it is hard to apply them into a large-scale 3D shape recognition contest directly. To this end, we have done the following three works to construct a trainable batch-wise optimal transport loss objective.

4.1 Importance-driven Distance Metric Learning

Assuming we are given two batches of samples, each batch has $n$ examples $\bm{X}\in\mathbb{R}^{d\times n}$. Let $\bm{x}_{i}\in\mathbb{R}^{d}$ be the representation of the $i^{th}$ shape. Additionally, let $\bm{r}$ and $\bm{c}$ be the $n$-dimensional probability vectors for two batches, where $r_{i}$ and $c_{i}$ denote the number of times shape $i$ occurs in $\bm{r}$ and $\bm{c}$ (normalized overall samples in $\bm{r}$ and $\bm{c}$). The optimal transport introduces a transportation plan $\bm{T}\in\mathbb{R}^{n\times n}$ such that $\bm{T}_{ij}$ describes how much of $r_{i}$ should be transported to $c_{j}$. As described in Equation (4), the optimal transport distance between batches $\bm{r}$ and $\bm{c}$ can be re-formulated as

[TABLE]

The learned optimal transportation plan $\bm{T}^{*}$ is a probability distribution [16], which aims to find the least amount of cost needed to transport the mass from batch $\bm{r}$ to batch $\bm{c}$. The unit of cost corresponds to transporting a sample by the unit of ground distance. Thus, $\bm{T}^{*}$ solved by Equation (5) prefers to assign higher importance values to samples with small ground distances while leaving fewer for others.

Utilizing such property, we define the importance-driven distance metric via imposing semantical information of samples. Specifically, we first define the ground distances for a pair of similar positive samples as

[TABLE]

where $\gamma$ is a hype-parameter controlling the extent of rescaling. This re-scaling operator shrinks large Euclidian distance between similar samples. After re-scaling $\bm{M}^{+}$, the learned $\bm{T}^{*}$ tends to put higher importance values on those similar samples which have far Euclidian distances among each other (a.k.a., hard postive samples), while putting lower on the others accordingly. Thus, it would accelerate the process that similar samples are getting close to each other. For dissimilar negative samples, we define the ground distances correspondingly as

[TABLE]

The hinge loss $\text{max}\{0,\varepsilon-{||f(\bm{x}_{i})-f(\bm{x}_{j})||_{2}^{2}}\}$ penalizes the dissimilar samples within the margin $\varepsilon$ and ignores the others. Thus, contrary to the above similar samples case, here the learned $\bm{T}^{*}$ will pay higher importance values on those dissimilar samples with small Euclidian distances (a.k.a., hard negative samples), while assigning fewer on the others. Thus, it could accelerate the process that dissimilar samples are getting apart to each other.

4.2 Batch-wise Optimal Transport Loss Learning

Based on the defined distances $\bm{M}^{+},\bm{M}^{-}$, and optimal transportation plan $\bm{T}^{*}$, now we can formulate a batch-wise optimal transport loss for metric learning. It can be viewed as an $n$-pairs extension version of the contrastive loss or triplet loss. We define the loss objective as

[TABLE]

where $\bm{Y}_{ij}$ is a binary label assigned to a pair of training batches. Let $\bm{Y}_{ij}=1$ if sample $\bm{x}_{i}$ and $\bm{x}_{j}$ are deemed similar, and $\bm{Y}_{ij}=0$ otherwise. An all-ones matrix is denoted as $\bm{I}\in\mathbb{R}^{n\times n}$ and $n$ is the size of each training batch. In practice, ${\bm{T}_{ij}}\bm{M}_{ij}^{+}$ and ${\bm{T}_{ij}}\bm{M}_{ij}^{-}$ can be regarded as the importance-driven distance metric for positive and negative samples, respectively. The optimal transportation plan ${\bm{T}^{*}}$ obtained by solving Equation (5) is a probability distribution of weights for emphasizing hard positive and negative samples during the loss objective optimization. We just write the loss objective regarding only one pair of batches here for simplicity. The overall data loss objective based on all training batches can be easily derived as $\sum\mathcal{L}$.

4.3 Batch Gradient Descent Optimization

We further derive the back-propagation form of the batch-wise optimal transport loss objective. The proposed loss objective can be embedded into a deep metric learning framework, so that the whole network can be trained discriminatively end-to-end via batch gradient descent.

Since the batch-wise optimal transport distance is a fully connected dense matrix of pairs-wise ground distance, its gradient can be deduced as network flow manner. Specifically, we compute the gradient of corresponding loss $\mathcal{L}(\bm{x}_{i},\bm{x}_{j};f)$ with respect to embedding representations $f(\bm{x}_{i})$ and $f(\bm{x}_{j})$ as follows,

[TABLE]

where $\bm{T}^{*}$ is the optimizer obtained from Equation (4). Motivated by the fast optimal distance computation [16, 18, 21], we relax the linear program in Equation (5) using the regularized entropy as in Equation (4). It allows us to approximately solve Equation (4) in $\mathcal{O}(n^{2})$ time via $\bm{T}^{*}={diag}(\bm{u})\bm{K}diag(\bm{v})$, where $n$ is the size of batch.

The $\bm{\delta}$ here is also a binary indicator assigned to the pairs. Let $\bm{\delta}_{ij}=1$ when the Eucildian distance between shape $\bm{x}_{i}$ and $\bm{x}_{j}$ is within the margin (i.e., $\varepsilon-||f(\bm{x}_{i})-f(\bm{x}_{j})||_{2}^{2}>0$), and $\bm{\delta}_{ij}=0$ otherwise. The $f(\bm{x}_{i})$ and $f(\bm{x}_{j})$ are feature representations obtained through deep neural networks. Therefore, the gradient with respect to the network can be computed easily with the chain-rule in a back-propagation fashion, as far as $\frac{{\partial\mathcal{L}}}{{\partial{f(\bm{x}_{i})}}}$ and $\frac{{\partial\mathcal{L}}}{{\partial{f(\bm{x}_{j})}}}$ are derived. We also note that the defined ground distance $\bm{M}^{+}$ and $\bm{M}^{-}$ are just used to determine the optimal transportation plan $\bm{T}^{*}$ for re-weighting the importance of similar positive and dissimilar negative samples. We do not consider them as variables to compute gradient in Equation (9) for gradient updating.

5 Experiments

In this section, we evaluated the performance of the proposed method with applications to 2D image recognition (i.e., retrieval and classification), sketch-based 3D shape retrieval, and 3D shape recognition tasks. Six widely used benchmark datasets were employed in our experiments, including MNIST [32], CIFAR10 [30], SHREC13 [33], SHREC14 [41], ModelNet10, and ModelNet40 [62].

5.1 Experimental settings

Architecture: Figure 2 illustrates network architecture of deep metric learning with our batch-wise loss objective.

Datasets: The MNIST [32] is a large handwritten digits dataset, which has $60,000$ $28\times 28$ black-and-white training images and 10,000 testing images. The CIFAR10 [30] dataset consists of $60,000$ $32\times 32$ RGB images in 10 different categories, with $6,000$ images per category. There are $50,000$ training images and $10,000$ test images. SHREC13 [33] and SHREC14 [41] are two large-scale datasets for sketch-based 3D shape retrieval. SHREC13 contains $7,200$ human-drawn sketches and $1,258$ 3D shapes from 90 different categories. For each category, $50$ sketches are used for training and remaining $30$ sketches are used for the test. There are $14$ 3D shapes per category generally. SHREC14 is larger than SHREC13, which has $13,680$ sketches and $8,987$ 3D shapes from $171$ categories. Each of the categories has $53$ 3D shapes on average. There are $8,550$ sketches for training and $5,130$ for test. ModelNet [3] is a large-scale 3D shape dataset, which contains $151,128$ 3D CAD models belonging to $660$ unique object categories [62]. There are two subsets of ModelNet can be used for evaluation. ModelNet10 contains $4,899$ 3D shapes from $10$ categories while ModelNet40 has $12,311$ shapes from 40 categories. In our experiments, we used the same training and test splits as in [62]. Specifically, we randomly selected $100$ unique shapes per category, where $80$ shapes were chosen for training and the remaining $20$ shapes for the test.

Evaluations: For retrieval, we used Euclidian distance to measure the similarity of the shapes based on their learned feature vectors output by the metric network as shown in Figure 2. Given a query from the test set, a ranked list of the remaining test samples was returned according to their distances to the query sample. We used the evaluation metrics for retrieval as in [63] when presenting our results. The metrics include nearest neighbor (NN) [14], first tier (FT) [54], second tier (ST) [13], E-measure (E) [11], discounted cumulated gain (DCG) [27], and mean average precision (mAP) [40]. For classification, we trained one-vs-all linear SVMs [8] to classify 2D images and 3D shapes using their features. The average category accuracy [62] was used to evaluate the classification performance.

Parameters settings: In our 2D image recognition, the learning rate and batch size were $0.01$ and $64$ respectively. Our optimizer had a momentum of $0.9$ and [math] weight decay rate. The regularized parameter $\lambda$ in Equation (5) was set to be $0.01$ while the re-scaling parameter $\gamma$ in Equation (6) being $10$. In the sketch-based 3D shape retrieval and 3D shape recognition experiments, the batch size was reduced to $32$. Meanwhile, the learning rate, weight decay and momentum remained the same as what has been used in 2D experiments. We increased the regularized parameter $\lambda$ to $10$, which is the same as the re-scaling parameter $\gamma$.

5.2 Evaluation of Proposed Method

5.2.1 2D Image Recognition

Firstly, we empirically evaluated the effect of our proposed method on two broadly used 2D images benchmark datasets, i.e., MNIST and CIFAR10. As illustrated in Figure 2, we used a Siamese-like symmetrical network structure, which employed Lenet-5 as its base CNN to obtain $256$-dimensional feature vectors for the images in both datasets. Training images were randomly shuffled at the start of each epoch. In each training step, the optimal transportation $\bm{T}^{*}$ between two batches of image features was approximated by iterating the Sinkhorn’s algorithm for 20 times. After each epoch, we computed all image features with the symmetrical network trained so far for classification and retrieval. The categorical accuracies provided by one-vs-rest linear SVMs and the retrieval mAPs given by the similarity measure based on the testing samples were recorded.

The left-hand and middle subfigures in Figure 3 present accuracy and mAP curves of the batch-wise optimal transport loss learning concerning the number of epochs. These figures illustrate the relationship between the convergence rate and recognition performance. Comparing with the pair-wise contrastive loss, our method posses a significantly faster convergence rate. On CIFAR10, it provides a retrieval mAP and a classification accuracy which are approximately $15\%$ and $10\%$ higher than the corresponding values achieved by the pair-wise loss at the end of $50$ epochs. The empirical results indicate that the importance-driven distance metric learning can effectively adjust the distribution of weights. It pays more attention to the hard positive and negative samples during the training process.

5.2.2 Sketch-based 3D Shape Retrieval

We then evaluated our method for sketch-based 3D shape retrieval on two large-scale benchmark datasets, i.e., SHREC13 and SHREC14. The right-hand two subfigures in Figure 3 demonstrate the mAP curves of our batch-wise optimal transport loss as compared to the pair-wise loss objective. As illustrated, our method is about $5$ times and $3$ times faster than LBWR on SHREC13 and SHREC14 respectively. Meanwhile, the retrieval performance is remarkably higher than the compared LBWR.

We also compared our method with several mainstream approaches for 3D shape retrieval, including CDMR [22], SBR-VC [33], SP [51], FDC [51], Siamese network [58], DCML [19], DB-VLAT [55], and LWBR [63]. The evaluation criteria include NN, FT, ST, E, DCG, and mAP.

As summarized in Table 2 and Table 2, our batch-wise optimal transport loss based method achieved the best retrieval performance with respect to all evaluation metrics on SHREC13 and SHREC14. Among compared methods, CDMR, DCML, Siamese network, and LWBR are all deep metric learning based approaches. They measured similarity based on pairs of samples and mapped data into an embedding metric space through different pooling schemes. In contrast, our proposed batch-wise optimal transport loss objective can correctly re-weight the importance value of samples, mainly focus on those hard samples. Thus, our approach obtained better retrieval performance. Its mAP reaches 0.754, which is slightly better than LBWR and significantly better than other methods. Furthermore, the advantage of our approach is enlarged on SHREC14 because this dataset has more severe intra-class and cross-modality variations. As a consequence, the mAP of our proposed method is $0.591$, which is $0.190$, $0.305$, and 0.363 higher than LBWR, DCML and Siamese network, respectively.

5.2.3 3D Shapes Recognition

We finally verified the proposed method for 3D shape recognition on two large-scale 3D shape datasets, i.e., ModelNet10 and ModelNet40. Pair-wise loss and triplet loss suffer from slow convergence rate because they are not capable of exploring all available semantical information within training batches simultaneously. To alleviate this problem, we used random sampling techniques (i.e., recurrently shuffle the training batches during each epoch) to loop over as many randomly sampled pairs as possible. It is expected that the random pairs based loss objective could make full use of all information so that the finally learned semantic metric could be balanced correctly. The left-hand subfigure in Figure 4 presents the mAP curves of batch-wise optimal transport loss and other compared loss objectives for 3D shape retrieval. Similarly, the batch-wise optimal transport loss objective still has significantly faster convergence rate and can achieve a decent retrieval performance within a small number of epochs (i.e., $5$ epochs).

We also examined two different probability distributions, i.e., uniformly distributed mean value ($\nu=\frac{1}{{{n^{2}}}}$) and random numbers in the interval $(0,1)$, as alternatives of the optimal transportation plan. Uniformly distributed mean value weights in the batch-wise loss imply that samples are equally important for later metric learning. Uniformly distributed random weights randomly mark some samples as hard samples within a pair of batches during the learning process. The right-hand subfigure in Figure 4 illustrates comparison results of the retrieval performance concerning the number of epochs for these three re-weighting strategies. It demonstrates that the convergence rate of optimal re-weighted is much faster than the others.

The detailed comparison results are summarized in Table 3. We compared the batch-wise optimal transport loss with other designed benchmark methods using NN, FT, ST, E, DCG and mAP on the ModelNet40 dataset. As illustrated in Figure 4 and Table 3, learning with batch-wise optimal transport loss objective has considerably faster convergence rate than other benchmark methods. It takes only a few epochs (i.e., $5$ epochs) to achieve mAP at $83.8\%$ and accuracy at $90.3\%$, which are better than others after $200$ epochs. It demonstrates that the learned optimal transportation plan can correctly re-weight the training samples according to their importance values during the metric learning process. Moreover, solving Equation (5) to learn optimal transportation plan ${\bm{T}^{*}}$ is not computational expensive in practice. The average running time required by one epoch of individual pair loss objective is $2.51$ seconds, and that of batch-wise optimal transport loss objective takes $9.02$ seconds.

Here, we analyzed the role of learned importance-driven distance metric $\bm{T}^{*}\bm{M}^{*}$ in our work. It is composed of the optimal transportation plan $\bm{T}^{*}\in\mathbb{R}^{n\times n}$ and the semantical information embedded ground distances $\bm{M}^{*}\in\mathbb{R}^{n\times n}$. The element $\bm{M}_{ij}^{*}$ is filled with the distances of batch-wise similar positive samples $\bm{M}_{ij}^{+}$ and the distances of batch-wise dissimilar negative samples $\bm{M}_{ij}^{-}$. The right-hand subfigure in Figure 5 shows that far similar positive and adjacent dissimilar negative samples (i.e., hard samples) are sparsely distributed. The left-hand subfigure is the optimal transportation plan which is actually a probability distribution [16]. The color map reveals the learned metric ensures higher importance weights to those few samples with small ground distance while giving less on the remaining ones.

In the end, we compared our method with state-of-the-art approaches for shape retrieval and classification, including MVCNN [52], GIFT [4], DeepPano [46] and et al. The detailed comparison results are summarized in Table 4. Compared to these approaches, our method based on batch-wise optimal transport loss learning can achieve the (almost) state-of-the-art performance on both tasks.

6 Conclusion

In this paper, we proposed a novel batch-wise optimal transport loss objective to learn an importance-driven distance metric. The learned distance metric can effectively emphasize hard samples according to their importance weights. We then formulated the proposed loss objective into an end-to-end deep metric learning network for recognition. We evaluated the performance and versatility of our method with various visual recognition tasks, including 2D image recognition, 2D sketch-based 3D shape cross-modality retrieval, and multiview based 3D shape recognition. The empirical results verified the proposed method could generically accelerate the convergence rate while achieving excellent recognition performance. Our future work will involve facilitating such a trend and applying this importance-driven distance metric learning to more widespread applications. For example, 3D point cloud classification, segmentation, 3D scene reconstruction, cross-modality correspondence among visual, audio, and text.