Learnable Parameter Similarity

TL;DR

This paper introduces LPS, a method that learns to measure the semantic similarity between trained models' parameters, revealing relationships between visual tasks and aiding transfer learning.

Contribution

It proposes a novel second-order neural network approach for parameter similarity measurement and creates ModelSet500 as a benchmark for this task.

Findings

LPS effectively measures parameter similarity across models.

ModelSet500 provides a comprehensive benchmark for similarity learning.

Experiments validate the method's ability to uncover task relations.

Abstract

Most of the existing approaches focus on specific visual tasks while ignoring the relations between them. Estimating task relation sheds light on the learning of high-order semantic concepts, e.g., transfer learning. How to reveal the underlying relations between different visual tasks remains largely unexplored. In this paper, we propose a novel \textbf{L}earnable \textbf{P}arameter \textbf{S}imilarity (\textbf{LPS}) method that learns an effective metric to measure the similarity of second-order semantics hidden in trained models. LPS is achieved by using a second-order neural network to align high-dimensional model parameters and learning second-order similarity in an end-to-end way. In addition, we create a model set called ModelSet500 as a parameter similarity learning benchmark that contains 500 trained models. Extensive experiments on ModelSet500 validate the effectiveness of the proposed method. Code will be released at \url{https://github.com/Wanggcong/learnable-parameter-similarity}.

Tables

Table 1. Table 1: Task classification.

methods	top-1 (%)	top-5 (%)	top-10 (%)
random prediction	2.0	9.6	18.3
only one FC layer	5.3	17.5	29.3
Frobenius+FC,w/o alignment	5.5	18.3	29.0
Frobenius+FC, w/ alignment (Ours)	79.8	94.8	98.0

Table 2. Table 2: Task retrieval.

methods	rank-1 (%)	rank-5 (%)	rank-10 (%)
random prediction	10.0	41.0	65.1
only one FC layer	15.0	55.8	83.3
Frobenius+FC, w/o alignment	13.3	58.3	80.0
Frobenius+FC, w/ alignment (Ours)	80.7	95.0	95.8

Equations

$${\theta }_{t+1}={\theta }_t-{\alpha }_t\nabla f(D;{\theta }_t)$$

$${\varphi }_{t+1}={\varphi }_t-{\beta }_t\nabla h(\mathcal{D};{\varphi }_t)$$

$$F_l\odot {\theta }_l^g=F_{l+1}^g$$

$$F_{l+1}\odot {\theta }_{l+1}^g=F_{l+2}^g$$

$$s_l^{g_i,g_j,c_i,c_j}={\theta }_l^{g_i,c_i}\otimes {\varphi }_l^{g_j^{{}^{\prime }},c_j}$$

$$s_l^{g_i,g_j^{{}^{\prime }}}=\sum _{c_j}\underset{c_i}{\max }{s}_l^{g_i,g_j^{{}^{\prime }},c_i,c_j}$$

$$s_l^{g_i,g_j^{{}^{\prime }}}={\theta }_l^{g_i,c}\otimes {\varphi }_l^{g_j^{{}^{\prime }},c}$$

$$s_l^{r_1,g_j},s_l^{r_2,g_j},..,s_l^{r_{g_l},g_j}=sort(s_l^{1,g_j},s_l^{2,g_j},..,s_l^{g_l,g_j})$$

$$\mathcal{L}=-\sum _{i=1}^N{\mathcal{Y}}_i\log (\frac{1}{W}\sum _{l=1}^L{w}_l{q}_i^l)$$

Full text

1 Introduction

Purpose-specific visual tasks have achieved greater commercial success by focusing on specific optimization problems, e.g., face recognition, object classification, object detection, visual object tracking, and instance segmentation. Can we exploit the underlying relations between different tasks and extend these task-specific methods to task-generic ones? Or can we connect shallow AI to general AI via task relations? Task relation sheds light on the learning of high-order semantic concepts.

Lots of evidence reveals transfer learning approaches [14, 18, 9] that exploit the underlying relations between different tasks can further improve purpose-specific visual tasks with less labeled data. For instance, domain adaption methods [18, 9] attempt to gain knowledge from source tasks and then apply it to a different but related target task. It is assumed that the knowledge learned from source tasks can help the learning of the target task.

Driven by these transfer learning methods, one would think: how to measure the relations between different tasks? Existing methods that offer partial solutions for this problem can be categorized into two groups. In the first group, a wide variety of transfer learning methods simply assume that tasks are related or unrelated based on human intuition or experience. For example, the knowledge gained from car recognition could be applied to truck recognition because cars intuitively look like trucks. However, one drawback of these methods is that human intuition could be different from machine learning principles. A negative transfer [16] could happen when human intuition is unreliable and the source domain data could lead to the reduced performance in the target domain. When the number of source domains is very large in some scenarios, it is hard to directly tell which is the best one for transfer learning.

In the second group, some methods attempt to jointly optimize multiple tasks and estimate task relations by cross-validation. For example, a taskonomy method [20] computes an affinity matrix among tasks based on whether the solution for one task can be sufficiently easily read out of the representation trained for another task. It uses $22\times 25$ transfer networks for the first-order transfer of 26 tasks. However, this pipeline requires a large amount of computation cost to jointly train all of the subsets of a task set. When a new task comes, it is needed to jointly train this new task and old tasks, which strongly limits its applications in real-world scenarios.

To address these drawbacks of existing methods, we propose a novel Learnable Parameter Similarity (LPS) method that learns second-order similarity to measure task relations by using trained task-specific models. Our observation is that the distance between intra-task models is closer than that between inter-task models. Let $\mathcal{T}$ denote a set of tasks. For each task $\mathcal{T}_{i}\in\mathcal{T}$, we repeat the training procedure $m_{i}$ times and thus obtain $M=\sum\limits{{m_{i}}}$ trained models. We then use these task-specific models as metadata points to train a second-order neural network to measure the parameter similarity.

Different from existing transfer learning methods, the LPS method measures task relation using task-specific models that are trained on independent task-specific datasets without jointly optimizing two many subsets of a task set. LPS pays attention to higher-order semantics/concepts, as illustrated in Figure 1. Data points produce a data similarity metric. Data similarity metrics produce a parameter similarity metric. If data is the zero-order similarity and data similarity is the first-order similarity, then parameter similarity can be regarded as the second-order similarity. LPS is also different from learning to learn methods. The former is to learn second-order similarity based on first-order similarity while the later is to learn hyper-parameters to know how to learn, which still focuses on the optimization of the first-order similarity.

Overall, the key contributions of this paper are:

• •

We propose a novel Learnable Parameter Similarity (LPS) method that learns second-order similarity to measure task relation by using trained task-specific models instead of jointly training a large number of transfer networks.

• •

We introduce a hierarchical second-order network to deal with high-dimensional unaligned deep models and learn an effective parameter representation.

• •

We create a parameter similarity learning benchmark called ModelSet500 and extensive experiments on ModelSet500 validate the effectiveness of the proposed method.

2 Related work

2.1 Transfer Learning

Transfer Learning [14, 2, 13, 16, 8, 18, 9, 12, 6] is to transfer knowledge from a source domain to a target domain, which has already achieved significant success in many areas including classification, regression, and clustering. Lots of approaches simply assume that source and target tasks are related or unrelated. For example, Tzeng et. al. [18] proposed to transfer the knowledge from RGB image based classification to depth image based classification. Liang et. al. [9] proposed to transfer the person re-identification knowledge from one scene to another scene. In addition, some methods jointly optimize multiple tasks and estimate task relations by cross-validation. For example, Rosenstein et al. [16] proposed to detect and avoid negative transfer using very little data from the target task and empirically showed that dissimilar tasks may hurt the performance of the target task. Zamir et. al. [20] proposed to transfer multiple different tasks to target tasks based on an affinity matrix of transferabilities. These approaches either assume that those tasks that are intuitively related have a positive impact on the transfer learning or attempt to learn task relation by jointly optimizing multiple labeled datasets. Different from these methods, the proposed method learns task relation from trained task-specific models.

2.2 Meta-learning/Learning to Learn

Meta-learning, also known as learning to learn, is a subfield of machine learning that focuses on automated learning algorithms. Recently, lots of approaches [21, 1, 4, 10, 11] aimed to automatically search hyper-parameters by using learning to learn models. For example, Zoph and Le [21] used a recurrent network to generate the model descriptions of neural networks and trained the RNN with reinforcement learning to maximize the expected accuracy of the generated architectures on a validation set. Liu et. al. [10] used a sequential model-based optimization strategy to search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Andrychowicz et. al. [1] attempted to learn an LSTM based neural optimizer to learn how to optimize neural networks. Finn et. al. [4] introduced a meta-learning method based on learning easily adaptable model parameters through gradient descent. However, these methods aim to automatically learn the better first-order similarity by optimizing hyper-parameters while the proposed method is to learn second-order similarity.

3 Second-Order Similarity

We first consider data similarity, which is also termed first-order similarity in this paper. For a learning task, let $D$ denote a training dataset with $n$ data points $\{x_{i},y_{i}\}_{i=1}^{n}$. Let $f(D;{\theta})$ be an objective function $f$ with parameters $\theta$ on $D$. The objective function is optimized by the standard gradient descent as follows

[TABLE]

where $\alpha_{t}$ is a step size multiplier at $t$. After a sequence of updates, the algorithm converges to a minimizer $\theta^{*}=argmin_{\theta}f(D;\theta)$. With the parameter $\theta^{*}$, the learning task can measure similarity between data points. In this way, given a set of learning tasks, we can obtain a set of trained task-specific parameters. Now the question is how to measure the similarity between different task-specific parameters?

Data similarity/metric learning aims to minimize the distance between intra-class data points and maximize the distance between inter-class data points. Similar to data similarity learning, the goal of parameter similarity/metric learning is to minimize the distance between intra-task models and maximize the distance between inter-task models, as illustrated in Figure 1. Intra-task models are obtained by repeating the training procedure many times for each task.

The intra-task models are diverse and contain rich statistical semantics due to the intractable non-convexity of deep networks and the coupled relationship between data points and parameters in convolution operation. Data points and parameters share the same semantics. For example, car classification and bus classification share a similar model while car classification and bird classification could produce dissimilar models. That is, the distance between car classification and bus classification models is closer than that between car classification and bird classification models. Learning to distinguish intra-task and inter-task parameters can uncover the underlying parameter patterns of local solutions and thus provides a parameter similarity metric for task relation.

Let $\mathcal{T}$ denote a set of tasks $\{\mathcal{T}_{i}\}_{i=1}^{N}$. For $\mathcal{T}_{i}\in\mathcal{T}$, we train $m_{i}$ deep models with a task label $\mathcal{Y}_{i}$. We obtain a model set $\mathcal{D}$ with $M$ models $\{(\theta^{*}_{j},\mathcal{Y}_{j})\}_{j=1}^{M}$ from $N$ tasks, where $M=\sum\limits{{m_{j}}}$. $\theta^{*}_{j}$ is a trained model, which can be regarded as a metadata point. We then use these $M$ metadata points to train the second-order similarity learning by

[TABLE]

where $h$ is a second-order similarity objective function with parameters $\phi$ on $\mathcal{D}$ and $\beta_{t}$ is a step size multiplier at $t$.

Second-order similarity learning is naturally different from existing learning to learn methods. First, the goal of second-order similarity learning is to learn parameter similarity based on first-order similarity while learning to learn methods learn hyper-parameters of an optimizer to navigate data similarity learning, which is naturally the first-order similarity learning. Second, second-order similarity learning optimizes model parameters and second-order model parameters (different from “hyper-parameter") separately while learning to learn methods optimize both model parameters and hyper-parameters simultaneously. Such a disjoint optimization method guarantees that trained deep models have converged to a good local solution.

4 Second-Order Neural Networks

In this section, we introduce a hierarchical second-order neural network to learn an effective parameter representation for second-order similarity, as shown in Figure 2. Specifically, the second-order network consists of $L$ branches to deal with $L$ semantic levels of $L$ layers. For each branch, we attempt to align metadata and then project it into a common space for parameter similarity learning with a fully connected layer. Finally, we weight the losses of $L$ different branches to control the relative importance.

Although the idea of parameter similarity learning is similar to data similarity learning, it is a challenging problem because of high-dimensional metadata points (higher than images, e.g., 96M for ResNet-50), changeable order of filters and their channels at a layer, and hierarchical semantics.

One of the most challenging problems is that the order of filters and their channels is changeable when repeating the training procedure. Therefore, directly using fully connected layers to project these metadata points into a common space cannot obtain good performance. We prove that the order of filters and their channels at a layer could be shuffled during different training procedures as follows.

Consider the convolution operation in visual tasks. Let $F_{l}$ be feature maps of size $(c_{l},H_{l},W_{l})$ at the $l$-th convolutional layer. $F_{l}^{c}$ is the $c$-th feature map of $F_{l}$ ($c\in[1,c_{l}]$). $\odot$ is the convolution operator. We re-use $\theta_{l}$ as filters of size $(g_{l},c_{l},h_{l},w_{l})$ at the $l$-th convolutional layer for simplification (do not be confused by $\theta_{l}$, $\theta_{j}^{*}$ and $\theta_{t}$ mentioned above). $\theta_{l}^{g}$ is the $g$-th filter of $\theta_{l}$ ($g\in[1,g_{l}]$). $\theta_{l}^{g,c}$ is the $c$-th channel of $\theta_{l}^{g}$. $\forall g\in[1,g_{l}]$, $\theta_{l}^{g}$ produces a feature map $F_{l+1}^{g}$. The convolution operation is given by

[TABLE]

In this way, $g_{l}$ filters produce $g_{l}$ feature maps as output. We can see that the convolution operation is independent of the order of $g_{l}$ filters $\{\theta_{l}^{g}\}_{g=1}^{g_{l}}$. Although $g_{l}$ shuffled filters at the $l$-layer naturally lead to $g_{l}$ shuffled channels of $F_{l}$, it does not reduce informative features. The convolution operation for each filter at a layer is symmetric. Therefore, the order of $\theta_{l}$ is changeable. We perform the next convolution operation by

[TABLE]

According to Eq. (3), shuffled ${\{\theta_{l}^{g}\}}_{g=1}^{g_{l}}$ leads to shuffled ${\{F_{l+1}^{g}\}}_{g=1}^{g_{l}}$, where ${g_{l}}=c_{l+1}$. According to Eq. (4), the order of $g_{l+1}$ channels of $F_{l+2}^{g}$ is determined by the order of ${\{F_{l+1}^{g}\}}_{g=1}^{g_{l}}$ and the channel order of the filter $\theta_{l+1}^{g}$. Because the order of ${\{F_{l+1}^{g}\}}_{g=1}^{g_{l}}$ is changed, the channel order of the filter $\theta_{l+1}^{g}$ have to be changed if we fix the channel order of the output $F_{l+2}$. For example, if we exchange the order of two filters $\theta_{l}^{g}$ and $\theta_{l}^{g^{{}^{\prime}}}$, and also exchange the channel $g$ and $g^{{}^{\prime}}$ of $\theta_{l+1}^{j}$ for any $j\in[1,g_{l}]$, the output of the network is fixed. That is, there are a large number of filter map combinations such that the output is the same.

Let $\mathcal{O}_{l}^{g}$ and $\mathcal{O}_{l}^{c}$ denote the filter order and channel order of $\theta_{l}$. We denote the order of $\theta_{l}$ as $(\mathcal{O}_{l}^{c},\mathcal{O}_{l}^{g})$. We define the order chain rule of convolutional filters as follows

• •

If $l=1$, then $\mathcal{O}_{l}^{c}$ is fixed because it is constrained by the channels of natural images, e.g., RGB. $\mathcal{O}_{l}^{g}$ determines $\mathcal{O}_{l+1}^{c}$

• •

If $l\geq 2$, then $\mathcal{O}_{l}^{c}$ is determined by $\mathcal{O}_{l-1}^{g}$ while $\mathcal{O}_{l}^{g}$ determines $\mathcal{O}_{l+1}^{c}$.

• •

If $l=L$, then $\mathcal{O}_{l}^{c}$ is determined $\mathcal{O}_{l-1}^{g}$. $\mathcal{O}_{l}^{g}$ is fixed because it is constrained by the loss of neural networks (or labels).

The order chain rule provides an ideal case that intra-task models are only affected by the changeable order of filters and their channels. In fact, intra-task deep models are also affected by hierarchical semantics and non-convex optimization. Therefore, give two deep models with a shared task label, it is difficult to align the changeable order of filters and their channels, which confuses parameter similarity learning.

To address this problem, we propose to transform $(\mathcal{O}_{l}^{c},\mathcal{O}_{l}^{g})$ of different intra-task models into a “standard" order to align these model parameters. Let ${\phi}_{l}$ is the second-order filters of size $(g_{l}^{{}^{\prime}},c_{l},h_{l},w_{l})$. ${\theta}_{l}$ and ${\phi}_{l}$ could have different number filters at a layer. $\forall{c_{i}},{c_{j}}\in[1,c_{l}]$, $\forall g_{i}\in[1,g_{l}]$, $\forall g_{j}^{{}^{\prime}}\in[1,{g_{l}^{{}^{\prime}}}]$ and $\forall l\in[2,L]$, we first compute the Frobenius inner product between filter maps and second-order filter maps by

[TABLE]

where $\otimes$ denotes the Frobenius inner product. In Eq. (5), we compute the Frobenius inner product between all pairs of $\theta_{l}^{g_{i}}$ and $\phi_{l}^{g_{j}^{{}^{\prime}}}$ filter maps. Without any prior information, we cannot directly compute the standard $\mathcal{O}_{l}^{c}$. We estimate the standard $\mathcal{O}_{l}^{c}$ based on the max matching between $\theta_{l}^{g_{i}}$ and $\phi_{l}^{g_{j}^{{}^{\prime}}}$. We have

[TABLE]

When $l=1$, we do not need to align $\mathcal{O}_{l}^{c}$ by the max operation. $\forall{c}\in[1,c_{l}]$, $\forall g_{i}\in[1,g_{l}]$, and $\forall g_{j}^{{}^{\prime}}\in[1,{g_{l}^{{}^{\prime}}}]$, we directly compute $s_{l}^{g_{i},g_{j}^{{}^{\prime}}}$ by

[TABLE]

$\forall l\in[1,L-1]$, we then estimate the standard $\mathcal{O}_{l}^{g}$ by using the sort operation

[TABLE]

where $r_{1},r_{2},...,r_{g_{l}}$ is the new filter order indexes. Based on the order chain rule, $\mathcal{O}_{l}^{c}$ is fixed when $l=L$. Therefore, we do not need to align $\mathcal{O}_{l}^{g}$.

After aligning $\mathcal{O}_{l}^{g}$, we add a fully connected layer to project the aligned parameter representation into common space. Finally, we train a second-order model by using a conventional metric learning loss, e.g., cross-entropy loss [17] or triplet loss [3]. In this paper, we simply use the cross-entropy loss for model similarity measure

[TABLE]

where ${{\mathcal{Y}_{i}}}$ is the $i$-dimensional value of the one-hot label $\mathcal{Y}$. $q_{i}^{l}$ represents the probability of the $i$-th visual task at the $l$-th branch. $w_{l}$ controls the relative importance of $L$ branches. $W=\sum\limits_{l=1}^{L}{{w_{l}}}$. When measuring the parameter similarity between two deep models, we extract parameter features from the last fully connected layer of the second-order network. We normalize parameter features and use the cosine similarity to compute the parameter similarity.

5 Experiments

In this section, we conduct extensive experiments to validate the effectiveness of our proposed method. All experiments are conducted with Pytorch [15].

Dataset and Modelset. The CIFAR-100 dataset [7] has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.

To measure model-based parameter similarity, we create a model set called ModelSet500 based on CIFAR-100. Specially, we split 100 classes into 50 groups as 50 visual tasks with task labels 0$\sim$49. Each task contains 2 classes. For each task, we repeat the training procedure 10 times to obtain 10 deep models. Finally, we obtain 500 deep models for 50 tasks.

First-order and Second-order Network Implementation. We implement a ResNet-20 network [5] as the first-order network in our experiments. The network consists of one convolutional layer, three residual blocks, one global average pooling layer, and one fully-connected layer. The blocks of ResNet-20 consist of 6 convolutional layers, respectively (excluding $1\times 1$ convolutional layers which are used for downsampling in residual blocks). When computing the parameter similarity measure, we omit the last fully-connected layer and $1\times 1$ convolutional layers. We use SGD with a mini-batch size of 128. The learning rate starts from 0.1 and is divided by 10 after 80 epochs and 120 epochs, respectively. We train ResNet-20 for 160 epochs. We use a weight decay of 0.0001 and a momentum of 0.9. We set the size of padding to 4 and perform random cropping with $32\times 32$ and random horizontal flip.

When implementing the second-order network, we use SGD with a mini-batch size 1. The learning rate starts from 0.001 and is divided by 10 after 40 epochs and 80 epochs, respectively. We train 100 epochs. We set the loss weights $w_{1}=1.0$ and $w_{2}=w_{3}=...=w_{9}=0.0$ in Section 5.1  5.2 and 5.3 because the first branch plays a major role in Eq. (9). We analyze the performance of different branches in Section 5.4 and 5.5.

5.1 Task classification

We first adopt the conventional image classification evaluation metric for task classification on ModelSet500. The goal of task classification is to predict task labels of trained models. For each task, we sample eight deep models for training while the other two for test. Therefore, the training set contains 400 deep models while the test set contains 100 deep models. We train the second-order network by classifying 50 task classes.

To evaluate the effectiveness of the second-order network, we set three baselines, i.e., random prediction without using any second-order model, only fully connected layer, Frobenius inner product without aligning filters and their channels. Compared with these three baselines, we can see that the proposed method achieves better performance. As shown in Table 1, our method achieves 79.8% top-1 accuracy while the three baselines are lower than 6%. The main reason may be that the changeable order of filters and their channels confuses the second-order neural networks.

5.2 Task retrieval

We then adopt the conventional image retrieval evaluation metric for task retrieval on ModelSet500. Given a query deep model, the goal of task retrieval is to retrieve the target deep models from a gallery. We split ModelSet500 into a training set and a test set. The training set includes 40 tasks and the test set includes 10 tasks. For each task in the test set, we sample two deep models for query models while the others are used for gallery models. Task retrieval is challenging and practical in real-world scenarios. It can be used to retrieve similar visual tasks for transfer learning. Different from task classification, we train the second-order network by classifying 40 task classes and extract the fully connected layer as parameter representation during the test phase. We use the normalized parameter representation for task retrieval.

We also set three baselines like task classification. Compared with these three baselines, we can see that the proposed method achieves better performance. As shown in Table 2, our method achieves 80.7% rank-1 accuracy while the three baselines are lower than 15%. The reason is the same as Section 5.1.

5.3 Task transferability

To validate that the parameter similarity can be used to measure task transferability, we conduct an experiment to study task transferability with respect to parameter similarity. We use the task retrieval setting to obtain parameter representation, as discussed in Section 5.2. We evaluate task transferability by training first-order models for one task (a labeled source domain) and applying these models to another task (an unlabeled target domain) for data feature extraction and k-means clustering. We use adjusted rand index (ARI) [19] for evaluation, which is widely used in cluster analysis.

As shown in Figure 3a, although there are some outliers, we observe that parameter similarity is roughly proportional to task transferability (ARI). In most of the cases, the higher parameter similarity score the source and target tasks have, the better task transferability they can achieve.

5.4 Effect of different branches

To show the relative importance of different branches for the parameter similarity measure, we conduct an experiment on ModelSet500 by keeping one of $L$ branches while removing the other $L-1$ branches. That is, when evaluating the $l^{{}^{\prime}}$-th branch, we set $w_{l}=1.0$ if $l=l^{{}^{\prime}}$, otherwise $w_{l}=0.0$. We use the task classification setting. Different from other experiments, the learning rate in this experiment is divided by 10 after 150 epochs and 225 epochs, respectively. We train each branch for 300 epochs because it takes more time to align the order of both filters and their channels.

As shown in Figure 3b, it is observed that the first branch is the most important branch and it can achieve 79.8% top-1 accuracy. From the 2rd to 13th branch, the second-order network still can learn a weak discriminative parameter representation since the results are higher than top-1 accuracy of random prediction, i.e., 2.0% (50 classes). However, from the 13th to 19th layer, the second-order cannot learn a discriminative parameter representation since the results are nearly reduced to 2.0% top-1 accuracy. The reason could be that the order chain rule leads to the cumulative error of $(\mathcal{O}_{l}^{c},\mathcal{O}_{l}^{g})$ alignment at higher layers.

Besides, it is still hard to fuse all of the branches to improve better performance because the 2rd to 19th branch is much lower. This is the reason why we only use the first branch for task classification and retrieval.

5.5 Effectiveness of the alignment methods

To show the effectiveness of the alignment methods, we conduct an experiment by isolating different alignment methods, i.e., filter alignment (filt.) and channel alignment (chan.). The training setting is the same as Section 5.4. We only analyze the 2rd to 19th branch because the 1st branch only contains one unaligned case, i.e., unaligned filters (see Eq. (7)). As shown in Figure 3c, it is observed that both alignment methods are important for parameter similarity learning.

6 Advantages and disadvantages

The proposed method comes with advantages and disadvantages compared with the existing methods. The advantages are that the learnable parameter similarity model does not need to jointly train a large number of transfer networks to estimate task relations. The second-order network does not need to align convolutional filters if the first-order network does not contain convolutional layers in some non-image scenarios. Learnable parameter similarity can be also used to quickly retrieve related tasks from the model market (set) for transfer learning.

The disadvantages are that the proposed learnable parameter similarity cannot measure the similarity between different networks and it is still hard to fuse all of the branches to improve better performance. The reason could be that the alignment at higher layers are still unsatisfactory.

7 Conclusion

In this paper, we present a Learnable Parameter Similarity (LPS) method to reveal the underlying relations between different visual tasks. Our approach learns task relations by using task-specific trained deep models instead of jointly training a large number of transfer networks. We present a hierarchical second-order network to deal with high-dimensional unaligned deep models. We evaluate the LPS method on a new parameter similarity learning benchmark ModelSet500 and extensive experiments show the effectiveness of the proposed method. Future work is to further explore the fusion of different layers and the alignment of the convolutional filters at higher layers and develop a parameter similarity metric for heterogeneous networks.