Using UNet and PSPNet to explore the reusability principle of CNN parameters

TL;DR

This paper investigates the reusability of CNN parameters in deep networks like UNet and PSPNet for segmentation and auto-encoder tasks, revealing insights into which parameters can be effectively reused to reduce training data needs.

Contribution

It experimentally quantifies parameter reusability in CNN layers, highlighting differences between Batch Normalization and Convolution layers, and clarifies reasons behind successful transfer learning.

Findings

Batch normalization running mean and variance are crucial for reusability.

Convolution layer weights are highly sensitive and less reusable.

Biases in convolution layers are easily reusable.

Abstract

How to reduce the requirement on training dataset size is a hot topic in deep learning community. One straightforward way is to reuse some pre-trained parameters. Some previous work like Deep transfer learning reuse the model parameters trained for the first task as the starting point for the second task, and semi-supervised learning is trained upon a combination of labeled and unlabeled data. However, the fundamental reason of the success of these methods is unclear. In this paper, the reusability of parameters in each layer of a deep convolutional neural network is experimentally quantified by using a network to do segmentation and auto-encoder task. This paper proves that network parameters can be reused for two reasons: first, the network features are general; Second, there is little difference between the pre-trained parameters and the ideal network parameters. Through the use of parameter replacement and comparison, we demonstrate that reusability is different in BN(Batch Normalization)[7] layer and Convolution layer and some observations: (1)Running mean and running variance plays an important role than Weight and Bias in BN layer.(2)The weight and bias can be reused in BN layers.( 3) The network is very sensitive to the weight of convolutional layer.(4) The bias in Convolution layers are not sensitive, and it can be reused directly.