Feature discriminativity estimation in CNNs for transfer learning

TL;DR

This paper analyzes the thresholds used to select features in CNNs for transfer learning, discovering a strong correlation with problem size, and proposes a unified model for estimating these thresholds to improve transfer learning efficiency.

Contribution

It introduces a methodology for estimating feature discriminativity thresholds in CNNs based on problem size, enhancing transfer learning performance.

Findings

Strong correlation (R^2 > 90%) between problem size and threshold value.

Proposed a unified model for threshold estimation.

Potential to improve transfer learning by better feature selection.

Abstract

The purpose of feature extraction on convolutional neural networks is to reuse deep representations learnt for a pre-trained model to solve a new, potentially unrelated problem. However, raw feature extraction from all layers is unfeasible given the massive size of these networks. Recently, a supervised method using complexity reduction was proposed, resulting in significant improvements in performance for transfer learning tasks. This approach first computes the discriminative power of features, and then discretises them using thresholds computed for the task. In this paper, we analyse the behaviour of these thresholds, with the purpose of finding a methodology for their estimation. After a comprehensive study, we find a very strong correlation between problem size and threshold value, with coefficient of determination above 90%. These results allow us to propose a unified model for threshold estimation, with potential application to transfer learning tasks.