On the Transferability of Representations in Neural Networks Between Datasets and Tasks

TL;DR

This paper investigates how well different layers of deep neural networks transfer learned features across various datasets and tasks, providing empirical insights into the hierarchical nature of learned representations.

Contribution

It offers a systematic study of layer-wise transferability in deep networks across multiple datasets and tasks, highlighting empirical patterns and insights.

Findings

Lower layers transfer better across tasks

Higher layers are more task-specific

Transferability varies with dataset similarity

Abstract

Deep networks, composed of multiple layers of hierarchical distributed representations, tend to learn low-level features in initial layers and transition to high-level features towards final layers. Paradigms such as transfer learning, multi-task learning, and continual learning leverage this notion of generic hierarchical distributed representations to share knowledge across datasets and tasks. Herein, we study the layer-wise transferability of representations in deep networks across a few datasets and tasks and note some interesting empirical observations.