Investigating Learning in Deep Neural Networks using Layer-Wise Weight Change

TL;DR

This paper explores how different layers in deep CNNs change their weights during training, revealing that later layers tend to undergo more significant relative weight changes across various architectures and tasks.

Contribution

It introduces a method to measure per-layer weight change during training and uncovers consistent trends across multiple CNN architectures and vision tasks.

Findings

Later layers exhibit greater relative weight change than earlier layers.

Weight change patterns are consistent across different CNN architectures.

Insights may inform improved training strategies for deep neural networks.

Abstract

Understanding the per-layer learning dynamics of deep neural networks is of significant interest as it may provide insights into how neural networks learn and the potential for better training regimens. We investigate learning in Deep Convolutional Neural Networks (CNNs) by measuring the relative weight change of layers while training. Several interesting trends emerge in a variety of CNN architectures across various computer vision classification tasks, including the overall increase in relative weight change of later layers as compared to earlier ones.