Understanding the Benefits of Image Augmentations

TL;DR

This paper investigates how image augmentations influence different layers of residual neural networks using CKA, revealing layer-specific effects based on network depth, initialization, and augmentation type, with implications for transfer learning and layer freezing.

Contribution

It provides a detailed analysis of layer-wise effects of image augmentations on ResNets, highlighting the impact of network depth, initialization, and augmentation type on learned weights.

Findings

Deeper layers are more affected by augmentations when initialized with ImageNet weights.

Augmentations using two images influence weights more than single-image augmentations.

Layer effects vary with network depth and training method.

Abstract

Image Augmentations are widely used to reduce overfitting in neural networks. However, the explainability of their benefits largely remains a mystery. We study which layers of residual neural networks (ResNets) are most affected by augmentations using Centered Kernel Alignment (CKA). We do so by analyzing models of varying widths and depths, as well as whether their weights are initialized randomly or through transfer learning. We find that the pattern of how the layers are affected depends on the model's depth, and that networks trained with augmentation that use information from two images affect the learned weights significantly more than augmentations that operate on a single image. Deeper layers of ResNets initialized with ImageNet-1K weights and fine-tuned receive more impact from the augmentations than early layers. Understanding the effects of image augmentations on CNNs will have a variety of applications, such as determining how far back one needs to fine-tune a network and which layers should be frozen when implementing layer freezing algorithms.