Mitigating Gradient Overlap in Deep Residual Networks with Gradient Normalization for Improved Non-Convex Optimization

TL;DR

This paper introduces Z-score Normalization (ZNorm) to mitigate gradient overlap in ResNets, improving training efficiency and optimization in deep, non-convex neural networks.

Contribution

The paper proposes ZNorm as a novel technique to standardize gradients and reduce overlap effects in ResNets, enhancing training in deep non-convex models.

Findings

ZNorm reduces gradient overlap and improves training stability.

ZNorm enhances optimization efficiency in deep residual networks.

ZNorm leads to better convergence in non-convex scenarios.

Abstract

In deep learning, Residual Networks (ResNets) have proven effective in addressing the vanishing gradient problem, allowing for the successful training of very deep networks. However, skip connections in ResNets can lead to gradient overlap, where gradients from both the learned transformation and the skip connection combine, potentially resulting in overestimated gradients. This overestimation can cause inefficiencies in optimization, as some updates may overshoot optimal regions, affecting weight updates. To address this, we examine Z-score Normalization (ZNorm) as a technique to manage gradient overlap. ZNorm adjusts the gradient scale, standardizing gradients across layers and reducing the negative impact of overlapping gradients. Our experiments demonstrate that ZNorm improves training process, especially in non-convex optimization scenarios common in deep learning, where finding optimal solutions is challenging. These findings suggest that ZNorm can affect the gradient flow, enhancing performance in large-scale data processing where accuracy is critical.