Designing Network Design Strategies Through Gradient Path Analysis

TL;DR

This paper introduces a novel network design approach based on analyzing gradient paths during backpropagation, aiming to improve model expressiveness and learning ability.

Contribution

It proposes gradient path analysis as a new strategy for designing network architectures, focusing on backpropagation paths rather than data paths.

Findings

Gradient path design strategies are theoretically justified.

Experimental results show improved network expressiveness.

Strategies are effective at layer, stage, and network levels.

Abstract

Designing a high-efficiency and high-quality expressive network architecture has always been the most important research topic in the field of deep learning. Most of today's network design strategies focus on how to integrate features extracted from different layers, and how to design computing units to effectively extract these features, thereby enhancing the expressiveness of the network. This paper proposes a new network design strategy, i.e., to design the network architecture based on gradient path analysis. On the whole, most of today's mainstream network design strategies are based on feed forward path, that is, the network architecture is designed based on the data path. In this paper, we hope to enhance the expressive ability of the trained model by improving the network learning ability. Due to the mechanism driving the network parameter learning is the backward propagation algorithm, we design network design strategies based on back propagation path. We propose the gradient path design strategies for the layer-level, the stage-level, and the network-level, and the design strategies are proved to be superior and feasible from theoretical analysis and experiments.