A Theoretical View of Linear Backpropagation and Its Convergence

TL;DR

This paper provides a theoretical analysis of LinBP, a linear variant of backpropagation, showing it can converge faster than traditional BP in neural network training and adversarial tasks, supported by extensive experiments.

Contribution

It offers the first theoretical convergence analysis of LinBP and demonstrates its advantages over BP in neural network training and adversarial attack scenarios.

Findings

LinBP can lead to faster convergence than BP.

Theoretical analysis confirms empirical advantages of LinBP.

Extensive experiments validate the convergence benefits.

Abstract

Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.