Reverse Back Propagation to Make Full Use of Derivative

TL;DR

This paper introduces a reverse back-propagation method that reuses the traditional process to optimize input loss, improving learning efficiency and effectiveness without additional inference costs.

Contribution

It proposes a novel reverse back-propagation approach, analyzes its principles, and reformulates weight initialization to enhance neural network training.

Findings

Better adaptation to larger learning rates

Improved learning performance over vanilla back-propagation

Effective on datasets like MNIST, CIFAR10, CIFAR100

Abstract

The development of the back-propagation algorithm represents a landmark in neural networks. We provide an approach that conducts the back-propagation again to reverse the traditional back-propagation process to optimize the input loss at the input end of a neural network for better effects without extra costs during the inference time. Then we further analyzed its principles and advantages and disadvantages, reformulated the weight initialization strategy for our method. And experiments on MNIST, CIFAR10, and CIFAR100 convinced our approaches could adapt to a larger range of learning rate and learn better than vanilla back-propagation.