Fixed-Weight Difference Target Propagation

TL;DR

This paper introduces Fixed-Weight Difference Target Propagation (FW-DTP), a biologically plausible training method for deep networks that maintains fixed feedback weights, simplifying training while achieving better performance than traditional methods.

Contribution

The paper proposes FW-DTP, which keeps feedback weights constant during training, eliminating the need for layer-wise autoencoders and frequent feedback updates, thus improving practicality and performance.

Findings

FW-DTP outperforms baseline DTP on four classification datasets.

FW-DTP maintains effective target propagation with fixed feedback weights.

The method simplifies training and addresses autoencoder-related drawbacks.

Abstract

Target Propagation (TP) is a biologically more plausible algorithm than the error backpropagation (BP) to train deep networks, and improving practicality of TP is an open issue. TP methods require the feedforward and feedback networks to form layer-wise autoencoders for propagating the target values generated at the output layer. However, this causes certain drawbacks; e.g., careful hyperparameter tuning is required to synchronize the feedforward and feedback training, and frequent updates of the feedback path are usually required than that of the feedforward path. Learning of the feedforward and feedback networks is sufficient to make TP methods capable of training, but is having these layer-wise autoencoders a necessary condition for TP to work? We answer this question by presenting Fixed-Weight Difference Target Propagation (FW-DTP) that keeps the feedback weights constant during training. We confirmed that this simple method, which naturally resolves the abovementioned problems of TP, can still deliver informative target values to hidden layers for a given task; indeed, FW-DTP consistently achieves higher test performance than a baseline, the Difference Target Propagation (DTP), on four classification datasets. We also present a novel propagation architecture that explains the exact form of the feedback function of DTP to analyze FW-DTP.