Locally Supervised Learning with Periodic Global Guidance

TL;DR

This paper introduces Periodically Guided Local Learning (PGL), a training method that periodically reintroduces the global objective into locally supervised neural networks to improve their generalization performance with minimal additional memory.

Contribution

The paper proposes PGL, a novel training scheme that enhances generalization in locally supervised neural networks by periodically incorporating global guidance.

Findings

PGL significantly improves generalization performance.

PGL achieves these gains with low memory overhead.

Experimental results confirm effectiveness across datasets and architectures.

Abstract

Locally supervised learning aims to train a neural network based on a local estimation of the global loss function at each decoupled module of the network. Auxiliary networks are typically appended to the modules to approximate the gradient updates based on the greedy local losses. Despite being advantageous in terms of parallelism and reduced memory consumption, this paradigm of training severely degrades the generalization performance of neural networks. In this paper, we propose Periodically Guided local Learning (PGL), which reinstates the global objective repetitively into the local-loss based training of neural networks primarily to enhance the model's generalization capability. We show that a simple periodic guidance scheme begets significant performance gains while having a low memory footprint. We conduct extensive experiments on various datasets and networks to demonstrate the effectiveness of PGL, especially in the configuration with numerous decoupled modules.