Can Local Learning Match Self-Supervised Backpropagation?

TL;DR

This paper develops a theoretical link between local self-supervised learning algorithms and global backpropagation, proposing novel local methods that match the performance of global algorithms on image datasets.

Contribution

The paper introduces a theoretical framework connecting local and global self-supervised learning, and proposes new local algorithms that achieve comparable performance to backpropagation-based methods.

Findings

Local-SSL algorithms can implement the same weight updates as global BP-SSL under certain conditions.

Novel local-SSL variants improve similarity of gradient updates to global BP-SSL.

Best local-SSL method matches global BP-SSL performance on CIFAR-10, STL-10, Tiny ImageNet.

Abstract

While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.