Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data

TL;DR

This paper investigates biologically plausible local learning rules for neural networks, demonstrating that self-supervised contrastive and non-contrastive methods can learn hierarchical structures efficiently, unlike direct feedback rules.

Contribution

It introduces and compares two types of local learning rules, showing that self-supervised contrastive and non-contrastive rules successfully learn hierarchical data structures.

Findings

Self-supervised contrastive and non-contrastive rules learn hierarchical structures efficiently.

Direct feedback rules fail to learn the tasks due to input-specific nonlinearities.

Self-supervised rules are as data-efficient as supervised backpropagation.

Abstract

The brain learns abstract representations of high-dimensional sensory input, but the plasticity rules that enable such learning are unknown. We study biologically plausible algorithms on the Random Hierarchy Model (RHM), an artificial dataset designed to investigate how deep neural networks learn the intrinsic hierarchical structure of high-dimensional data. We focus on two types of local learning rules that avoid both a long convergence time and the use of a symmetric error network. The first type uses direct feedback signals to approximate error propagation from the output layer. The second type uses layerwise self-supervised contrastive or non-contrastive loss functions that do not explicitly approximate errors at the output layer. We show that all rules of the first type fail to solve the tasks of the RHM and trace this failure back to input-specific nonlinearities (`masking') that are implemented in full backpropagation and are essential for learning complex tasks. However, algorithms of the second type are able to learn the hierarchical hidden structure of the RHM tasks and are as data-efficient as supervised backpropagation training, while being compatible with known rules of synaptic plasticity in cortex.