Tourbillon: a Physically Plausible Neural Architecture

TL;DR

Tourbillon introduces a neural architecture that overcomes key physical constraints of backpropagation, using self-supervised autoencoders and non-symmetric connections, achieving competitive results on standard benchmarks.

Contribution

It presents a novel, physically plausible neural architecture combining circular autoencoders and non-symmetric connections, addressing backpropagation limitations.

Findings

Achieves comparable performance to backpropagation on MNIST, Fashion MNIST, CIFAR10.

Outperforms other physically plausible algorithms like feedback alignment.

Demonstrates viability of the architecture on standard datasets.

Abstract

In a physical neural system, backpropagation is faced with a number of obstacles including: the need for labeled data, the violation of the locality learning principle, the need for symmetric connections, and the lack of modularity. Tourbillon is a new architecture that addresses all these limitations. At its core, it consists of a stack of circular autoencoders followed by an output layer. The circular autoencoders are trained in self-supervised mode by recirculation algorithms and the top layer in supervised mode by stochastic gradient descent, with the option of propagating error information through the entire stack using non-symmetric connections. While the Tourbillon architecture is meant primarily to address physical constraints, and not to improve current engineering applications of deep learning, we demonstrate its viability on standard benchmark datasets including MNIST, Fashion MNIST, and CIFAR10. We show that Tourbillon can achieve comparable performance to models trained with backpropagation and outperform models that are trained with other physically plausible algorithms, such as feedback alignment.