Recurrent Joint Embedding Predictive Architecture with Recurrent Forward Propagation Learning

TL;DR

This paper proposes a biologically inspired recurrent vision network that predicts future image patches using a novel learning algorithm, Recurrent-Forward Propagation, which avoids traditional backpropagation complexities.

Contribution

It introduces a new recurrent joint embedding predictive architecture with a biologically plausible learning algorithm that ensures efficient training without representational collapse.

Findings

Mathematically proven that the algorithm implements exact gradient descent.

Empirical evidence shows the network learns efficiently.

Avoids backpropagation through time, making it more biologically plausible.

Abstract

Conventional computer vision models rely on very deep, feedforward networks processing whole images and trained offline with extensive labeled data. In contrast, biological vision relies on comparatively shallow, recurrent networks that analyze sequences of fixated image patches, learning continuously in real-time without explicit supervision. This work introduces a vision network inspired by these biological principles. Specifically, it leverages a joint embedding predictive architecture incorporating recurrent gated circuits. The network learns by predicting the representation of the next image patch (fixation) based on the sequence of past fixations, a form of self-supervised learning. We show mathematical and empirically that the training algorithm avoids the problem of representational collapse. We also introduce \emph{Recurrent-Forward Propagation}, a learning algorithm that avoids biologically unrealistic backpropagation through time or memory-inefficient real-time recurrent learning. We show mathematically that the algorithm implements exact gradient descent for a large class of recurrent architectures, and confirm empirically that it learns efficiently. This paper focuses on these theoretical innovations and leaves empirical evaluation of performance in downstream tasks, and analysis of representational similarity with biological vision for future work.