Top-down inference in an early visual cortex inspired hierarchical Variational Autoencoder

TL;DR

This paper introduces hierarchical Variational Autoencoders inspired by the visual cortex to model hierarchical computations in early visual processing, demonstrating emergent neural-like representations and the importance of top-down inference.

Contribution

It develops a neuroscience-inspired hierarchical VAE architecture that captures cortical-like representations and highlights the significance of top-down inference in visual modeling.

Findings

Emergence of cortical-like representations in VAE models

Top-down inference is crucial for learning complex posterior moments

High-level texture representations are stable across architectures

Abstract

Interpreting computations in the visual cortex as learning and inference in a generative model of the environment has received wide support both in neuroscience and cognitive science. However, hierarchical computations, a hallmark of visual cortical processing, has remained impervious for generative models because of a lack of adequate tools to address it. Here we capitalize on advances in Variational Autoencoders (VAEs) to investigate the early visual cortex with sparse coding hierarchical VAEs trained on natural images. We design alternative architectures that vary both in terms of the generative and the recognition components of the two latent-layer VAE. We show that representations similar to the one found in the primary and secondary visual cortices naturally emerge under mild inductive biases. Importantly, a nonlinear representation for texture-like patterns is a stable property of the high-level latent space resistant to the specific architecture of the VAE, reminiscent of the secondary visual cortex. We show that a neuroscience-inspired choice of the recognition model, which features a top-down processing component is critical for two signatures of computations with generative models: learning higher order moments of the posterior beyond the mean and image inpainting. Patterns in higher order response statistics provide inspirations for neuroscience to interpret response correlations and for machine learning to evaluate the learned representations through more detailed characterization of the posterior.