Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules

TL;DR

This paper introduces a novel neural network approach that generates images through sequential synaptic learning rules, making the process interpretable and visualizable, and enhances image quality with a convolutional U-Net.

Contribution

The work demonstrates how synaptic learning rules can produce human-interpretable images and integrates a U-Net for denoising, bridging fast weight programming with visual interpretability.

Findings

Generated images have respectable visual quality without explicit image biases.

The approach offers human-interpretable insights into synaptic learning processes.

Adding a U-Net improves the quality of generated images.

Abstract

Work on fast weight programmers has demonstrated the effectiveness of key/value outer product-based learning rules for sequentially generating a weight matrix (WM) of a neural net (NN) by another NN or itself. However, the weight generation steps are typically not visually interpretable by humans, because the contents stored in the WM of an NN are not. Here we apply the same principle to generate natural images. The resulting fast weight painters (FPAs) learn to execute sequences of delta learning rules to sequentially generate images as sums of outer products of self-invented keys and values, one rank at a time, as if each image was a WM of an NN. We train our FPAs in the generative adversarial networks framework, and evaluate on various image datasets. We show how these generic learning rules can generate images with respectable visual quality without any explicit inductive bias for images. While the performance largely lags behind the one of specialised state-of-the-art image generators, our approach allows for visualising how synaptic learning rules iteratively produce complex connection patterns, yielding human-interpretable meaningful images. Finally, we also show that an additional convolutional U-Net (now popular in diffusion models) at the output of an FPA can learn one-step "denoising" of FPA-generated images to enhance their quality. Our code is public.