On Random Weights for Texture Generation in One Layer Neural Networks

TL;DR

This paper investigates why one-layer CNNs with random weights can effectively generate textures, providing theoretical insights into their frequency modulation capabilities and the impact of non-linearities like ReLU.

Contribution

It offers a theoretical explanation for the effectiveness of random-weight one-layer CNNs in texture generation and analyzes the effects of ReLU non-linearity on image variability.

Findings

Random weights in one-layer CNNs can preserve and modulate frequency components.

Without non-linearity, there are infinite solutions for minimal energy.

ReLU non-linearity can reduce image variability by limiting solutions.

Abstract

Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures. More interestingly, it has also been experimentally shown that only one layer with random filters can also model textures although with less variability. In this paper we ask the question as to why one layer CNNs with random filters are so effective in generating textures? We theoretically show that one layer convolutional architectures (without a non-linearity) paired with the an energy function used in previous literature, can in fact preserve and modulate frequency coefficients in a manner so that random weights and pretrained weights will generate the same type of images. Based on the results of this analysis we question whether similar properties hold in the case where one uses one convolution layer with a non-linearity. We show that in the case of ReLu non-linearity there are situations where only one input will give the minimum possible energy whereas in the case of no nonlinearity, there are always infinite solutions that will give the minimum possible energy. Thus we can show that in certain situations adding a ReLu non-linearity generates less variable images.