Demystifying Randomly Initialized Networks for Evaluating Generative Models

TL;DR

This paper investigates the effectiveness of using randomly initialized neural networks as feature extractors for evaluating generative models, providing theoretical and empirical insights into their reliability and complementarity with trained networks.

Contribution

It offers a rigorous analysis of random network features versus trained features and empirically demonstrates their comparable effectiveness in generative model evaluation.

Findings

Random features evaluate generative models effectively.

Random and trained features can be used complementarily.

Empirical guidelines for selecting random networks are provided.

Abstract

Evaluation of generative models is mostly based on the comparison between the estimated distribution and the ground truth distribution in a certain feature space. To embed samples into informative features, previous works often use convolutional neural networks optimized for classification, which is criticized by recent studies. Therefore, various feature spaces have been explored to discover alternatives. Among them, a surprising approach is to use a randomly initialized neural network for feature embedding. However, the fundamental basis to employ the random features has not been sufficiently justified. In this paper, we rigorously investigate the feature space of models with random weights in comparison to that of trained models. Furthermore, we provide an empirical evidence to choose networks for random features to obtain consistent and reliable results. Our results indicate that the features from random networks can evaluate generative models well similarly to those from trained networks, and furthermore, the two types of features can be used together in a complementary way.