LGSQE: Lightweight Generated Sample Quality Evaluatoin

TL;DR

LGSQE is a lightweight method for evaluating the quality of individual generated samples using a trained classifier, providing a simple and efficient alternative to existing metrics like FID.

Contribution

The paper introduces LGSQE, a novel, low-complexity approach for assessing sample quality in generative models, applicable as a post-processing tool and for model evaluation.

Findings

LGSQE correlates well with FID in ranking generative models.

It achieves comparable evaluation performance with significantly lower complexity.

Experiments on CIFAR-10 and MNIST validate its effectiveness.

Abstract

Despite prolific work on evaluating generative models, little research has been done on the quality evaluation of an individual generated sample. To address this problem, a lightweight generated sample quality evaluation (LGSQE) method is proposed in this work. In the training stage of LGSQE, a binary classifier is trained on real and synthetic samples, where real and synthetic data are labeled by 0 and 1, respectively. In the inference stage, the classifier assigns soft labels (ranging from 0 to 1) to each generated sample. The value of soft label indicates the quality level; namely, the quality is better if its soft label is closer to 0. LGSQE can serve as a post-processing module for quality control. Furthermore, LGSQE can be used to evaluate the performance of generative models, such as accuracy, AUC, precision and recall, by aggregating sample-level quality. Experiments are conducted on CIFAR-10 and MNIST to demonstrate that LGSQE can preserve the same performance rank order as that predicted by the Frechet Inception Distance (FID) but with significantly lower complexity.