Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance

TL;DR

This paper introduces a self-calibrating classifier guidance method for score-based generative models, improving class-conditional image generation quality when labeled data is scarce by regularizing the classifier with energy-based principles.

Contribution

It proposes a novel self-calibration technique that uses energy-based model principles to regularize classifiers, enhancing conditional generation with limited labeled data.

Findings

Significant improvement in conditional generation quality with fewer labels.

Effective regularization of classifiers using unlabeled data.

Enhanced stability and accuracy in class-conditional image synthesis.

Abstract

Score-based generative models (SGMs) are a popular family of deep generative models that achieve leading image generation quality. Early studies extend SGMs to tackle class-conditional generation by coupling an unconditional SGM with the guidance of a trained classifier. Nevertheless, such classifier-guided SGMs do not always achieve accurate conditional generation, especially when trained with fewer labeled data. We argue that the problem is rooted in the classifier's tendency to overfit without coordinating with the underlying unconditional distribution. To make the classifier respect the unconditional distribution, we propose improving classifier-guided SGMs by letting the classifier regularize itself. The key idea of our proposed method is to use principles from energy-based models to convert the classifier into another view of the unconditional SGM. Existing losses for unconditional SGMs can then be leveraged to achieve regularization by calibrating the classifier's internal unconditional scores. The regularization scheme can be applied to not only the labeled data but also unlabeled ones to further improve the classifier. Across various percentages of fewer labeled data, empirical results show that the proposed approach significantly enhances conditional generation quality. The enhancements confirm the potential of the proposed self-calibration technique for generative modeling with limited labeled data.