MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data

TL;DR

This paper introduces MS$^3$D, a novel RG flow-based regularization method that improves GAN training stability and quality when limited data is available by enforcing multi-scale gradient consistency.

Contribution

The paper proposes a new regularization technique based on RG flow principles, addressing overfitting in GANs trained with limited data, which is a novel approach in this context.

Findings

Enhances GAN stability with limited data

Enables high-quality image generation from few samples

Reduces overfitting and mode collapse

Abstract

Generative adversarial networks (GANs) have made impressive advances in image generation, but they often require large-scale training data to avoid degradation caused by discriminator overfitting. To tackle this issue, we investigate the challenge of training GANs with limited data, and propose a novel regularization method based on the idea of renormalization group (RG) in physics.We observe that in the limited data setting, the gradient pattern that the generator obtains from the discriminator becomes more aggregated over time. In RG context, this aggregated pattern exhibits a high discrepancy from its coarse-grained versions, which implies a high-capacity and sensitive system, prone to overfitting and collapse. To address this problem, we introduce a \textbf{m}ulti-\textbf{s}cale \textbf{s}tructural \textbf{s}elf-\textbf{d}issimilarity (MS$^3$D) regularization, which constrains the gradient field to have a consistent pattern across different scales, thereby fostering a more redundant and robust system. We show that our method can effectively enhance the performance and stability of GANs under limited data scenarios, and even allow them to generate high-quality images with very few data.