Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study

TL;DR

This paper introduces a novel black-box approach to diagnose and calibrate intra-mode collapse in GANs without access to training data or model parameters, revealing that mode collapse persists in state-of-the-art GANs.

Contribution

It proposes new statistical tools for diagnosing and rectifying intra-mode collapse in GANs under black-box constraints, a setting rarely explored before.

Findings

Intra-mode collapse is prevalent in current GANs.

The proposed methods effectively diagnose and calibrate mode collapse.

Black-box diagnosis and calibration are feasible and effective.

Abstract

Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse, and what to do if so. Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters. This paper explores to diagnose GAN intra-mode collapse and calibrate that, in a novel black-box setting: no access to training data, nor the trained model parameters, is assumed. The new setting is practically demanded, yet rarely explored and significantly more challenging. As a first stab, we devise a set of statistical tools based on sampling, that can visualize, quantify, and rectify intra-mode collapse. We demonstrate the effectiveness of our proposed diagnosis and calibration techniques, via extensive simulations and experiments, on unconditional GAN image generation (e.g., face and vehicle). Our study reveals that the intra-mode collapse is still a prevailing problem in state-of-the-art GANs and the mode collapse is diagnosable and calibratable in black-box settings. Our codes are available at: https://github.com/VITA-Group/BlackBoxGANCollapse.