Evaluating Post-Training Compression in GANs using Locality-Sensitive Hashing

TL;DR

This paper investigates post-training compression of GANs using existing methods like clipping and quantization, introduces LSH-based metrics for evaluation, and analyzes the trade-offs between compression levels, sample quality, and diversity.

Contribution

It demonstrates that existing compression techniques can be applied post-training to GANs without retraining and introduces LSH-based metrics for more robust and efficient evaluation.

Findings

High compression levels distort generated samples.

LSH-based metrics improve outlier robustness and evaluation efficiency.

Compression induces a trade-off between sample quality and diversity.

Abstract

The analysis of the compression effects in generative adversarial networks (GANs) after training, i.e. without any fine-tuning, remains an unstudied, albeit important, topic with the increasing trend of their computation and memory requirements. While existing works discuss the difficulty of compressing GANs during training, requiring novel methods designed with the instability of GANs training in mind, we show that existing compression methods (namely clipping and quantization) may be directly applied to compress GANs post-training, without any additional changes. High compression levels may distort the generated set, likely leading to an increase of outliers that may negatively affect the overall assessment of existing k-nearest neighbor (KNN) based metrics. We propose two new precision and recall metrics based on locality-sensitive hashing (LSH), which, on top of increasing the outlier robustness, decrease the complexity of assessing an evaluation sample against $n$ reference samples from $O(n)$ to $O(\log(n))$, if using LSH and KNN, and to $O(1)$, if only applying LSH. We show that low-bit compression of several pre-trained GANs on multiple datasets induces a trade-off between precision and recall, retaining sample quality while sacrificing sample diversity.