Barcode Method for Generative Model Evaluation driven by Topological Data Analysis

TL;DR

This paper introduces the barcode method, inspired by topological data analysis, for evaluating generative models' fidelity and diversity, outperforming existing metrics like FID and precision-recall, with minimal assumptions.

Contribution

The study presents a novel barcode algorithm for generative model evaluation that is assumption-free and hyperparameter-light, improving accuracy over traditional methods.

Findings

Barcode outperforms existing evaluation metrics.

Normality assumptions have notable drawbacks.

Method is validated on real-world datasets and theoretical models.

Abstract

Evaluating the performance of generative models in image synthesis is a challenging task. Although the Fr\'echet Inception Distance is a widely accepted evaluation metric, it integrates different aspects (e.g., fidelity and diversity) of synthesized images into a single score and assumes the normality of embedded vectors. Recent methods such as precision-and-recall and its variants such as density-and-coverage have been developed to separate fidelity and diversity based on k-nearest neighborhood methods. In this study, we propose an algorithm named barcode, which is inspired by the topological data analysis and is almost free of assumption and hyperparameter selections. In extensive experiments on real-world datasets as well as theoretical approach on high-dimensional normal samples, it was found that the 'usual' normality assumption of embedded vectors has several drawbacks. The experimental results demonstrate that barcode outperforms other methods in evaluating fidelity and diversity of GAN outputs. Official codes can be found in https://github.com/minjeekim00/Barcode.