Domain-agnostic and Multi-level Evaluation of Generative Models

TL;DR

MPEGO is a flexible, multi-level evaluation framework for generative models that is domain-agnostic and customizable, providing hierarchical performance insights from low-level features to global characteristics.

Contribution

The paper introduces MPEGO, a novel, hierarchical evaluation framework for generative models that is adaptable across various domains and feature complexities.

Findings

MPEGO effectively evaluates generative models across multiple datasets.

The framework offers detailed, multi-level insights into generation quality.

Results show MPEGO's flexibility and practical utility in diverse scenarios.

Abstract

While the capabilities of generative models heavily improved in different domains (images, text, graphs, molecules, etc.), their evaluation metrics largely remain based on simplified quantities or manual inspection with limited practicality. To this end, we propose a framework for Multi-level Performance Evaluation of Generative mOdels (MPEGO), which could be employed across different domains. MPEGO aims to quantify generation performance hierarchically, starting from a sub-feature-based low-level evaluation to a global features-based high-level evaluation. MPEGO offers great customizability as the employed features are entirely user-driven and can thus be highly domain/problem-specific while being arbitrarily complex (e.g., outcomes of experimental procedures). We validate MPEGO using multiple generative models across several datasets from the material discovery domain. An ablation study is conducted to study the plausibility of intermediate steps in MPEGO. Results demonstrate that MPEGO provides a flexible, user-driven, and multi-level evaluation framework, with practical insights on the generation quality. The framework, source code, and experiments will be available at https://github.com/GT4SD/mpego.