Vertical Validation: Evaluating Implicit Generative Models for Graphs on Thin Support Regions

TL;DR

This paper introduces Vertical Validation, a novel evaluation method for implicit graph generative models that focuses on assessing their ability to generate molecules in scarcely supported regions, addressing a key gap in current validation techniques.

Contribution

The paper proposes Vertical Validation, a new evaluation framework that creates thin support regions for better assessment of generative models' ability to produce novel graphs.

Findings

Vertical Validation effectively identifies models that generate in sparse regions.

The method improves model selection for generating novel molecules.

It enhances detection of overfitting and memorization in generative models.

Abstract

There has been a growing excitement that implicit graph generative models could be used to design or discover new molecules for medicine or material design. Because these molecules have not been discovered, they naturally lie in unexplored or scarcely supported regions of the distribution of known molecules. However, prior evaluation methods for implicit graph generative models have focused on validating statistics computed from the thick support (e.g., mean and variance of a graph property). Therefore, there is a mismatch between the goal of generating novel graphs and the evaluation methods. To address this evaluation gap, we design a novel evaluation method called Vertical Validation (VV) that systematically creates thin support regions during the train-test splitting procedure and then reweights generated samples so that they can be compared to the held-out test data. This procedure can be seen as a generalization of the standard train-test procedure except that the splits are dependent on sample features. We demonstrate that our method can be used to perform model selection if performance on thin support regions is the desired goal. As a side benefit, we also show that our approach can better detect overfitting as exemplified by memorization.