Evaluation Metrics and Methods for Generative Models in the Wireless PHY Layer

TL;DR

This paper introduces a set of evaluation metrics and methods tailored for generative models in the wireless physical layer, emphasizing interpretability and relevance to wireless applications, validated on real-world data.

Contribution

It proposes novel, application-motivated evaluation metrics and methods for wireless generative models, addressing limitations of existing metrics like MMD.

Findings

Proposed metrics provide consistent, explainable evaluation results.

Spectral efficiency and codebook fingerprinting effectively validate generated channels.

Traditional metrics like MMD are insufficient alone for evaluation.

Abstract

Generative models are typically evaluated by direct inspection of their generated samples, e.g., by visual inspection in the case of images. Further evaluation metrics like the Fr\'echet inception distance or maximum mean discrepancy are intricate to interpret and lack physical motivation. These observations make evaluating generative models in the wireless PHY layer non-trivial. This work establishes a framework consisting of evaluation metrics and methods for generative models applied to the wireless PHY layer. The proposed metrics and methods are motivated by wireless applications, facilitating interpretation and understandability for the wireless community. In particular, we propose a spectral efficiency analysis for validating the generated channel norms and a codebook fingerprinting method to validate the generated channel directions. Moreover, we propose an application cross-check to evaluate the generative model's samples for training machine learning-based models in relevant downstream tasks. Our analysis is based on real-world measurement data and includes the Gaussian mixture model, variational autoencoder, diffusion model, and generative adversarial network as generative models. Our results under a fair comparison in terms of model architecture indicate that solely relying on metrics like the maximum mean discrepancy produces insufficient evaluation outcomes. In contrast, the proposed metrics and methods exhibit consistent and explainable behavior.