Gram-MMD: A Texture-Aware Metric for Image Realism Assessment

TL;DR

Gram-MMD (GMMD) is a new image realism metric that captures fine-grained textural and structural features using Gram matrices and MMD, outperforming existing metrics in various datasets and scenarios.

Contribution

We introduce GMMD, a texture-aware realism metric leveraging Gram matrices and MMD, with a hyperparameter selection protocol and extensive experiments demonstrating its effectiveness.

Findings

GMMD captures finer textural details than semantic-level metrics.

GMMD maintains correct realism rankings in cross-domain scenarios.

Experiments show GMMD's effectiveness across multiple backbone architectures.

Abstract

Evaluating the realism of generated images remains a fundamental challenge in generative modeling. Existing distributional metrics such as the Frechet Inception Distance (FID) and CLIP-MMD (CMMD) compare feature distributions at a semantic level but may overlook fine-grained textural information that can be relevant for distinguishing real from generated images. We introduce Gram-MMD (GMMD), a realism metric that leverages Gram matrices computed from intermediate activations of pretrained backbone networks to capture correlations between feature maps. By extracting the upper-triangular part of these symmetric Gram matrices and measuring the Maximum Mean Discrepancy (MMD) between an anchor distribution of real images and an evaluation distribution, GMMD produces a representation that encodes textural and structural characteristics at a finer granularity than global embeddings. To select the hyperparameters of the metric, we employ a meta-metric protocol based on controlled degradations applied to MS-COCO images, measuring monotonicity via Spearman's rank correlation and Kendall's tau. We conduct experiments on both the KADID-10k database and the RAISE realness assessment dataset using various backbone architectures, including DINOv2, DC-AE, Stable Diffusion's VAE encoder, VGG19, and the AlexNet backbone from LPIPS, among others. We also demonstrate on a cross-domain driving scenario (KITTI / Virtual KITTI / Stanford Cars) that CMMD can incorrectly rank real images as less realistic than synthetic ones due to its semantic bias, while GMMD preserves the correct ordering. Our results suggest that GMMD captures complementary information to existing semantic-level metrics.