DiffSim: Taming Diffusion Models for Evaluating Visual Similarity

TL;DR

DiffSim leverages pretrained diffusion models to evaluate visual similarity more effectively than traditional metrics, capturing semantic, style, and appearance details aligned with human perception.

Contribution

This paper introduces DiffSim, a novel method using diffusion models for measuring visual similarity, addressing limitations of existing metrics in capturing perceptual and semantic details.

Findings

DiffSim outperforms traditional metrics in aligning with human visual preferences.

Introduction of Sref and IP benchmarks for style and instance similarity evaluation.

DiffSim achieves state-of-the-art results across multiple visual similarity benchmarks.

Abstract

Diffusion models have fundamentally transformed the field of generative models, making the assessment of similarity between customized model outputs and reference inputs critically important. However, traditional perceptual similarity metrics operate primarily at the pixel and patch levels, comparing low-level colors and textures but failing to capture mid-level similarities and differences in image layout, object pose, and semantic content. Contrastive learning-based CLIP and self-supervised learning-based DINO are often used to measure semantic similarity, but they highly compress image features, inadequately assessing appearance details. This paper is the first to discover that pretrained diffusion models can be utilized for measuring visual similarity and introduces the DiffSim method, addressing the limitations of traditional metrics in capturing perceptual consistency in custom generation tasks. By aligning features in the attention layers of the denoising U-Net, DiffSim evaluates both appearance and style similarity, showing superior alignment with human visual preferences. Additionally, we introduce the Sref and IP benchmarks to evaluate visual similarity at the level of style and instance, respectively. Comprehensive evaluations across multiple benchmarks demonstrate that DiffSim achieves state-of-the-art performance, providing a robust tool for measuring visual coherence in generative models.