Latent Guidance in Diffusion Models for Perceptual Evaluations

TL;DR

This paper introduces Perceptual Manifold Guidance (PMG), a novel method guiding latent diffusion models with perceptual features to improve No-Reference Image Quality Assessment, achieving state-of-the-art results.

Contribution

It is the first to guide diffusion models with perceptual features for NR-IQA, enhancing perceptual consistency and performance.

Findings

High correlation of hyperfeatures with human perception.

State-of-the-art performance on IQA datasets.

Easy integration with existing pretrained models.

Abstract

Despite recent advancements in latent diffusion models that generate high-dimensional image data and perform various downstream tasks, there has been little exploration into perceptual consistency within these models on the task of No-Reference Image Quality Assessment (NR-IQA). In this paper, we hypothesize that latent diffusion models implicitly exhibit perceptually consistent local regions within the data manifold. We leverage this insight to guide on-manifold sampling using perceptual features and input measurements. Specifically, we propose Perceptual Manifold Guidance (PMG), an algorithm that utilizes pretrained latent diffusion models and perceptual quality features to obtain perceptually consistent multi-scale and multi-timestep feature maps from the denoising U-Net. We empirically demonstrate that these hyperfeatures exhibit high correlation with human perception in IQA tasks. Our method can be applied to any existing pretrained latent diffusion model and is straightforward to integrate. To the best of our knowledge, this paper is the first work on guiding diffusion model with perceptual features for NR-IQA. Extensive experiments on IQA datasets show that our method, LGDM, achieves state-of-the-art performance, underscoring the superior generalization capabilities of diffusion models for NR-IQA tasks.