DiffVQA: Video Quality Assessment Using Diffusion Feature Extractor

TL;DR

This paper introduces DiffVQA, a novel video quality assessment framework that leverages diffusion models for robust feature extraction, significantly improving alignment with human perception and generalization across diverse datasets.

Contribution

The paper proposes a diffusion model-based feature extractor for VQA, enhancing performance and generalization over existing CNN and ViT methods.

Findings

DiffVQA outperforms existing methods on multiple datasets.

DiffVQA demonstrates strong cross-dataset generalization.

The diffusion-based features improve correlation with human perceptual scores.

Abstract

Video Quality Assessment (VQA) aims to evaluate video quality based on perceptual distortions and human preferences. Despite the promising performance of existing methods using Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), they often struggle to align closely with human perceptions, particularly in diverse real-world scenarios. This challenge is exacerbated by the limited scale and diversity of available datasets. To address this limitation, we introduce a novel VQA framework, DiffVQA, which harnesses the robust generalization capabilities of diffusion models pre-trained on extensive datasets. Our framework adapts these models to reconstruct identical input frames through a control module. The adapted diffusion model is then used to extract semantic and distortion features from a resizing branch and a cropping branch, respectively. To enhance the model's ability to handle long-term temporal dynamics, a parallel Mamba module is introduced, which extracts temporal coherence augmented features that are merged with the diffusion features to predict the final score. Experiments across multiple datasets demonstrate DiffVQA's superior performance on intra-dataset evaluations and its exceptional generalization across datasets. These results confirm that leveraging a diffusion model as a feature extractor can offer enhanced VQA performance compared to CNN and ViT backbones.