Revisiting Video Quality Assessment from the Perspective of Generalization

TL;DR

This paper reexamines Video Quality Assessment (VQA) focusing on its ability to generalize across datasets, revealing that smoothing the weight loss landscape via adversarial perturbations enhances performance and leads to state-of-the-art results.

Contribution

It introduces a novel perspective on VQA by analyzing the weight loss landscape and applying adversarial weight perturbations to improve generalization capabilities.

Findings

Adversarial weight perturbations effectively smooth the loss landscape.

Improved cross-dataset generalization by up to 1.8%.

Enhanced fine-tuning performance by up to 3%.

Abstract

The increasing popularity of short video platforms such as YouTube Shorts, TikTok, and Kwai has led to a surge in User-Generated Content (UGC), which presents significant challenges for the generalization performance of Video Quality Assessment (VQA) tasks. These challenges not only affect performance on test sets but also impact the ability to generalize across different datasets. While prior research has primarily focused on enhancing feature extractors, sampling methods, and network branches, it has largely overlooked the generalization capabilities of VQA tasks. In this work, we reevaluate the VQA task from a generalization standpoint. We begin by analyzing the weight loss landscape of VQA models, identifying a strong correlation between this landscape and the generalization gaps. We then investigate various techniques to regularize the weight loss landscape. Our results reveal that adversarial weight perturbations can effectively smooth this landscape, significantly improving the generalization performance, with cross-dataset generalization and fine-tuning performance enhanced by up to 1.8% and 3%, respectively. Through extensive experiments across various VQA methods and datasets, we validate the effectiveness of our approach. Furthermore, by leveraging our insights, we achieve state-of-the-art performance in Image Quality Assessment (IQA) tasks. Our code is available at https://github.com/XinliYue/VQA-Generalization.