Semi-supervised Learning of Perceptual Video Quality by Generating Consistent Pairwise Pseudo-Ranks

TL;DR

This paper introduces a semi-supervised learning framework for no-reference video quality assessment that effectively utilizes unlabelled videos through pseudo-ranking and consistency regularization, improving performance with limited labeled data.

Contribution

The work proposes a novel SSL method using pairwise pseudo-ranks and strong-weak augmentations, along with new spatial-temporal features, to enhance VQA accuracy with minimal labeled data.

Findings

Improved correlation with human perception on VQA datasets.

Effective use of unlabelled videos through pseudo-ranking.

Enhanced features based on spatial and temporal entropic differences.

Abstract

Designing learning-based no-reference (NR) video quality assessment (VQA) algorithms for camera-captured videos is cumbersome due to the requirement of a large number of human annotations of quality. In this work, we propose a semi-supervised learning (SSL) framework exploiting many unlabelled and very limited amounts of labelled authentically distorted videos. Our main contributions are two-fold. Leveraging the benefits of consistency regularization and pseudo-labelling, our SSL model generates pairwise pseudo-ranks for the unlabelled videos using a student-teacher model on strongweak augmented videos. We design the strong-weak augmentations to be quality invariant to use the unlabelled videos effectively in SSL. The generated pseudo-ranks are used along with the limited labels to train our SSL model. Our primary focus in SSL for NR VQA is to learn the mapping from video feature representations to the quality scores. We compare various feature extraction methods and show that our SSL framework can lead to improved performance on these features. In addition to the existing features, we present a spatial and temporal feature extraction method based on predicting spatial and temporal entropic differences. We show that these features help achieve a robust performance when trained with limited data providing a better baseline to apply SSL. Extensive experiments on three popular VQA datasets demonstrate that a combination of our novel SSL approach and features achieves an impressive performance in terms of correlation with human perception, even though the number of human-annotated videos may be limited.