Making Video Quality Assessment Models Sensitive to Frame Rate Distortions

TL;DR

This paper enhances video quality assessment models by integrating temporal features to better detect distortions caused by frame rate variations, significantly improving accuracy across diverse datasets.

Contribution

It introduces a simple fusion framework that combines GREED's temporal features with existing VQA models, improving sensitivity to frame rate distortions.

Findings

Fusion significantly boosts model performance on various datasets.

Temporal features improve robustness against frame rate variations.

GREED-based features enhance detection of frame rate related distortions.

Abstract

We consider the problem of capturing distortions arising from changes in frame rate as part of Video Quality Assessment (VQA). Variable frame rate (VFR) videos have become much more common, and streamed videos commonly range from 30 frames per second (fps) up to 120 fps. VFR-VQA offers unique challenges in terms of distortion types as well as in making non-uniform comparisons of reference and distorted videos having different frame rates. The majority of current VQA models require compared videos to be of the same frame rate, but are unable to adequately account for frame rate artifacts. The recently proposed Generalized Entropic Difference (GREED) VQA model succeeds at this task, using natural video statistics models of entropic differences of temporal band-pass coefficients, delivering superior performance on predicting video quality changes arising from frame rate distortions. Here we propose a simple fusion framework, whereby temporal features from GREED are combined with existing VQA models, towards improving model sensitivity towards frame rate distortions. We find through extensive experiments that this feature fusion significantly boosts model performance on both HFR/VFR datasets as well as fixed frame rate (FFR) VQA databases. Our results suggest that employing efficient temporal representations can result much more robust and accurate VQA models when frame rate variations can occur.