A Perceptual Quality Metric for Video Frame Interpolation

TL;DR

This paper introduces a new perceptual quality metric specifically designed for evaluating video frame interpolation results, leveraging spatio-temporal features and trained on a dedicated dataset, outperforming existing metrics.

Contribution

The paper proposes a novel video-specific perceptual quality metric using spatio-temporal features and a new dataset, improving correlation with human perception.

Findings

Outperforms state-of-the-art quality metrics on video frame interpolation evaluation.

Utilizes spatio-temporal modules based on Swin Transformer for better feature extraction.

Provides publicly available code and dataset for further research.

Abstract

Research on video frame interpolation has made significant progress in recent years. However, existing methods mostly use off-the-shelf metrics to measure the quality of interpolation results with the exception of a few methods that employ user studies, which is time-consuming. As video frame interpolation results often exhibit unique artifacts, existing quality metrics sometimes are not consistent with human perception when measuring the interpolation results. Some recent deep learning-based perceptual quality metrics are shown more consistent with human judgments, but their performance on videos is compromised since they do not consider temporal information. In this paper, we present a dedicated perceptual quality metric for measuring video frame interpolation results. Our method learns perceptual features directly from videos instead of individual frames. It compares pyramid features extracted from video frames and employs Swin Transformer blocks-based spatio-temporal modules to extract spatio-temporal information. To train our metric, we collected a new video frame interpolation quality assessment dataset. Our experiments show that our dedicated quality metric outperforms state-of-the-art methods when measuring video frame interpolation results. Our code and model are made publicly available at \url{https://github.com/hqqxyy/VFIPS}.