USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s

TL;DR

The paper introduces USTC-TD, a new high-quality image and video dataset for evaluating coding schemes, and provides a comprehensive benchmark analysis to guide future research in image/video coding.

Contribution

It presents a diverse, high-resolution dataset for image/video coding evaluation and offers extensive benchmarking of existing coding schemes using multiple quality metrics.

Findings

USTC-TD effectively addresses shortcomings of previous datasets.

Benchmark results highlight strengths and weaknesses of current coding schemes.

The dataset and benchmarks facilitate future research in image/video coding.

Abstract

Image/video coding has been a remarkable research area for both academia and industry for many years. Testing datasets, especially high-quality image/video datasets are desirable for the justified evaluation of coding-related research, practical applications, and standardization activities. We put forward a test dataset namely USTC-TD, which has been successfully adopted in the practical end-to-end image/video coding challenge of the IEEE International Conference on Visual Communications and Image Processing (VCIP) in 2022 and 2023. USTC-TD contains 40 images at 4K spatial resolution and 10 video sequences at 1080p spatial resolution, featuring various content due to the diverse environmental factors (e.g. scene type, texture, motion, view) and the designed imaging factors (e.g. illumination, lens, shadow). We quantitatively evaluate USTC-TD on different image/video features (spatial, temporal, color, lightness), and compare it with the previous image/video test datasets, which verifies its excellent compensation for the shortcomings of existing datasets. We also evaluate both classic standardized and recently learned image/video coding schemes on USTC-TD using objective quality metrics (PSNR, MS-SSIM, VMAF) and subjective quality metric (MOS), providing an extensive benchmark for these evaluated schemes. Based on the characteristics and specific design of the proposed test dataset, we analyze the benchmark performance and shed light on the future research and development of image/video coding. All the data are released online: https://esakak.github.io/USTC-TD.