Subjective evaluation of traditional and learning-based image coding methods

TL;DR

This study compares traditional and learning-based image coding methods through subjective experiments, revealing that CNN and GAN methods outperform traditional ones at low bit-rates but show uncertain advantages at high bit-rates, with some objective metrics failing to accurately assess learned image quality.

Contribution

It provides a comprehensive subjective evaluation of traditional versus learning-based image coding methods, highlighting their performance differences across bit-rates and limitations of current quality metrics.

Findings

CNN and GAN methods outperform traditional coding at low bit-rates.

Performance of CNN methods at high bit-rates is inconclusive.

Objective quality metrics often do not accurately reflect learned image quality.

Abstract

We conduct a subjective experiment to compare the performance of traditional image coding methods and learning-based image coding methods. HEVC and VVC, the state-of-the-art traditional coding methods, are used as the representative traditional methods. The learning-based methods used contain not only CNN-based methods, but also a GAN-based method, all of which are advanced or typical. Single Stimuli (SS), which is also called Absolute Category Rating (ACR), is adopted as the methodology of the experiment to obtain perceptual quality of images. Additionally, we utilize some typical and frequently used objective quality metrics to evaluate the coding methods in the experiment as comparison. The experiment shows that CNN-based and GAN-based methods can perform better than traditional methods in low bit-rates. In high bit-rates, however, it is hard to verify whether CNN-based methods are superior to traditional methods. Because the GAN method does not provide models with high target bit-rates, we cannot exactly tell the performance of the GAN method in high bit-rates. Furthermore, some popular objective quality metrics have not shown the ability well to measure quality of images generated by learning-based coding methods, especially the GAN-based one.