On the Computation of PSNR for a Set of Images or Video

TL;DR

This paper examines various methods for computing PSNR for image and video sets, highlighting how different approaches affect results and emphasizing the need for standardized reporting practices.

Contribution

It analyzes the impact of arithmetic versus geometric mean calculations of MSE on PSNR, depending on the MSE distribution, and discusses implications for consistent evaluation.

Findings

The difference between arithmetic and geometric mean PSNR depends on MSE distribution.

Restoration problems show larger PSNR differences due to exponential MSE distribution.

Compression problems exhibit smaller PSNR differences with narrower MSE distributions.

Abstract

When comparing learned image/video restoration and compression methods, it is common to report peak-signal to noise ratio (PSNR) results. However, there does not exist a generally agreed upon practice to compute PSNR for sets of images or video. Some authors report average of individual image/frame PSNR, which is equivalent to computing a single PSNR from the geometric mean of individual image/frame mean-square error (MSE). Others compute a single PSNR from the arithmetic mean of frame MSEs for each video. Furthermore, some compute the MSE/PSNR of Y-channel only, while others compute MSE/PSNR for RGB channels. This paper investigates different approaches to computing PSNR for sets of images, single video, and sets of video and the relation between them. We show the difference between computing the PSNR based on arithmetic vs. geometric mean of MSE depends on the distribution of MSE over the set of images or video, and that this distribution is task-dependent. In particular, these two methods yield larger differences in restoration problems, where the MSE is exponentially distributed and smaller differences in compression problems, where the MSE distribution is narrower. We hope this paper will motivate the community to clearly describe how they compute reported PSNR values to enable consistent comparison.