Understanding SSIM

TL;DR

This paper critically examines the widely used SSIM index, revealing unexpected mathematical properties that can lead to misleading image quality assessments and potentially adverse effects when used in deep learning.

Contribution

It provides a detailed analysis of SSIM's mathematical properties, highlighting issues that challenge its reliability in image quality evaluation and neural network training.

Findings

SSIM can produce unexpected and sometimes undefined results.

Using SSIM as a loss function may misguide neural network training.

SSIM's properties can lead to incorrect conclusions in image quality assessment.

Abstract

The use of the structural similarity index (SSIM) is widespread. For almost two decades, it has played a major role in image quality assessment in many different research disciplines. Clearly, its merits are indisputable in the research community. However, little deep scrutiny of this index has been performed. Contrary to popular belief, there are some interesting properties of SSIM that merit such scrutiny. In this paper, we analyze the mathematical factors of SSIM and show that it can generate results, in both synthetic and realistic use cases, that are unexpected, sometimes undefined, and nonintuitive. As a consequence, assessing image quality based on SSIM can lead to incorrect conclusions and using SSIM as a loss function for deep learning can guide neural network training in the wrong direction.