Shift-tolerant Perceptual Similarity Metric

TL;DR

This paper introduces a shift-tolerant perceptual similarity metric that remains robust against small misalignments between images, improving upon existing metrics like LPIPS by considering architectural design choices.

Contribution

The paper develops a novel neural network-based perceptual similarity metric that is robust to small shifts, addressing a key limitation of current metrics like LPIPS.

Findings

The new metric is tolerant to imperceptible shifts.

It remains consistent with human similarity judgments.

Architectural elements like anti-aliasing and skip connections are crucial for robustness.

Abstract

Existing perceptual similarity metrics assume an image and its reference are well aligned. As a result, these metrics are often sensitive to a small alignment error that is imperceptible to the human eyes. This paper studies the effect of small misalignment, specifically a small shift between the input and reference image, on existing metrics, and accordingly develops a shift-tolerant similarity metric. This paper builds upon LPIPS, a widely used learned perceptual similarity metric, and explores architectural design considerations to make it robust against imperceptible misalignment. Specifically, we study a wide spectrum of neural network elements, such as anti-aliasing filtering, pooling, striding, padding, and skip connection, and discuss their roles in making a robust metric. Based on our studies, we develop a new deep neural network-based perceptual similarity metric. Our experiments show that our metric is tolerant to imperceptible shifts while being consistent with the human similarity judgment.