The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric

TL;DR

This paper introduces LASI, a perceptual similarity metric based on linear embeddings constructed at inference-time, which rivals deep learning methods in image quality assessment with similar computational efficiency.

Contribution

The paper presents a novel inference-time method for constructing perceptual embeddings without training, enabling a competitive and computationally efficient perceptual metric called LASI.

Findings

LASI performs competitively with deep feature-based methods like LPIPS and PIM.

Increasing embedding dimensionality improves perceptual task performance.

LASI is fully differentiable and scalable, suitable for parallel computation.

Abstract

We show how perceptual embeddings of the visual system can be constructed at inference-time with no training data or deep neural network features. Our perceptual embeddings are solutions to a weighted least squares (WLS) problem, defined at the pixel-level, and solved at inference-time, that can capture global and local image characteristics. The distance in embedding space is used to define a perceptual similarity metric which we call LASI: Linear Autoregressive Similarity Index. Experiments on full-reference image quality assessment datasets show LASI performs competitively with learned deep feature based methods like LPIPS (Zhang et al., 2018) and PIM (Bhardwaj et al., 2020), at a similar computational cost to hand-crafted methods such as MS-SSIM (Wang et al., 2003). We found that increasing the dimensionality of the embedding space consistently reduces the WLS loss while increasing performance on perceptual tasks, at the cost of increasing the computational complexity. LASI is fully differentiable, scales cubically with the number of embedding dimensions, and can be parallelized at the pixel-level. A Maximum Differentiation (MAD) competition (Wang & Simoncelli, 2008) between LASI and LPIPS shows that both methods are capable of finding failure points for the other, suggesting these metrics can be combined.