Discriminating image representations with principal distortions

TL;DR

This paper introduces a framework using Fisher information to compare image representations based on their local geometries, revealing differences in models and potential links to human perception.

Contribution

The paper presents a novel method for comparing image representations through local geometry analysis using Fisher information, enabling differentiation of models based on local sensitivities.

Findings

Identified principal distortions that differentiate models

Compared early visual system models using local geometry

Revealed architecture and training differences in neural networks

Abstract

Image representations (artificial or biological) are often compared in terms of their global geometric structure; however, representations with similar global structure can have strikingly different local geometries. Here, we propose a framework for comparing a set of image representations in terms of their local geometries. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus distortions, and use this as a substrate for a metric on the local geometry in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by finding a pair of "principal distortions" that maximize the variance of the models under this metric. As an example, we use this framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we apply our method to a set of deep neural network models and reveal differences in the local geometry that arise due to architecture and training types. These examples demonstrate how our framework can be used to probe for informative differences in local sensitivities between complex models, and suggest how it could be used to compare model representations with human perception.