Understanding image representations by measuring their equivariance and equivalence

TL;DR

This paper explores the mathematical properties of image representations, such as equivariance, invariance, and equivalence, providing methods to empirically analyze these properties in CNNs and other representations.

Contribution

It introduces new methods to empirically measure equivariance, invariance, and equivalence in image representations, enhancing understanding of their structure and behavior.

Findings

Identified layers in CNNs where geometric invariances are achieved

Proposed methods to empirically measure representation properties

Clarified the relationship between different image representations

Abstract

Despite the importance of image representations such as histograms of oriented gradients and deep Convolutional Neural Networks (CNN), our theoretical understanding of them remains limited. Aiming at filling this gap, we investigate three key mathematical properties of representations: equivariance, invariance, and equivalence. Equivariance studies how transformations of the input image are encoded by the representation, invariance being a special case where a transformation has no effect. Equivalence studies whether two representations, for example two different parametrisations of a CNN, capture the same visual information or not. A number of methods to establish these properties empirically are proposed, including introducing transformation and stitching layers in CNNs. These methods are then applied to popular representations to reveal insightful aspects of their structure, including clarifying at which layers in a CNN certain geometric invariances are achieved. While the focus of the paper is theoretical, direct applications to structured-output regression are demonstrated too.