How Shift Equivariance Impacts Metric Learning for Instance Segmentation

TL;DR

This paper provides a formal analysis of how shift equivariance in CNNs affects metric learning for instance segmentation, revealing fundamental limits and practical guidelines for tile-based approaches.

Contribution

It offers the first comprehensive formal analysis of shift equivariance in encoder-decoder CNNs for metric learning, establishing capacity limits and conditions to avoid discontinuities.

Findings

Standard encoder-decoder CNNs can distinguish at most f^{dl} same-looking objects.

To prevent discontinuities, training must use output windows larger than f^l.

Empirical results on synthetic data support the theoretical analysis.

Abstract

Metric learning has received conflicting assessments concerning its suitability for solving instance segmentation tasks. It has been dismissed as theoretically flawed due to the shift equivariance of the employed CNNs and their respective inability to distinguish same-looking objects. Yet it has been shown to yield state of the art results for a variety of tasks, and practical issues have mainly been reported in the context of tile-and-stitch approaches, where discontinuities at tile boundaries have been observed. To date, neither of the reported issues have undergone thorough formal analysis. In our work, we contribute a comprehensive formal analysis of the shift equivariance properties of encoder-decoder-style CNNs, which yields a clear picture of what can and cannot be achieved with metric learning in the face of same-looking objects. In particular, we prove that a standard encoder-decoder network that takes $d$-dimensional images as input, with $l$ pooling layers and pooling factor $f$, has the capacity to distinguish at most $f^{dl}$ same-looking objects, and we show that this upper limit can be reached. Furthermore, we show that to avoid discontinuities in a tile-and-stitch approach, assuming standard batch size 1, it is necessary to employ valid convolutions in combination with a training output window size strictly greater than $f^l$, while at test-time it is necessary to crop tiles to size $n\cdot f^l$ before stitching, with $n\geq 1$. We complement these theoretical findings by discussing a number of insightful special cases for which we show empirical results on synthetic data.