Improving Equivariance in State-of-the-Art Supervised Depth and Normal Predictors

TL;DR

This paper identifies that current state-of-the-art depth and normal predictors lack cropping-and-resizing equivariance and proposes a regularization method to explicitly enforce this property, improving their accuracy and robustness.

Contribution

The authors introduce an equivariant regularization technique that enhances cropping-and-resizing equivariance in depth and normal predictors across CNN and Transformer models.

Findings

Improved equivariance in depth and normal predictions.

Enhanced accuracy on Taskonomy and NYU-v2 datasets.

Applicable to both supervised and semi-supervised learning.

Abstract

Dense depth and surface normal predictors should possess the equivariant property to cropping-and-resizing -- cropping the input image should result in cropping the same output image. However, we find that state-of-the-art depth and normal predictors, despite having strong performances, surprisingly do not respect equivariance. The problem exists even when crop-and-resize data augmentation is employed during training. To remedy this, we propose an equivariant regularization technique, consisting of an averaging procedure and a self-consistency loss, to explicitly promote cropping-and-resizing equivariance in depth and normal networks. Our approach can be applied to both CNN and Transformer architectures, does not incur extra cost during testing, and notably improves the supervised and semi-supervised learning performance of dense predictors on Taskonomy tasks. Finally, finetuning with our loss on unlabeled images improves not only equivariance but also accuracy of state-of-the-art depth and normal predictors when evaluated on NYU-v2. GitHub link: https://github.com/mikuhatsune/equivariance