Intriguing Properties of Dynamic Sampling Networks

TL;DR

This paper introduces a unifying operator called 'warping' that generalizes dynamic sampling methods in deep learning, providing theoretical insights, stability conditions, and visualization tools for training such networks.

Contribution

It develops a formal framework for dynamic sampling, analyzes its properties, and introduces new tools for understanding training dynamics in these models.

Findings

Warper generalizes existing dynamic sampling methods.

Identifies asymmetry between forward and backward passes.

Provides conditions for stable training of dynamic sampling networks.

Abstract

Dynamic sampling mechanisms in deep learning architectures have demonstrated utility across many computer vision models, though the theoretical analysis of these structures has not yet been unified. In this paper we connect the various dynamic sampling methods by developing and analyzing a novel operator which generalizes existing methods, which we term "warping". Warping provides a minimal implementation of dynamic sampling which is amenable to analysis, and can be used to reconstruct existing architectures including deformable convolutions, active convolutional units, and spatial transformer networks. Using our formalism, we provide statistical analysis of the operator by modeling the inputs as both IID variables and homogeneous random fields. Extending this analysis, we discover a unique asymmetry between the forward and backward pass of the model training. We demonstrate that these mechanisms represent an entirely different class of orthogonal operators to the traditional translationally invariant operators defined by convolutions. With a combination of theoretical analysis and empirical investigation, we find the conditions necessary to ensure stable training of dynamic sampling networks. In addition, statistical analysis of discretization effects are studied. Finally, we introduce a novel loss landscape visualization which utilizes gradient update information directly, to better understand learning behavior.