# How ConvNets model Non-linear Transformations

## TL;DR

This paper provides a theoretical analysis of deep convolutional networks, showing how they achieve invariance to non-linear transformations through depth and hierarchy, and introduces Transformation Networks as a generalization.

## Contribution

It introduces Transformation Networks and offers a theoretical framework explaining how ConvNets gain invariance and hierarchical modeling capabilities.

## Key findings

- Deeper networks increase invariance to transformations.
- Hierarchical architectures generate invariance more efficiently.
- ConvNets can be invariant to non-linear transformations despite local pooling.

## Abstract

In this paper, we theoretically address three fundamental problems involving deep convolutional networks regarding invariance, depth and hierarchy. We introduce the paradigm of Transformation Networks (TN) which are a direct generalization of Convolutional Networks (ConvNets). Theoretically, we show that TNs (and thereby ConvNets) are can be invariant to non-linear transformations of the input despite pooling over mere local translations. Our analysis provides clear insights into the increase in invariance with depth in these networks. Deeper networks are able to model much richer classes of transformations. We also find that a hierarchical architecture allows the network to generate invariance much more efficiently than a non-hierarchical network. Our results provide useful insight into these three fundamental problems in deep learning using ConvNets.