From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines

TL;DR

This paper establishes a theoretical connection between deep networks and indefinite kernel machines in reproducing kernel Krein space, showing how deep models can be transformed into flat kernel representations with implications for capacity control and sparsity.

Contribution

It introduces a method to convert deep networks into equivalent kernel machines in Krein space, providing new insights into their capacity and sparsity properties.

Findings

Deep networks can be represented as indefinite kernel machines in Krein space.

The transformation offers bounds on Rademacher complexity for capacity control.

Flat representations exhibit Lp-norm regularization with 0<p<1, indicating sparsity.

Abstract

In this paper we explore a connection between deep networks and learning in reproducing kernel Krein space. Our approach is based on the concept of push-forward - that is, taking a fixed non-linear transform on a linear projection and converting it to a linear projection on the output of a fixed non-linear transform, pushing the weights forward through the non-linearity. Applying this repeatedly from the input to the output of a deep network, the weights can be progressively "pushed" to the output layer, resulting in a flat network that has the form of a fixed non-linear map (whose form is determined by the structure of the deep network) followed by a linear projection determined by the weight matrices - that is, we take a deep network and convert it to an equivalent (indefinite) kernel machine. We then investigate the implications of this transformation for capacity control and uniform convergence, and provide a Rademacher complexity bound on the deep network in terms of Rademacher complexity in reproducing kernel Krein space. Finally, we analyse the sparsity properties of the flat representation, showing that the flat weights are (effectively) Lp-"norm" regularised with 0<p<1 (bridge regression).