Kernel Dependence Network

TL;DR

The paper introduces Kernel Dependence Network (KNet), a deep learning model trained via spectral methods that maximizes dependence between features and labels using HSIC, with automatic network architecture determination.

Contribution

It presents a novel spectral training approach for deep networks using dependence maximization and eigenvalue-based architecture selection, with theoretical guarantees.

Findings

Spectral training of deep networks via dependence maximization.

Automatic determination of network width and depth.

Theoretical guarantees of global optimality.

Abstract

We propose a greedy strategy to spectrally train a deep network for multi-class classification. Each layer is defined as a composition of linear weights with the feature map of a Gaussian kernel acting as the activation function. At each layer, the linear weights are learned by maximizing the dependence between the layer output and the labels using the Hilbert Schmidt Independence Criterion (HSIC). By constraining the solution space on the Stiefel Manifold, we demonstrate how our network construct (Kernel Dependence Network or KNet) can be solved spectrally while leveraging the eigenvalues to automatically find the width and the depth of the network. We theoretically guarantee the existence of a solution for the global optimum while providing insight into our network's ability to generalize.