Deep Networks with Adaptive Nystr\"om Approximation

TL;DR

This paper introduces a flexible neural network architecture that replaces dense layers with a Nyström kernel approximation, maintaining performance while reducing parameters, especially effective with small datasets.

Contribution

It presents a novel neural network design integrating Nyström kernel approximation, compatible with multiple kernels and suitable for small training datasets.

Findings

Achieves comparable performance to standard architectures on SVHN and CIFAR100.

Has fewer learnable parameters, beneficial for small datasets.

Compatible with any kernel function and multiple kernels.

Abstract

Recent work has focused on combining kernel methods and deep learning to exploit the best of the two approaches. Here, we introduce a new architecture of neural networks in which we replace the top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nystr{\"o}m approximation. Our approach is easy and highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that our architecture has the same performance than standard architecture on datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which makes it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class.