# Do Kernel and Neural Embeddings Help in Training and Generalization?

## TL;DR

This paper investigates how kernel and neural embeddings influence training efficiency and generalization in neural networks, demonstrating that certain representations can improve both aspects.

## Contribution

It introduces a comprehensive experimental analysis of various kernel-based representations fed into neural networks, highlighting their impact on optimization and generalization.

## Key findings

- Kernel representations improve training convergence
- Certain embeddings enhance generalization performance
- Optimal kernels align data better with labels

## Abstract

Recent results on optimization and generalization properties of neural networks showed that in a simple two-layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines the convergence of the optimization during training. Such analyses also provide upper bounds on the generalization error. We experimentally investigate the implications of these results to deeper networks via embeddings. We regard the layers preceding the final hidden layer as producing different representations of the input data which are then fed to the two-layer model. We show that these representations improve both optimization and generalization. In particular, we investigate three kernel representations when fed to the final hidden layer: the Gaussian kernel and its approximation by random Fourier features, kernels designed to imitate representations produced by neural networks and finally an optimal kernel designed to align the data with target labels. The approximated representations induced by these kernels are fed to the neural network and the optimization and generalization properties of the final model are evaluated and compared.