Lecture notes: Efficient approximation of kernel functions

TL;DR

These lecture notes provide a comprehensive overview of kernel functions, focusing on the Random Fourier Features approximation, including mathematical background, theoretical analysis, and error estimation techniques.

Contribution

The notes compile essential mathematical background and detailed proofs for understanding and analyzing the Random Fourier Features approximation in kernel methods.

Findings

Provides detailed proofs of error bounds for Random Fourier Features

Summarizes key properties of kernels and Hilbert spaces

Includes concentration results for approximation error

Abstract

These lecture notes endeavour to collect in one place the mathematical background required to understand the properties of kernels in general and the Random Fourier Features approximation of Rahimi and Recht (NIPS 2007) in particular. We briefly motivate the use of kernels in Machine Learning with the example of the support vector machine. We discuss positive definite and conditionally negative definite kernels in some detail. After a brief discussion of Hilbert spaces, including the Reproducing Kernel Hilbert Space construction, we present Mercer's theorem. We discuss the Random Fourier Features technique and then present, with proofs, scalar and matrix concentration results that help us estimate the error incurred by the technique. These notes are the transcription of 10 lectures given at IIT Delhi between January and April 2020.