Learning Representation from Neural Fisher Kernel with Low-rank Approximation

TL;DR

This paper introduces the Neural Fisher Kernel (NFK), a kernel-based representation method for neural networks, and proposes a low-rank approximation technique that enables scalable, high-quality data representations for various learning tasks.

Contribution

The paper defines NFK for neural networks, demonstrates its low-rank structure, and develops an efficient approximation algorithm for large-scale applications.

Findings

Low-rank NFK approximations are effective for data representation.

High-quality representations are achieved across supervised and unsupervised models.

The method scales well to large datasets and complex networks.

Abstract

In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as a unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low rank approximation of NFK, which scales to large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.