Why Size Matters: Feature Coding as Nystrom Sampling

TL;DR

This paper presents a kernel-based perspective on feature coding using Nystrom sampling, providing theoretical bounds on approximation quality and insights into the effects of dictionary size in machine learning pipelines.

Contribution

It introduces a novel kernel method framework for feature coding with Nystrom sampling, offering bounds on approximation errors and explaining the impact of codebook size.

Findings

Bounds on kernel approximation improve with larger dictionaries

Increasing dictionary size enhances accuracy up to a saturation point

The model explains the positive effects of deeper, layered representations

Abstract

Recently, the computer vision and machine learning community has been in favor of feature extraction pipelines that rely on a coding step followed by a linear classifier, due to their overall simplicity, well understood properties of linear classifiers, and their computational efficiency. In this paper we propose a novel view of this pipeline based on kernel methods and Nystrom sampling. In particular, we focus on the coding of a data point with a local representation based on a dictionary with fewer elements than the number of data points, and view it as an approximation to the actual function that would compute pair-wise similarity to all data points (often too many to compute in practice), followed by a Nystrom sampling step to select a subset of all data points. Furthermore, since bounds are known on the approximation power of Nystrom sampling as a function of how many samples (i.e. dictionary size) we consider, we can derive bounds on the approximation of the exact (but expensive to compute) kernel matrix, and use it as a proxy to predict accuracy as a function of the dictionary size, which has been observed to increase but also to saturate as we increase its size. This model may help explaining the positive effect of the codebook size and justifying the need to stack more layers (often referred to as deep learning), as flat models empirically saturate as we add more complexity.