Large-Scale Feature Learning With Spike-and-Slab Sparse Coding

TL;DR

This paper introduces a scalable spike-and-slab sparse coding method optimized for GPU architectures, significantly improving feature learning for large-scale object recognition tasks with many classes.

Contribution

The authors develop a novel GPU-compatible inference procedure for S3C, enabling it to handle larger datasets and more latent factors, outperforming previous models on CIFAR datasets.

Findings

Improved supervised learning over sparse coding and ssRBM on CIFAR-10.

Scales effectively to large class numbers on CIFAR-100.

Won the NIPS 2011 Transfer Learning Challenge.

Abstract

We consider the problem of object recognition with a large number of classes. In order to overcome the low amount of labeled examples available in this setting, we introduce a new feature learning and extraction procedure based on a factor model we call spike-and-slab sparse coding (S3C). Prior work on S3C has not prioritized the ability to exploit parallel architectures and scale S3C to the enormous problem sizes needed for object recognition. We present a novel inference procedure for appropriate for use with GPUs which allows us to dramatically increase both the training set size and the amount of latent factors that S3C may be trained with. We demonstrate that this approach improves upon the supervised learning capabilities of both sparse coding and the spike-and-slab Restricted Boltzmann Machine (ssRBM) on the CIFAR-10 dataset. We use the CIFAR-100 dataset to demonstrate that our method scales to large numbers of classes better than previous methods. Finally, we use our method to win the NIPS 2011 Workshop on Challenges In Learning Hierarchical Models? Transfer Learning Challenge.