Example Selection For Dictionary Learning

TL;DR

This paper investigates active example selection strategies in dictionary learning with sparse activations, demonstrating that certain heuristics can accelerate the learning process compared to uniform sampling.

Contribution

It introduces and evaluates heuristic and saliency-inspired selection algorithms, showing their potential to speed up dictionary learning.

Findings

Some selection algorithms improve learning speed

Active selection can outperform uniform sampling

Speculation on why certain heuristics are effective

Abstract

In unsupervised learning, an unbiased uniform sampling strategy is typically used, in order that the learned features faithfully encode the statistical structure of the training data. In this work, we explore whether active example selection strategies - algorithms that select which examples to use, based on the current estimate of the features - can accelerate learning. Specifically, we investigate effects of heuristic and saliency-inspired selection algorithms on the dictionary learning task with sparse activations. We show that some selection algorithms do improve the speed of learning, and we speculate on why they might work.