Rapid Learning with Stochastic Focus of Attention

TL;DR

This paper introduces a method to halt feature evaluation early for easy-to-classify examples, significantly reducing computation in online margin-based learning algorithms like Pegasos without sacrificing accuracy.

Contribution

It proposes an attention mechanism that stops feature evaluation early, reducing computational cost from linear to square root order in the number of features.

Findings

Reduces feature evaluation from O(n) to O(√n) on average

Maintains prediction accuracy despite early stopping

Demonstrates effectiveness on MNIST dataset

Abstract

We present a method to stop the evaluation of a decision making process when the result of the full evaluation is obvious. This trait is highly desirable for online margin-based machine learning algorithms where a classifier traditionally evaluates all the features for every example. We observe that some examples are easier to classify than others, a phenomenon which is characterized by the event when most of the features agree on the class of an example. By stopping the feature evaluation when encountering an easy to classify example, the learning algorithm can achieve substantial gains in computation. Our method provides a natural attention mechanism for learning algorithms. By modifying Pegasos, a margin-based online learning algorithm, to include our attentive method we lower the number of attributes computed from $n$ to an average of $O(\sqrt{n})$ features without loss in prediction accuracy. We demonstrate the effectiveness of Attentive Pegasos on MNIST data.