The Attentive Perceptron

TL;DR

The paper introduces the Attentive Perceptron, a modified perceptron algorithm that uses a focus of attention mechanism to evaluate fewer features on easy examples, significantly speeding up training and prediction with minimal accuracy loss.

Contribution

It presents a novel attention mechanism for perceptrons that reduces feature evaluation, improving efficiency while maintaining accuracy.

Findings

Achieves significant speedups in training and prediction.

Maintains high accuracy with fewer feature evaluations.

Effectively filters easy examples to focus computation on hard ones.

Abstract

We propose a focus of attention mechanism to speed up the Perceptron algorithm. Focus of attention speeds up the Perceptron algorithm by lowering the number of features evaluated throughout training and prediction. Whereas the traditional Perceptron evaluates all the features of each example, the Attentive Perceptron evaluates less features for easy to classify examples, thereby achieving significant speedups and small losses in prediction accuracy. Focus of attention allows the Attentive Perceptron to stop the evaluation of features at any interim point and filter the example. This creates an attentive filter which concentrates computation at examples that are hard to classify, and quickly filters examples that are easy to classify.