The Role of Weight Shrinking in Large Margin Perceptron Learning

TL;DR

This paper proposes a modified perceptron algorithm that incorporates weight shrinking to improve margin maximization, providing theoretical guarantees and demonstrating competitive performance in soft margin tasks.

Contribution

Introduces weight shrinking into the perceptron with margin, enabling finite-step approximation of the maximum margin hyperplane with theoretical guarantees.

Findings

Algorithm attains desirable margin approximation in finite steps

Experimental results show competitive performance in soft margin tasks

Provides new theoretical insights into margin-error-driven learning

Abstract

We introduce into the classical perceptron algorithm with margin a mechanism that shrinks the current weight vector as a first step of the update. If the shrinking factor is constant the resulting algorithm may be regarded as a margin-error-driven version of NORMA with constant learning rate. In this case we show that the allowed strength of shrinking depends on the value of the maximum margin. We also consider variable shrinking factors for which there is no such dependence. In both cases we obtain new generalizations of the perceptron with margin able to provably attain in a finite number of steps any desirable approximation of the maximal margin hyperplane. The new approximate maximum margin classifiers appear experimentally to be very competitive in 2-norm soft margin tasks involving linear kernels.