Learning by dilution in a Neural Network

TL;DR

This paper introduces a dilution method for perceptrons that enhances storage capacity and generalization, with a Hebb-like algorithm achieving optimal performance in teacher-student scenarios.

Contribution

It presents a novel dilution technique for perceptrons, calculates the critical storage capacity, and introduces an effective Hebb-like algorithm for improved generalization.

Findings

Perceptrons can store about N patterns by removing a fraction of weights.

The critical storage capacity depends on the remaining bond concentration.

The Hebb-like dilution algorithm reaches optimal generalization in teacher scenarios.

Abstract

A perceptron with N random weights can store of the order of N patterns by removing a fraction of the weights without changing their strengths. The critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated. A simple Hebb-like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability.