New Insights on Learning Rules for Hopfield Networks: Memory and Objective Function Minimisation

TL;DR

This paper revisits learning rules for Hopfield networks, framing them as descent algorithms for various cost functions, introduces new cost functions, and experimentally compares different learning methods and self-coupling effects.

Contribution

It presents a new perspective on Hopfield network learning rules as descent algorithms, proposes novel cost functions, and investigates self-coupling's impact on memory capacity.

Findings

Newton's method improves learning efficiency

New cost functions enhance memory retrieval

Self-coupling effects on capacity are empirically analyzed

Abstract

Hopfield neural networks are a possible basis for modelling associative memory in living organisms. After summarising previous studies in the field, we take a new look at learning rules, exhibiting them as descent-type algorithms for various cost functions. We also propose several new cost functions suitable for learning. We discuss the role of biases (the external inputs) in the learning process in Hopfield networks. Furthermore, we apply Newtons method for learning memories, and experimentally compare the performances of various learning rules. Finally, to add to the debate whether allowing connections of a neuron to itself enhances memory capacity, we numerically investigate the effects of self coupling. Keywords: Hopfield Networks, associative memory, content addressable memory, learning rules, gradient descent, attractor networks