Neural Networks Built from Unreliable Components

TL;DR

This paper investigates the robustness of neural networks and associative memories when internal computational components are noisy, demonstrating that reliable recall is still achievable through optimized inference algorithms and revealing a threshold phenomenon.

Contribution

It introduces an analysis of associative memory performance with noisy internal components and proposes methods to optimize inference parameters based on noise statistics.

Findings

Final error probability can be made very small despite internal noise.

A threshold phenomenon exists in noisy associative memories.

Optimization of inference parameters improves recall reliability.

Abstract

Recent advances in associative memory design through strutured pattern sets and graph-based inference algorithms have allowed the reliable learning and retrieval of an exponential number of patterns. Both these and classical associative memories, however, have assumed internally noiseless computational nodes. This paper considers the setting when internal computations are also noisy. Even if all components are noisy, the final error probability in recall can often be made exceedingly small, as we characterize. There is a threshold phenomenon. We also show how to optimize inference algorithm parameters when knowing statistical properties of internal noise.