The Rules-and-Facts Model for Simultaneous Generalization and Memorization in Neural Networks

TL;DR

This paper introduces the Rules-and-Facts (RAF) model, a theoretical framework that explains how neural networks can simultaneously learn underlying rules and memorize specific facts, highlighting the roles of overparameterization and regularization.

Contribution

The paper presents the RAF model, bridging rule generalization and memorization analysis, providing a minimal setting for understanding dual learning capabilities in neural networks.

Findings

Overparameterization supports simultaneous rule learning and memorization.

Regularization and kernel choice influence capacity allocation.

The model offers a theoretical basis for neural networks' dual learning ability.

Abstract

A key capability of modern neural networks is their capacity to simultaneously learn underlying rules and memorize specific facts or exceptions. Yet, theoretical understanding of this dual capability remains limited. We introduce the Rules-and-Facts (RAF) model, a minimal solvable setting that enables precise characterization of this phenomenon by bridging two classical lines of work in the statistical physics of learning: the teacher-student framework for generalization and Gardner-style capacity analysis for memorization. In the RAF model, a fraction $1 - \varepsilon$ of training labels is generated by a structured teacher rule, while a fraction $\varepsilon$ consists of unstructured facts with random labels. We characterize when the learner can simultaneously recover the underlying rule - allowing generalization to new data - and memorize the unstructured examples. Our results quantify how overparameterization enables the simultaneous realization of these two objectives: sufficient excess capacity supports memorization, while regularization and the choice of kernel or nonlinearity control the allocation of capacity between rule learning and memorization. The RAF model provides a theoretical foundation for understanding how modern neural networks can infer structure while storing rare or non-compressible information.