Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting

TL;DR

This paper investigates how Restricted Boltzmann Machines (RBMs) learn structured data in a teacher-student framework, revealing how data complexity, correlations, and regularization influence learning performance and generalization.

Contribution

It extends the teacher-student RBM framework to arbitrary hidden units, analyzes the impact of data structure and regularization, and links findings to the lottery ticket hypothesis.

Findings

Performance is independent of teacher patterns and student hidden units without correlations.

Critical data amount decreases with more teacher patterns and correlations.

Low inference temperature impedes learning teacher patterns.

Abstract

Restricted Boltzmann machines (RBM) are generative models capable to learn data with a rich underlying structure. We study the teacher-student setting where a student RBM learns structured data generated by a teacher RBM. The amount of structure in the data is controlled by adjusting the number of hidden units of the teacher and the correlations in the rows of the weights, a.k.a. patterns. In the absence of correlations, we validate the conjecture that the performance is independent of the number of teacher patters and hidden units of the student RBMs, and we argue that the teacher-student setting can be used as a toy model for studying the lottery ticket hypothesis. Beyond this regime, we find that the critical amount of data required to learn the teacher patterns decreases with both their number and correlations. In both regimes, we find that, even with a relatively large dataset, it becomes impossible to learn the teacher patterns if the inference temperature used for regularization is kept too low. In our framework, the student can learn teacher patterns one-to-one or many-to-one, generalizing previous findings about the teacher-student setting with two hidden units to any arbitrary finite number of hidden units.