Unexpected Benefits of Self-Modeling in Neural Systems

TL;DR

This paper demonstrates that self-modeling in neural networks acts as a form of self-regularization, reducing complexity and improving parameter efficiency, which may explain some benefits observed in machine learning and biological systems.

Contribution

It provides empirical evidence that self-modeling leads to simpler, more regularized neural networks across various architectures and tasks, revealing a new regularization mechanism.

Findings

Self-modeling reduces network weight distribution width.

Self-modeling decreases the real log canonical threshold (RLCT).

Greater emphasis on self-modeling enhances complexity reduction.

Abstract

Self-models have been a topic of great interest for decades in studies of human cognition and more recently in machine learning. Yet what benefits do self-models confer? Here we show that when artificial networks learn to predict their internal states as an auxiliary task, they change in a fundamental way. To better perform the self-model task, the network learns to make itself simpler, more regularized, more parameter-efficient, and therefore more amenable to being predictively modeled. To test the hypothesis of self-regularizing through self-modeling, we used a range of network architectures performing three classification tasks across two modalities. In all cases, adding self-modeling caused a significant reduction in network complexity. The reduction was observed in two ways. First, the distribution of weights was narrower when self-modeling was present. Second, a measure of network complexity, the real log canonical threshold (RLCT), was smaller when self-modeling was present. Not only were measures of complexity reduced, but the reduction became more pronounced as greater training weight was placed on the auxiliary task of self-modeling. These results strongly support the hypothesis that self-modeling is more than simply a network learning to predict itself. The learning has a restructuring effect, reducing complexity and increasing parameter efficiency. This self-regularization may help explain some of the benefits of self-models reported in recent machine learning literature, as well as the adaptive value of self-models to biological systems. In particular, these findings may shed light on the possible interaction between the ability to model oneself and the ability to be more easily modeled by others in a social or cooperative context.