Symbolic Synthesis of Neural Networks

TL;DR

This paper introduces G-SSNNs, neural modules that incorporate symbolic features to improve data efficiency and generalization, bridging neural and symbolic methods for better learning from limited data.

Contribution

The paper presents G-SSNNs, a novel neural architecture that integrates symbolic features, enabling predictable modulation of generalization and automatic extraction of meaningful symbolic semantics.

Findings

Injected symbolic features influence data efficiency and generalization patterns.

Training G-SSNNs reveals semantic properties of symbolic programs.

G-SSNNs demonstrate improved generalization with symbolic feature integration.

Abstract

Neural networks adapt very well to distributed and continuous representations, but struggle to generalize from small amounts of data. Symbolic systems commonly achieve data efficient generalization by exploiting modularity to benefit from local and discrete features of a representation. These features allow symbolic programs to be improved one module at a time and to experience combinatorial growth in the values they can successfully process. However, it is difficult to design a component that can be used to form symbolic abstractions and which is adequately overparametrized to learn arbitrary high-dimensional transformations. I present Graph-based Symbolically Synthesized Neural Networks (G-SSNNs), a class of neural modules that operate on representations modified with synthesized symbolic programs to include a fixed set of local and discrete features. I demonstrate that the choice of injected features within a G-SSNN module modulates the data efficiency and generalization of baseline neural models, creating predictable patterns of both heightened and curtailed generalization. By training G-SSNNs, we also derive information about desirable semantics of symbolic programs without manual engineering. This information is compact and amenable to abstraction, but can also be flexibly recontextualized for other high-dimensional settings. In future work, I will investigate data efficient generalization and the transferability of learned symbolic representations in more complex G-SSNN designs based on more complex classes of symbolic programs. Experimental code and data are available at https://github.com/shlomenu/symbolically_synthesized_networks .