Characterising the Inductive Biases of Neural Networks on Boolean Data
Chris Mingard, Lukas Seier, Niclas G\"oring, Andrei-Vlad Badelita, Charles London, Ard Louis
TL;DR
This paper provides an analytically tractable case study linking neural network inductive biases, training dynamics, and generalisation by analyzing depth-2 networks trained on Boolean functions, revealing interpretable feature emergence.
Contribution
It introduces a novel framework connecting network inductive bias, feature learning, and generalisation using Boolean function training, with detailed analysis of training dynamics.
Findings
Predictable training dynamics in depth-2 networks
Emergence of interpretable features during training
Link between inductive bias and generalisation
Abstract
Deep neural networks are renowned for their ability to generalise well across diverse tasks, even when heavily overparameterized. Existing works offer only partial explanations (for example, the NTK-based task-model alignment explanation neglects feature learning). Here, we provide an end-to-end, analytically tractable case study that links a network's inductive prior, its training dynamics including feature learning, and its eventual generalisation. Specifically, we exploit the one-to-one correspondence between depth-2 discrete fully connected networks and disjunctive normal form (DNF) formulas by training on Boolean functions. Under a Monte Carlo learning algorithm, our model exhibits predictable training dynamics and the emergence of interpretable features. This framework allows us to trace, in detail, how inductive bias and feature formation drive generalisation.
Peer Reviews
Decision·Submitted to ICLR 2026
1. Novel framework for understanding inductive bias of NNs. 2. New insights relating DNF complexity and learnability. 3. The paper is mostly clearly written. 4. Thorough empirical validation.
1. The main weakness is that the framework limits the input dimension substantially, allowing only n≤7. Therefore, it does not model high dimensional settings, which are key in NN applications. Since the data is binary, the resulting datasets are also very small. 2. No experiments were performed with NNs and algorithms used in practice – this can strengthen the conclusions of the paper. 3. Missing reference– Bronstein et al. (UAI 2022) that study the inductive bias of NNs on read-once DNFs. Br
1. The paper presents a clear and internally consistent analysis. The proposed DFCN–DNF correspondence is an effective way to formalize inductive bias, allowing the authors to make explicit statements about which functions a network represents. 2. The results go beyond earlier simplicity-bias discussions by providing class-specific scaling laws (as shown in Table 2). This clarifies that different Boolean function families exhibit distinct probabilistic behavior rather than a single universal exp
1. The work offers limited novelty. The simplicity bias of neural networks has been documented extensively, and this paper largely reformulates it within a discrete combinatorial model. 2. The theoretical depth is modest. the Boolean setting restricts the scope of the conclusions. 3. The MCMC and greedy search methods used for training differ substantially from gradient-based optimization, so their connection to real neural-network behavior remains unclear. 4. The presentation of related work co
Overall, the paper offers an interesting and principled analysis of a simplified theoretical model. The authors succeed in showing the connections between DNFs and DFCNs and how the function complexity leads to inductive biases and insights into networks generalization. This mapping is indeed insightful, and offers an alternative view towards representation learning.
My main concern revolves around the impact and generalization of the approach to other not so controlled settings. Indeed, while the authors aim to tie the generalization performance to derived inductive properties based on boolean function complexity, it remains highly uncertain how and if this discrete Boolean framework translates to more standard continuous, multi layer and high dimensional NNs. Even if this work's aim is a proof of concept or a presentation of the connections between disjun
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications