Softly Induced Functional Simplicity: Implications for Neural Network Generalisation, Robustness, and Distillation

TL;DR

This paper investigates how soft symmetry-based inductive biases influence the complexity, generalisation, and robustness of neural network solutions, demonstrating that lower complexity solutions lead to better abstractions and transferability.

Contribution

It introduces a novel analysis of soft symmetry respecting biases and their role in inducing pseudo-Goldstone modes, linking functional complexity to generalisation and robustness in neural networks.

Findings

Lower complexity solutions are more generalisable.

Solutions with reduced complexity are more robust to input perturbations.

Lower complexity solutions are more efficiently distillable.

Abstract

Learning robust and generalisable abstractions from high-dimensional input data is a central challenge in machine learning and its applications to high-energy physics (HEP). Solutions of lower functional complexity are known to produce abstractions that generalise more effectively and are more robust to input perturbations. In complex hypothesis spaces, inductive biases make such solutions learnable by shaping the loss geometry during optimisation. In a HEP classification task, we show that a soft symmetry respecting inductive bias creates approximate degeneracies in the loss, which we identify as pseudo-Goldstone modes. We quantify functional complexity using metrics derived from first principles Hessian analysis and via compressibility. Our results demonstrate that solutions of lower complexity give rise to abstractions that are more generalisable, robust, and efficiently distillable.