Neural annealing and visualization of autoregressive neural networks in the Newman-Moore model

TL;DR

This paper investigates the limitations of autoregressive neural networks in modeling complex glassy systems, revealing that chaotic loss landscapes and mode collapse hinder their ability to find degenerate ground states in the Newman-Moore model.

Contribution

It introduces neural annealing to study glassy systems and uncovers fundamental trainability issues caused by fracton excitations and chaotic landscapes.

Findings

Neural annealing dynamics are globally unstable due to chaotic loss landscapes.

Neural networks struggle to find degenerate ground states because of mode collapse.

Glassiness and fracton excitations cause trainability issues in neural network models.

Abstract

Artificial neural networks have been widely adopted as ansatzes to study classical and quantum systems. However, some notably hard systems such as those exhibiting glassiness and frustration have mainly achieved unsatisfactory results despite their representational power and entanglement content, thus, suggesting a potential conservation of computational complexity in the learning process. We explore this possibility by implementing the neural annealing method with autoregressive neural networks on a model that exhibits glassy and fractal dynamics: the two-dimensional Newman-Moore model on a triangular lattice. We find that the annealing dynamics is globally unstable because of highly chaotic loss landscapes. Furthermore, even when the correct ground state energy is found, the neural network generally cannot find degenerate ground-state configurations due to mode collapse. These findings indicate that the glassy dynamics exhibited by the Newman-Moore model caused by the presence of fracton excitations in the configurational space likely manifests itself through trainability issues and mode collapse in the optimization landscape.