Learning disentangled representation for classical models

TL;DR

This paper uses the information bottleneck method and $eta$-VAE to find disentangled representations of classical models, revealing connections to physical properties and proposing a physics-informed neural network architecture.

Contribution

It introduces a novel approach applying $eta$-VAE to classical models, revealing physical insights and proposing a modified architecture for complex models with non-binary variables.

Findings

Disentangled features relate to physical order parameters in the Ising model.

Bernoulli decoder learns a mean-field Hamiltonian at fixed temperature.

Proposed $eta^2$-VAE enforces thermal fluctuations in complex classical models.

Abstract

Finding disentangled representation plays a predominant role in the success of modern deep learning applications, but the results lack a straightforward explanation. Here we apply the information bottleneck method and its $\beta$-VAE implementation to find the disentangled low-dimensional representation of classical models. For the Ising model, our results reveal a deep connection between the disentangled features and the physical order parameters, and the widely-used Bernoulli decoder is found to be learning a mean-field Hamiltonian at fixed temperature. This analogy motivates us to extend the application of $\beta$-VAE to more complex classical models with non-binary variables using different decoder neural network and propose a modified architecture $\beta^2$-VAE to enforce thermal fluctuations in generated samples. Our work provides a way to design novel physics-informed algorithm that can yield learned features in potential correspondence with real physical properties.