Fine-tuning Vision Transformers for the Prediction of State Variables in Ising Models

TL;DR

This paper demonstrates that Vision Transformers can effectively predict state variables in 2D Ising model simulations, outperforming CNNs with limited data, and suggests potential for broader applications in physics simulations.

Contribution

Introduces the application of Vision Transformers to predict Ising model states, showing superior performance over CNNs with small datasets.

Findings

ViT outperforms CNNs in Ising model state prediction

ViT requires fewer microstate images for accurate predictions

Potential for applying ViT to other physical simulations

Abstract

Transformers are state-of-the-art deep learning models that are composed of stacked attention and point-wise, fully connected layers designed for handling sequential data. Transformers are not only ubiquitous throughout Natural Language Processing (NLP), but, recently, they have inspired a new wave of Computer Vision (CV) applications research. In this work, a Vision Transformer (ViT) is applied to predict the state variables of 2-dimensional Ising model simulations. Our experiments show that ViT outperform state-of-the-art Convolutional Neural Networks (CNN) when using a small number of microstate images from the Ising model corresponding to various boundary conditions and temperatures. This work opens the possibility of applying ViT to other simulations, and raises interesting research directions on how attention maps can learn about the underlying physics governing different phenomena.