Controlling the low-temperature Ising model using spatiotemporal Markov decision theory

TL;DR

This paper introduces a new spatiotemporal Markov decision process framework to control the low-temperature Ising model, using auxiliary MDPs and Bellman equations to find optimal spin-flip policies.

Contribution

It develops the STMDP framework for spatial-temporal decision problems and applies it to control the Ising model, including constructing an auxiliary MDP for optimal policy analysis.

Findings

Optimal policies effectively drive the system to the all-plus state.

Auxiliary MDP simplifies the analysis of complex configuration spaces.

Numerical results demonstrate the policy's performance in the Ising model.

Abstract

We introduce the spatiotemporal Markov decision process (STMDP), a special type of Markov decision process that models sequential decision-making problems which are not only characterized by temporal, but also by spatial interaction structures. To illustrate the framework, we construct an STMDP inspired by the low-temperature two-dimensional Ising model on a finite, square lattice, evolving according to the Metropolis dynamics. We consider the situation in which an external decision maker aims to drive the system towards the all-plus configuration by flipping spins at specified moments in time. In order to analyze this problem, we construct an auxiliary MDP by means of a reduction of the configuration space to the local minima of the Hamiltonian. Leveraging the convenient form of this auxiliary MDP, we uncover the structure of the optimal policy by solving the Bellman equations in a recursive manner. Finally, we conduct a numerical study on the performance of the optimal policy obtained from the auxiliary MDP in the original Ising STMDP.