Possible Nematic Spin Liquid in Spin-$1$ Antiferromagnetic System on the Square Lattice: Implication for the Nematic Paramagnetic State of FeSe

TL;DR

This paper proposes a nematic quantum spin liquid phase in a frustrated spin-1 system, offering insights into the nematic paramagnetic state of FeSe and explaining experimental observations of its magnetic fluctuations.

Contribution

It introduces a candidate nematic quantum spin liquid phase in a frustrated spin-1 model, supported by large-scale DMRG calculations, linking theory to FeSe's properties.

Findings

Identification of a nematic quantum spin liquid phase

Absence of magnetic and quadrupolar orders in this phase

Explanation of FeSe's enhanced spin fluctuations

Abstract

The exotic normal state of iron chalcogenide superconductor FeSe, which exhibits vanishing magnetic order and possesses an electronic nematic order, triggered extensive explorations of its magnetic ground state. To understand its novel properties, we study the ground state of a highly frustrated spin-$1$ system with bilinear-biquadratic interactions using unbiased large-scale density matrix renormalization group. Remarkably, with increasing biquadratic interactions, we find a paramagnetic phase between N\'eel and stripe magnetic ordered phases. We identify this phase as a candidate of nematic quantum spin liquid by the compelling evidences, including vanished spin and quadrupolar orders, absence of lattice translational symmetry breaking, and a persistent non-zero lattice nematic order in the thermodynamic limit. The established quantum phase diagram natually explains the observations of enhanced spin fluctuations of FeSe in neutron scattering measurement and the phase transition with increasing pressure. This identified paramagnetic phase provides a new possibility to understand the novel properties of FeSe.