Fragility of the nematic spin liquid induced by diagonal couplings in the square-lattice SU(3) model

TL;DR

This study uses large-scale DMRG calculations to investigate the stability of a nematic spin liquid in the SU(3) square-lattice model, revealing that diagonal couplings destabilize the spin liquid in favor of magnetic order.

Contribution

It demonstrates that diagonal couplings rapidly suppress the nematic spin liquid, favoring three-sublattice magnetic order, thus clarifying the fragility of the spin liquid state.

Findings

Diagonal couplings destabilize the nematic spin liquid.

Three-sublattice order develops with infinitesimal additional coupling.

Nematic spin liquid is fragile against lattice frustration.

Abstract

We present a large-scale density matrix renormalization group (DMRG) study of the spin-$1$ SU(3) bilinear-biquadratic model on the square lattice, which was suggested to host a nematic spin liquid state in recent DMRG calculations. We report that this spin liquid appears to strongly compete with a three-sublattice magnetic order. To further study the competition between the two states, and the reason of the emergent nematic spin liquid, we included an additional next-nearest-neighbor SU(3) symmetric interactions along one of the two plaquette diagonal directions. This allows to tune the square-lattice model to the triangular-lattice model. By computing spin correlation functions and various order parameters, we find that the three-sublattice order may develop at infinitesimal additional new coupling, at least within the precision of our study. Compared with the previous findings that the nematic spin liquid is stable in extended parameter regions with additional couplings that respect the lattice symmetries of the square lattice, we argue that here the diagonal couplings, which frustrate the bipartite-lattice structure, rapidly suppress the two-sublattice fluctuations and the three-sublattice order thus wins. This numerical result is consistent with the conjecture that the nematic spin liquid emerges from the competition between two- and three-sublattice fluctuations.