# Role of the spin-orbit coupling in the Kugel-Khomskii model on the honeycomb lattice

## TL;DR

This paper investigates the interplay of spin-orbit coupling and superexchange interactions in a honeycomb lattice model, revealing conditions for zigzag magnetic order and connections to quantum spin liquids relevant for Kitaev materials.

## Contribution

It introduces a comprehensive approach combining perturbation theory and mean-field methods to analyze the Kugel-Khomskii model with spin-orbit coupling on the honeycomb lattice, highlighting the emergence of zigzag order.

## Key findings

- Zigzag ordered state is stabilized within nearest-neighbor interactions.
- The model shows competition between magnetic order and nonmagnetic quantum spin liquid states.
- Thermodynamic properties are characterized in relation to spin-orbit coupling effects.

## Abstract

We study the effective spin-orbital model for honeycomb-layered transition metal compounds, applying the second-order perturbation theory to the three-orbital Hubbard model with the anisotropic hoppings. This model is reduced to the Kitaev model in the strong spin-orbit coupling limit. Combining the cluster mean-field approximations with the exact diagonalization, we treat the Kugel-Khomskii type superexchange interaction and spin-orbit coupling on an equal footing to discuss ground-state properties. We find that a zigzag ordered state is realized in the model within nearest-neighbor interactions. We clarify how the ordered state competes with the nonmagnetic state, which is adiabatically connected to the quantum spin liquid state realized in a strong spin-orbit coupling limit. Thermodynamic properties are also addressed. The present work should provide another route to account for the Kitaev-based magnetic properties in candidate materials.