Time-Reversal Symmetry-Breaking Nematic Insulators near Quantum Spin Hall Phase Transitions

TL;DR

This paper explores how interactions and band hybridization influence phase transitions in quantum spin Hall systems, revealing the emergence of nematic insulators and different transition pathways.

Contribution

It demonstrates the conditions under which interaction-induced nematic insulators appear near quantum spin Hall phase transitions, highlighting the role of band hybridization and Coulomb interactions.

Findings

Weak hybridization leads to nematic insulators over a wide parameter range.

Interactions modify the topological phase transition, creating a broken symmetry state.

At intermediate hybridization, a quantum anomalous Hall insulator emerges.

Abstract

We study the phase diagram of a model quantum spin Hall system as a function of band inversion and band-coupling strength, demonstrating that when band hybridization is weak, an interaction-induced nematic insulator state emerges over a wide range of band inversion. This property is a consequence of the long-range Coulomb interaction, which favors interband phase coherence that is weakly dependent on momentum and therefore frustrated by the single-particle Hamiltonian at the band inversion point. For weak band hybridization, interactions convert the continuous gap closing topological phase transition at inversion into a pair of continuous phase transitions bounding a state with broken time-reversal and rotational symmetries. At intermediate band hybridization, the topological phase transition proceeds instead via a quantum anomalous Hall insulator state, whereas at strong hybridization interactions play no role. We comment on the implications of our findings for InAs/GaSb and HgTe/CdTe quantum spin Hall systems.