Nematic excitonic insulator in transition metal dichalcogenide moir\'e heterobilayers

TL;DR

This paper investigates how Coulomb interactions induce a nematic excitonic insulator phase in transition metal dichalcogenide moiré heterobilayers, which influences the topological phase transition and breaks rotational symmetry.

Contribution

It introduces a unified two-band model that explains the competition between nematic excitonic insulator and topological phases in TMD moiré heterobilayers.

Findings

Discovery of a nematic excitonic insulator phase preempting topological transition.

Identification of the role of Coulomb interactions and tunneling in phase competition.

Demonstration of the switching between s-wave and p-wave excitons with tunneling increase.

Abstract

We study the effect of inter-electron Coulomb interactions on the displacement field induced topological phase transition in transition metal dichalcogenide (TMD) moir\'e heterobilayers. We find a nematic excitonic insulator (NEI) phase that breaks the moir\'e superlattice's three-fold rotational symmetry and preempts the topological phase transition in both AA and AB stacked heterobilayers when the interlayer tunneling is weak, or when the Coulomb interaction is not strongly screened. The nematicity originates from the frustration between the nontrivial spatial structure of the interlayer tunneling, which is crucial to the existence of the topological Chern band, and the interlayer coherence induced by the Coulomb interaction that favors uniformity in layer pseudo-spin orientations. We construct a unified effective two-band model that captures the physics near the band inversion and applies to both AA and AB stacked heterobilayers. Within the two-band model, the competition between the NEI phase and the Chern insulator phase can be understood as the switching of the energetic order between the $s$-wave and the $p$-wave excitons upon increasing the interlayer tunneling.