# Nematic order on the surface of a three-dimensional topological insulator

## TL;DR

This paper investigates how strong electron-electron interactions induce nematic order on the surface of three-dimensional topological insulators, leading to symmetry-breaking phases with distinct electronic and spin properties.

## Contribution

It introduces the concept of spin-involved nematic order in topological insulator surfaces and characterizes the phase transitions and observable effects associated with this order.

## Key findings

- First-order transition at zero doping between isotropic and nematic phases.
- Thermal phase transition changes from first to second order at a tricritical point.
- Observable anisotropies and non-Fermi liquid behavior near the quantum critical point.

## Abstract

We study the spontaneous breaking of rotational symmetry in the helical surface state of three-dimensional topological insulators due to strong electron-electron interactions, focusing on time-reversal invariant nematic order. Owing to the strongly spin-orbit coupled nature of the surface state, the nematic order parameter is linear in the electron momentum and necessarily involves the electron spin, in contrast with spin-degenerate nematic Fermi liquids. For a chemical potential at the Dirac point (zero doping), we find a first-order phase transition at zero temperature between isotropic and nematic Dirac semimetals. This extends to a thermal phase transition that changes from first to second order at a finite-temperature tricritical point. At finite doping, we find a transition between isotropic and nematic helical Fermi liquids that is second order even at zero temperature. Focusing on finite doping, we discuss various observable consequences of nematic order, such as anisotropies in transport and the spin susceptibility, the partial breakdown of spin-momentum locking, collective modes and induced spin fluctuations, and non-Fermi liquid behavior at the quantum critical point and in the nematic phase.