Global phase diagram and quantum spin liquids in spin-1/2 triangular antiferromagnet

TL;DR

This study maps the phase diagram of a spin-1/2 triangular antiferromagnet, revealing magnetic orders and chiral spin liquids, with detailed analysis of phase transitions and topological properties using large-scale DMRG calculations.

Contribution

It provides the first comprehensive phase diagram including scalar chiral interactions and identifies a chiral spin liquid as a fractional quantum Hall state.

Findings

Identified a chiral spin liquid phase with fractional quantum Hall characteristics.

Mapped magnetic and non-magnetic phases in the J1-J2-Jchi model.

Found large spin triplet gap in odd sectors and small or vanishing gap in even sectors.

Abstract

We study the spin-$1/2$ Heisenberg model on the triangular lattice with the nearest-neighbor $J_1 > 0$, the next-nearest-neighobr $J_2 > 0$ Heisenberg interactions, and the additional scalar chiral interaction $J_{\chi}(\vec{S}_i \times \vec{S}_j) \cdot \vec{S}_k$ for the three spins in all the triangles using large-scale density matrix renormalization group calculation on cylinder geometry. With increasing $J_2$ ($J_2/J_1 \leq 0.3$) and $J_{\chi}$ ($J_{\chi}/J_1 \leq 1.0$) interactions, we establish a quantum phase diagram with the magnetically ordered $120^{\circ}$ phase, stripe phase, and non-coplanar tetrahedral phase. In between these magnetic order phases, we find a chiral spin liquid (CSL) phase, which is identified as a $\nu = 1/2$ bosonic fractional quantum Hall state with possible spontaneous rotational symmetry breaking. By switching on the chiral interaction, we find that the previously identified spin liquid in the $J_1 - J_2$ triangular model ($0.08 \lesssim J_2/J_1 \lesssim 0.15$) shows a phase transition to the CSL phase at very small $J_{\chi}$. We also compute spin triplet gap in both spin liquid phases, and our finite-size results suggest large gap in the odd topological sector but small or vanishing gap in the even sector. We discuss the implications of our results to the nature of the spin liquid phases.

Equations

$$H=J_1\sum _{⟨i,j⟩}{\vec{S}}_i\cdot {\vec{S}}_j+J_2\sum _{⟨⟨i,j⟩⟩}{\vec{S}}_i\cdot {\vec{S}}_j+J_{\chi }\sum _{△/▽}({\vec{S}}_i\times {\vec{S}}_j)\cdot {\vec{S}}_k,$$

Full text

I Introduction

Quantum spin liquid (QSL) is one kind of long-range entangled states with fractionalized quasiparticles Savary and Balents (2016). Since the proposal by P. W. Anderson, the concept of QSL has been playing an important role for understanding strongly correlated materials and unconventional superconductors Lee et al. (2006). Although QSLs have been pursued for more than two decades Rokhsar and Kivelson (1988); Baskaran (1989); Read and Sachdev (1991); Sachdev (1992); Yang et al. (1993); Balents et al. (2002); Yao and Kivelson (2007), only recently such novel states have been found in realistic spin models Kitaev (2006); Schroeter et al. (2007); Thomale et al. (2009); Nielsen et al. (2012); Bauer et al. (2014); Gong et al. (2014); He et al. (2014); Hu et al. (2015a); Meng et al. (2015); Gorohovsky et al. (2015); Kumar et al. (2015); Wietek et al. (2015); Hickey et al. (2016); Zhu et al. (2016), in which geometric frustration and competing interactions play important roles for developing spin liquid states.

One of the most promising spin liquid candidates is the antiferromagnet on the corner-sharing kagome lattice. Experimentally, spin liquid-like behaviors have been observed in several kagome materials such as herbertsmithite Mendels et al. (2007); Helton et al. (2007); Han et al. (2012); Fu et al. (2015); Norman (2016). Theoretically, the most extensively studied kagome model is the spin-$1/2$ kagome Heisenberg model with the nearest-neighbor (NN) interaction. Thanks to the recent large-scale Density Matrix Renormalization Group (DMRG) simulations Yan et al. (2011); Depenbrock et al. (2012), conventional orders have been excluded, leading to a QSL ground state. However, the nature of this spin liquid is still in debate. DMRG calculations suggest a gapped spin liquid Yan et al. (2011); Depenbrock et al. (2012); Jiang et al. (2012), seemingly consistent with a $Z_{2}$ topological order Depenbrock et al. (2012); Jiang et al. (2012). Recent tensor network state simulations identify the $Z_{2}$ topological order of the obtained variational wavefunction Mei et al. (2017), but so far the four degenerate ground states of the putative $Z_{2}$ QSL have not been found in exact diagonalization (ED) Waldtmann et al. (1998); Läuchli et al. (2016) and DMRG calculations, leaving this problem open. On the other hand, variational studies based on the fermionic parton wavefunctions find the gapless U(1) Dirac spin liquid rather than the gapped $Z_{2}$ spin liquid with the optimized variational energy Ran et al. (2007); Iqbal et al. (2013, 2014). Very recently, tensor renormalization group Jiang et al. (2016); Liao et al. (2017) and DMRG He et al. (2016) calculations also suggest the gapless spin liquid as a strong candidate. Interestingly, studies on the modified kagome models Messio et al. (2012); Gong et al. (2014); He et al. (2014); Gong et al. (2015); Bauer et al. (2014); Iqbal et al. (2015) find that the kagome spin liquid emerges near the phase boundaries of several ordered phases, suggesting possible strong competitions of the different physics in the kagome spin liquid regime. In particular, a fully gapped chiral spin liquid (CSL) Kalmeyer and Laughlin (1987); Wen et al. (1989) is found by switching on small further-neighbor  Gong et al. (2014); He et al. (2014) or chiral interactions Bauer et al. (2014) on the NN kagome model.

Another promising spin liquid candidate is the antiferromagnet on the edge-sharing triangular lattice. Although frustration is present in the spin-$1/2$ NN triangular model, it turns out to still exhibit a $120^{\circ}$ antiferromagnetic order Bernu et al. (1992); Capriotti et al. (1999). In the recent experiments on the triangular organic Mott insulators such as $\kappa$-(ET)2Cu2(CN)3 and EtMe3Sb[Pd(dmit)2]2 Shimizu et al. (2003); Kurosaki et al. (2005); Yamashita et al. (2008, 2009, 2010), spin liquid-like behaviors have been found. Theoretically, multi-spin exchange interactions, which can lead to the gapless spin Bose metal with a large spinon Fermi surface Misguich et al. (1999); Motrunich (2005); Sheng et al. (2009) and the gapless spin liquid with a quadratic band touching Mishmash et al. (2013); Bieri et al. (2016) depending on the strength of interaction, and the space anisotropic interaction Yunoki and Sorella (2006); Weng et al. (2006); Starykh and Balents (2007); Weichselbaum and White (2011) have been suggested to understand the spin-liquid behaviors in triangular materials.

Recently, a new spin liquid phase is found in the spin-$1/2$ triangular Heisenberg model with the NN $J_{1}$ and the next-nearest-neighbor (NNN) $J_{2}$ interactions for $0.08\lesssim J_{2}/J_{1}\lesssim 0.15$, which is sandwiched by a $120^{\circ}$ magnetic phase and a stripe magnetic order phase Mishmash et al. (2013); Kaneko et al. (2014); Li et al. (2015); Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016); Iqbal et al. (2016). This frustrating $J_{2}$ interaction is considered as a possible mechanism to understand the spin-liquid behaviors of the newly synthesized triangular materials YbMgGaO4 Paddison et al. (2016) and Ba3InIr2O9 Dey et al. (2017). For this $J_{1}-J_{2}$ model, DMRG calculations on cylinder system find the evidence of spin liquid including the two near-degenerate ground states in the even and odd topological sectors whose energy difference decays rapidly with growing cylinder width, and the fractionalized spin-$1/2$ quasiparticle revealed by inserting flux simulation and entanglement spectrum (ES) Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016). On the finite-size DMRG calculations, the spin triplet gap measured above the overall ground state (in the odd sector) is big ($\Delta_{\rm T}\sim 0.3J_{1}$) Zhu and White (2015); Hu et al. (2015b), seemingly consistent with a gapped spin liquid Zheng et al. (2015); Lu (2016). Nonetheless, the even and odd sectors show some distinct features in finite-size DMRG calculation. While the odd sector shows a short correlation length that could be consistent with the large gap, the even sector exhibits a much larger one Hu et al. (2015b); Saadatmand and McCulloch (2016), which may suggest smaller gap in the even sector. The low-lying entanglement spectrum in the even sector shows a Dirac node like structure, which is suggested as an implication of gapless spinon excitations Saadatmand and McCulloch (2016). The different DMRG results in the two sectors reasonably imply that either the putative gapped spin liquid is not yet well developed as the strong finite-size effects in numerical calculation, or a gapless spin liquid is possible. In the variational study, a U(1) Dirac gapless spin liquid indeed possesses the best variational energy Iqbal et al. (2016). The nature of this spin liquid remains an open question. To shed more light on this spin liquid phase, the modified $J_{1}-J_{2}$ triangular models have been investigated Yao et al. (2015); Hu et al. (2016); Wietek and Läuchli (2017); Misumi et al. (2017). Interestingly, the variational Hu et al. (2016) and ED calculations Wietek and Läuchli (2017) suggest a possible CSL at the neighbor of the $J_{1}-J_{2}$ spin liquid, which seems to be similar to the situation in the kagome model and deserves more studies. Besides, the quantum phase transition between the two spin liquid phases is also far from clear.

In this article, we study the spin-$1/2$ $J_{1}-J_{2}$ Heisenberg model on the triangular lattice with additional time-reversal symmetry (TRS) breaking chiral interaction $J_{\chi}$ using DMRG simulations. The model Hamiltonian is given as

[TABLE]

where $J_{1}$ and $J_{2}$ denote the NN and the NNN interactions, respectively. The scalar chiral interaction $J_{\chi}$ has the same magnitude for all the up ($\bigtriangleup$) and down ($\bigtriangledown$) triangles, and the three sites $i,j,k$ for $J_{\chi}$ follow the clockwise order in all the triangles as shown in Fig. 1(a). Physically, the scalar chiral interaction $J_{\chi}$ term can be induced in the Hubbard model with large $U$ in a magnetic field Sen and Chitra (1995); Motrunich (2006). Starting from the Hubbard model, a $t/U$ ($t$ and $U$ are the hopping and interaction respectively) expansion to the second order at half filling gives the effective chiral interaction $J_{\chi}(\vec{S}_{i}\times\vec{S}_{j})\cdot\vec{S}_{k}$ with $J_{\chi}\sim\Phi t^{3}/U^{2}$, where $\Phi$ is the magnetic flux enclosed by the triangle. We take $J_{1}=1.0$ as the energy scale. Using DMRG simulation, we obtain a quantum phase diagram as shown in Fig. 1(d). Besides the $120^{\circ}$ Néel phase, the stripe phase, and the time-reversal invariant spin liquid in the $J_{1}-J_{2}$ model (here we denote it as $J_{1}-J_{2}$ SL), we find a large regime of the non-coplanar tetrahedral order for large $J_{\chi}$, whose spin configuration is shown in Fig. 1(c). Below the tetrahedral phase for $J_{2}\lesssim 0.25$, we identify a CSL as the $\nu=1/2$ bosonic fractional quantum Hall state by observing the gapless chiral edge mode. The strong nematic order of bond energy suggests a possible spontaneous lattice rotational symmetry breaking and implies an emergent nematic CSL. By studying the spin triplet gap and entanglement spectrum, we observe a transition from the $J_{1}-J_{2}$ SL to the CSL at small chiral interaction. While we find a large spin triplet gap above the overall ground state (in the odd sector) in the CSL phase, the small triplet gap in the even sector suggests that on our studied system size the topological nature in the even sector may not have been fully developed. A possible reason is that this CSL regime generated by increasing $J_{\chi}$ is near the phase boundaries from the CSL to the neighboring phases. In the $J_{1}-J_{2}$ triangular model, the triplet gap in the even sector seems to be vanished, which could be consistent with the larger correlation length found in DMRG calculation Hu et al. (2015b); Saadatmand and McCulloch (2016) and may suggest a possible gapless spin liquid Iqbal et al. (2016), which deserves more studies.

We study the system with cylinder geometry using DMRG White (1992) with spin rotational SU(2) symmetry McCulloch and Gulácsi (2002). We choose two geometries that have one lattice direction parallel to either the $x$ axis (XC) or the $y$ axis (YC), as shown in Figs. 1(a)-(b). These cylinders are denoted as XC(YC)$L_{y}$-$L_{x}$, where $L_{y}$ and $L_{x}$ are the numbers of site along the two directions. To study the phase diagram and characterize the CSL phase, we perform calculations on the systems with $L_{y}$ up to $8$ and $10$. We keep up to $4000$ SU(2) states to obtain accurate results with the truncation error less than $10^{-5}$ in most calculations.

II Tetrahedral order and 120*∘* order

For $J_{2}=0.0$, the triangular model has a coplanar $120^{\circ}$ magnetic order at $J_{\chi}=0.0$ Bernu et al. (1992); Capriotti et al. (1999); Zheng et al. (2006); White and Chernyshev (2007). In the large $J_{\chi}$ limit, a classical spin analysis finds a tetrahedral magnetic state with the spins of the four sublattices pointing toward the corners of a tetrahedron Messio et al. (2011) (see Fig. 1(b)). In the classical picture, the $120^{\circ}$ ordered state has the energy per site $E_{120}=-3J_{1}/2+3J_{2}$, the stripe ordered state has the energy $E_{\rm stripe}=-J_{1}-J_{2}$, and the tetrahedral ordered state has the energy $E_{\rm tetra}=-J_{1}-J_{2}-8\sqrt{3}J_{\chi}/9$. Thus we can get a classical phase diagram in the $J_{2}-J_{\chi}$ plane. In Fig. 1(d), the dotdashed line denotes the classical phase boundary between the $120^{\circ}$ and the tetrahedral phase. For $J_{\chi}=0.0,J_{2}>0.125$, the stripe state and the tetrahedral state have the degenerate energy. By switching on the chiral interaction, the tetrahedral state immediately gets the lower energy. In quantum model, we first investigate whether this tetrahedral order could survive for the spin-$1/2$ system with strong quantum fluctuations. In Figs. 2(a)-(b), we demonstrate the spin correlations $\langle\vec{S}_{i}\cdot\vec{S}_{j}\rangle$ for $J_{2}=0.0,J_{\chi}=0.8$ on both the XC8-24 and YC8-24 cylinders. The dashed diamonds denote the unit cell of the spin correlation, which is consistent with the four sublattice structure of the tetrahedral order. The spin correlation functions decay quite slowly in both systems, indicating an established magnetic order. In Figs. 2(c)-(d), we show the corresponding spin structure factor $S(\vec{q})=\frac{1}{N}\sum_{i,j}\langle\vec{S}_{i}\cdot\vec{S}_{j}\rangle e^{i\vec{q}\cdot(\vec{r}_{i}-\vec{r}_{j})}$ of the tetrahedral state on both cylinder geometries, which has the ordering peaks at $\vec{q}=(0,\pi),(\pi,\pi/2)$ and $(\pi/2,\pi)$ on the XC8 and YC8 cylinders, respectively.

For small $J_{\chi}$ interaction we expect the $120^{\circ}$ magnetic order. In Fig. 3, we show the spin structure factor for $J_{\chi}=0.0,0.2$ on both the XC8 and YC6 cylinders. For a finite $J_{\chi}=0.2$, the characteristic peak of the $120^{\circ}$ order is still very stable, indicating the dominant three-sublattice spin structure.

III Quantum spin liquids

III.1 Chiral spin liquid

For showing our results of the chiral spin liquid phase, we choose the parameters with fixed $J_{2}=0.1$, where the system is in the $J_{1}-J_{2}$ SL in the absence of the chiral interaction Li et al. (2015); Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016); Iqbal et al. (2016) (we have also studied other $J_{2}$ such as $J_{2}=0.125$, which gives the same results). In our DMRG simulation of spin liquid phase on cylinder geometry, we control the even/odd parity of spinon flux in the ground state by removing or adding a spin-$1/2$ on each open edge of cylinder Yan et al. (2011); Zhu and White (2015); Hu et al. (2015b).

We first exclude the conventional orders in the CSL phase. We show the spin correlations in Fig. 4(a), where the correlations in the CSL phase decay faster than those in the $J_{1}-J_{2}$ SL, indicating the vanished magnetic order. In Fig. 4(b), we plot the triangle chiral order $\langle\chi_{\Delta_{i}}\rangle$ along the $x$ direction of cylinder. Different from the decayed chiral order in the $J_{1}-J_{2}$ triangular model Hu et al. (2015b); Saadatmand and McCulloch (2016), here it rapidly converges to finite value and seems to be robust with increasing system width. For $L_{y}=6,8,10$, the chiral orders in both sectors approach to each other, which agrees with the consistent local orders in different sectors of gapped spin liquid. Similar to the $J_{1}-J_{2}$ SL Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016), lattice translational symmetry is also preserved in the CSL phase, which we do not discuss in detail but show an example in Fig. 7.

Next, we characterize the CSL by identifying the conformal field theory (CFT) that describes the gapless edge excitations through entanglement spectrum Li and Haldane (2008). Since this strategy was proposed Li and Haldane (2008), the ES has been shown as a powerful tool to identify different topological orders with edge states Cincio and Vidal (2013); Zaletel et al. (2013). The nature of the CSLs in the kagome and honeycomb spin models have been characterized using ES Bauer et al. (2014); He et al. (2014); Hickey et al. (2016); Zhu et al. (2016). In Fig. 5, we show the ES of the reduced density matrix for half the cylinder in both sectors. By tracing out half of the degrees of freedom for the density matrix by bipartiting the cylinder, we obtain the reduced density matrix and its eigenvalues $\lambda_{i}$. We focus on the leading eigenvalues by showing $-\ln\lambda_{i}$ in Fig. 5. The spectra are labeled by the quantum number total spin $S^{z}$ and relative momentum quantum number along the $y$ direction $\Delta k_{y}$ Cincio and Vidal (2013); Zaletel et al. (2013). The leading spectra in both sectors have the degeneracy pattern $\{1,1,2,3,5,7,\cdots\}$ with increasing $\Delta k_{y}$ in each $S^{z}$ sector, which follow the chiral $SU(2)_{1}$ Wess-Zumino-Witten CFT theory of the $\nu=1/2$ fractional quantum Hall state Francesco et al. (2012). The spectra of the even and odd sectors are symmetric about $S^{z}=0$ and $1/2$ respectively, indicating a spin-$1/2$ at each end of cylinder in the odd (spinon) sector. The similar degeneracy pattern of entanglement spectra have also been found in the CSLs in the kagome and honeycomb spin models Bauer et al. (2014); He et al. (2014); Hickey et al. (2016); Zhu et al. (2016). We further demonstrate the near-degenerate ground states in the CSL phase. In Fig. 6, we show the bulk energies in both the even and odd sectors, where the energy difference drops fast with increasing $L_{y}$. For $L_{y}=10$, the energy difference is vanishing small, in consistent with the near-degenerate ground states in the two sectors.

For the $J_{1}-J_{2}$ SL, DMRG calculations find the large lattice nematic order in the odd sector (defined as the energy difference between the zigzag and vertical bonds) Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016), suggesting a spin liquid with possible spontaneous rotational symmetry breaking. However, the nematic order in the even sector exhibits the opposite nematic pattern from the odd sector, which seems to approach vanishing with increasing $L_{y}=6,8,10$ Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016). In the CSL phase, we also calculate the NN bond energy $\langle\vec{S}_{i}\cdot\vec{S}_{j}\rangle$ on the YC cylinder as shown in Figs. 7(a)-(b). On the YC8 cylinder, we also find the strong bond energy anisotropy and the nematic patterns are different in the two sectors. For studying the nematic order, we show the nematic order on different cylinders in Fig. 7(c) with a comparison to the data for $J_{\chi}=0.0$ Zhu and White (2015); Hu et al. (2015b); Saadatmand and McCulloch (2016). For $L_{y}=6,8,10$, while the nematic order in the odd sector keeps growing with $L_{y}$; in the even sector it also appears to approach zero. The overall behaviors of the nematic order in the CSL are consistent with those in the $J_{1}-J_{2}$ SL. We notice that the nematicity for the even sector in Fig. 7(c) shows a tendency to become positive with growing $L_{y}$. To shed more light on the nature of the nematicity, we calculate the bond energy for $L_{y}=12$ by keeping the SU(2) states up to $6000$. We find that while the nematic order in the odd sector shows a consistent behavior, the order in the even sector changes the pattern to that of the odd sector on the YC12 cylinder. Our results imply that in both the $J_{1}-J_{2}$ SL and the CSL, the even sector may also host a nematic order in large size, suggesting possible nematic spin liquids. While CSL has been discovered in several spin models, the nematic CSL with coexisting topological order and nematic order has not been reported in a microscopic spin model as far as we know. In a strong-coupling perspective, a nematic FQH may be viewed as a partially melted solid, where the nematic FQH is proximate to the phase with broken translational and rotational symmetries Fradkin et al. (2010). If the translational order is melted by tuning external parameter but nematic order is preserved, a nematic FQH might be obtained. Here, the CSL in the triangular model emerges at the neighbor of a stripe phase, which breaks translational and rotational symmetries. The possible nematicity of the CSL might be understood as a partially melted stripe order.

III.2 Transition between the two spin liquids

Now we study the quantum phase transition from the $J_{1}-J_{2}$ SL to the CSL. We choose $J_{2}=0.1$ and switch on the chiral interaction $J_{\chi}$. In Fig. 8(a), we show the ground-state energy on the YC8 cylinder as well as on the $6\times 6$ torus. The energy varies smoothly with growing $J_{\chi}$, and we notice the slight change of energy for $J_{\chi}\lesssim 0.02$. Then we compute the spin triplet gap $\Delta_{\rm T}$ on the YC8 cylinder based on the ground state with the lowest energy, which is in the odd sector as shown in Fig. 6. The triplet gap is obtained by sweeping the total spin-1 sector in the bulk of long cylinder Yan et al. (2011). We compare the obtained spin triplet gap by sweeping the spin-1 sector on the different system lengths, and we find the well converged triplet gap (one example can be found as the red square in Fig. 11(c)). In Fig. 8(b), we show the gap obtained by sweeping the middle $8\times 16$ sites in the spin-1 sector based on the ground state in the odd sector on the YC8-24 cylinder. The triplet gap changes slightly for $J_{\chi}\lesssim 0.02$. Above $J_{\chi}\simeq 0.02$, the gap grows fast, consistent with the non-zero gap in the CSL phase. The $J_{\chi}$ dependence of energy and triplet gap imply a possible phase transition at small $J_{\chi}$.

Next, we study the entanglement spectrum. As shown in Figs. 9(a)-(b) for $J_{\chi}=0.01,0.02$ in the odd sector with total spin $S^{z}=0$, the ES exhibit some features of the ES for $J_{\chi}=0.0$ Saadatmand and McCulloch (2016), where four eigenvalues are found below the higher spectrum. We also notice that with increasing $J_{\chi}$, some eigenvalues in the higher spectrum are decreasing gradually as marked by red in Figs. 9(a)-(b). For $J_{\chi}=0.05,0.1$ as shown in Figs. 9(c)-(d), the decreasing eigenvalues seem to merge with the low-lying levels, which are separated by an ES gap from the higher spectrum. The ES levels below the gap exhibit the near degenerate pattern $\{1,1,2,3\}$, which is consistent with the Laughlin CSL. The entanglement spectrum also suggests a phase transition at small $J_{\chi}$, which agrees with the transition suggested by energy and triplet gap in Fig. 8. In the even sector for $J_{\chi}=0.0$, the low-lying part of the ES shows a deformed two-spinon continuum structure Saadatmand and McCulloch (2016). By switching on the chiral interaction, the low-lying part of the ES quickly changes to the structure that looks like the one in the CSL, which are shown in Fig. 10 and may suggest a stronger tendency to the chiral state in the even sector.

We also compute the spin triplet gap. For a comparison, we demonstrate the results of the same calculation for the spin-$1/2$ $J_{1}-J_{2}-J_{3}$ kagome model. Here, we obtain the triplet gap by calculating the ground state on long cylinder first (for example YC8-40) and then sweeping the bulk sites in the total spin $S=1$ sector for the given bulk length $l_{x}$ Yan et al. (2011). For the $J_{1}-J_{2}$ triangular model shown in Fig. 11(a), the spin gap measured from the overall ground state (in the odd sector) is robust with increasing $l_{x}$. However, the gap measured from the ground state in the even sector decreases with $l_{x}$. On large $l_{x}$, the ground state in the even sector has the higher energy than the obtained state in the spin-1 sector, which suggests that the triplet gap is lower than the total energy difference between the two sectors and is unlikely to support a well established gapped spin liquid. For the $J_{1}$ kagome model ($J_{2}=J_{3}=0.0$) as shown in Fig. 11(b) on the YC8 cylinder, while the triplet gap measured from the overall ground state (in the even sector for the kagome spin liquid) is quite robust (consistent with the previous result Yan et al. (2011); Depenbrock et al. (2012)), the gap in the odd sector decreases fast with $l_{x}$ and tends to vanish, seemly similar to the behaviors found in Fig. 11(a).

In the CSL phase as shown in Fig. 11(c), the overall triplet gap is also robust; but the gap in the even sector still decreases with $l_{x}$. In the well established CSL phase, for example the CSL phase in the kagome model as shown in Fig. 11(d) Gong et al. (2014), one can find the robust triplet gap in both sectors. The decreasing gap of the triangular CSL in the even sector suggests that on our studied system size the topological nature in the even sector is not fully developed. A possible reason is that this CSL regime is very close to the phase boundaries from the CSL to the neighboring phases.

IV Quantum phase diagram

First of all, we study the phase transition from the tetrahedral phase to the CSL phase by calculating magnetic structure factor. In Fig. 12, we show the $J_{\chi}$ dependence of the tetrahedral structure factor peak on the YC8 cylinder, which is at $\vec{q}=(\pi/2,\pi)$. With increasing $J_{\chi}$, $S(\pi/2,\pi)$ shows a jump that characterizes the phase transition. Above the transition $J_{\chi}$, we find that the spin correlations decay quite slowly, consistent with a magnetic order developed. On the smaller YC6 and XC8 cylinders, the tetrahedral structure factor peak appears to increase smoothly, which may be owing to the finite-size effects. We show the phase boundary in Fig. 1 based on the results on the larger YC8 cylinder.

By increasing the chiral interaction $J_{\chi}$ in the $120^{\circ}$ phase for $J_{2}\lesssim 0.08$, the magnetic order is suppressed, leading to a transition from the magnetic order to the non-magnetic spin liquid phase. On our studied system size, we do not find sharp features to characterize this transition. Thus we estimate a qualitative phase boundary as shown in Fig. 1(c) by comparing spin correlation function and magnitude of spin structure factor on the YC8 cylinder with the results at $J_{2}=0.08,J_{\chi}=0$, where the system has the transition from the $120^{\circ}$ phase to the $J_{1}-J_{2}$ SL phase. We roughly take the parameter points which have the similar magnitudes of spin correlations and spin structure factor compared with those at $J_{2}=0.08,J_{\chi}=0$ as the phase boundary. In Fig. 13, we show the spin structure factor $S(\vec{q})$, which characterizes the present and absent magnetic order in the two phases.

As shown in Fig. 1(c), the stripe phase goes to the tetrahedral phase with or without an intermediate CSL phase depending on $J_{2}$. We calculate the spin correlations and structure factor on the YC8 cylinder. In both the stripe and the tetrahedral phase, spin structure factor shows the peak at $\vec{q}=(\pi/2,\pi)$. For the case with the intermediate CSL phase, we find that $S(\pi/2,\pi)$ first decreases with growing $J_{\chi}$ and then keeps small in the CSL phase; with further increasing $J_{\chi}$, $S(\pi/2,\pi)$ shows a jump at the transition to the tetraheral phase, which is shown for $J_{2}=0.225$ in Fig. 14. On the other hand, for $J_{2}\gtrsim 0.25$, $S(\pi/2,\pi)$ decreases fast with $J_{\chi}$ in the stripe phase and changes slowly in the tetrahedral phase, showing a kink to characterize the phase transition. We remark that on the smaller YC6 cylinder for the $J_{1}-J_{2}$ model, the magnetic order is weak at $0.15\lesssim J_{2}\lesssim 0.2$; however, on the larger YC and XC cylinders the stripe order is quite robust. We demonstrate our phase diagram based on the large-size results.

V Summary and discussion

We have studied the competing quantum phases of the spin-$1/2$ triangular $J_{1}-J_{2}$ Heisenberg model with the additional scalar chiral interaction $J_{\chi}$ for $0\leq J_{2}\leq 0.3$ and $0\leq J_{\chi}\leq 1.0$ by DMRG simulations. As shown in Fig. 1(c), we find five phases: a non-coplanar tetrahedral magnetic order phase, a $120^{\circ}$ order phase, a stripe order phase, a $J_{1}-J_{2}$ SL phase, and a chiral spin liquid (CSL) phase. The CSL is identified as the $\nu=1/2$ bosonic fractional quantum Hall state, which seems to arise as a result of quantum fluctuations around the phase boundaries of the classical magnetic orders. In particular, the CSL state exhibits a strong bond anisotropy, strongly suggesting a nematic CSL with coexisting topological order and nematic order. We argue that the nematicity can be understood as a partially melted stripe order. The emergent CSL induced from quantum fluctuations around classical phase boundaries has also been found in the kagome Gong et al. (2015); Zhu et al. (2016); Wietek et al. (2015) and honeycomb Hickey et al. (2016) models, which may represent a common mechanism to generate novel quantum phases from frustration. We remark that while the analytic analyses suggest that the CSL is unlikely to emerge in the triangular model built out of weakly coupled chains with chiral interaction Gorohovsky et al. (2015), DMRG results indicate that the CSL can emerge in the strong coupling regime.

By measuring the spin triplet gap and entanglement spectrum on the YC8 cylinder, we find a transition from the $J_{1}-J_{2}$ SL to the CSL at small chiral coupling. We also compute the spin triplet gap on cylinder geometry. While the gap above the overall ground state (in the odd sector) is robust, the one in the even sector seems to be small in both spin liquid phases. In the CSL phase, the small gap suggests that the even sector may be not well developed on our studied system sizes. To find a robust CSL in finite-size calculation, other interactions may be needed besides the chiral interaction. In the $J_{1}-J_{2}$ SL, the vanishing gap in the even sector also indicates that for a gapped spin liquid scenario the even sector is not well established. On the other hand, in the gapless spin liquid scenario Iqbal et al. (2016), one finds it inconsistent with the odd sector exhibiting a large gap based on DMRG calculations ($\Delta_{\rm T}\simeq 0.35J_{1}$ on the YC8 cylinder and the size scaling seems to suggest a large gap in the large-size limit Zhu and White (2015); Hu et al. (2015b)). However, the spin gap measured in DMRG calculations may also have large finite-size effects, which requires further investigations. We hope that future sutdies will be able to resolve between these scenarios more clearly.

This research is supported by the state of Florida (S.S.G.), National Science Foundation Grants DMR-1157490 (S.S.G. and K.Y.), DMR-1442366 (K.Y.), and DMR-1408560 (D.N.S.). Work at Los Alamos was supported by U.S. DOE National Nuclear Security Administration through Los Alamos National Laboratory LDRD Program (W.Z. and J.-X.Z.), and in part supported by the Center for Integrated Nanotechnologies, a U.S. DOE Basic Energy Sciences user facility.

Note Added.— While completing this work, we became aware of a related paper Saadatmand and McCulloch (2017) that also studies the robustness of the $J_{1}-J_{2}$ spin liquid against the chiral interaction in the same $J_{1}-J_{2}-J_{\chi}$ triangular model. We find the overall agreement with Ref. Saadatmand and McCulloch, 2017.