Topological superconductivity in Ni-based transition-metal trichalcogenide superconductors

TL;DR

This paper explores the pairing symmetries and topological properties of Ni-based transition-metal trichalcogenides using a two-orbital model, revealing nearly degenerate I-wave and chiral d-wave states with unique edge phenomena.

Contribution

It introduces a theoretical analysis of pairing symmetries and topological states in Ni-based trichalcogenides, highlighting their potential for studying electron correlation effects.

Findings

I-wave (A2g) and chiral d-wave states are dominant and nearly degenerate.

Both states exhibit nontrivial topological properties.

Chiral d-wave has chiral edge states; I-wave shows zero-energy Andreev bound states.

Abstract

Based on a two-orbital honeycomb lattice model and random phase approximation, we investigate the pairing symmetry of the Ni-based transition-metal trichalcogenide. We find that an I-wave (A2g) state and a chiral d-wave state are dominant and nearly degenerate for typical electron and hole dopings. These two states carry nontrivial topological properties, which are manifested by the presence of chiral edge states in the d+id-wave state and dispersionless Andreev bound state at zero energy in the I-wave state. Ni-based transition-metal trichalcogenides provide us a new platform to study the exotic phenomena emerged from electron-electron correlation effects.

Figures

Tables

Table 1. Table 1: Topological number ω ​ ( k y ) 𝜔 subscript 𝑘 𝑦 \omega(k_{y}) for I 𝐼 I -wave state with Δ = 0.005 Δ 0.005 \Delta=0.005 eV. Here, k y i subscript superscript 𝑘 𝑖 𝑦 k^{i}_{y} ( i = 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 𝑖 1 2 3 4 5 6 7 8 i=1,2,3,4,5,6,7,8 ) are defined in Fig. 10 .

	$\omega \left(k_y\right)$
	1
	0
	-2
	0
	-1
	0
	-2

Table 2. Table 2: Topological number ω ​ ( k x ) 𝜔 subscript 𝑘 𝑥 \omega(k_{x}) for I 𝐼 I -wave state with Δ = 0.005 Δ 0.005 \Delta=0.005 eV. Here, k x i subscript superscript 𝑘 𝑖 𝑥 k^{i}_{x} ( i = 1 , 2 , 3 , 4 , 5 𝑖 1 2 3 4 5 i=1,2,3,4,5 ) are defined in Fig. 10 .

	$\omega \left(k_x\right)$
	-2
	0
	2
	0

Equations

$${\text{H}}_0=\sum _{\mathbf{k}\alpha \beta }\sum _{l_1{l}_2\sigma }{h}_{l_1{l}_2}^{\alpha \beta }(k)c_{\mathbf{k}\alpha l_1\sigma }^{†}{c}_{\mathbf{k}\beta l_2\sigma }=\sum _{\mathbf{k}}{\psi }_{\mathbf{k}\sigma }^{†}h(\mathbf{k}){\psi }_{\mathbf{k}\sigma },$$

$$H_{int}=$$

$$+J_H\sum _{i,\alpha <\beta ,\sigma {\sigma }^{{}^{\prime }}}{c}_{i\alpha \sigma }^{†}{c}_{i\beta {\sigma }^{{}^{\prime }}}^{†}{c}_{i\alpha {\sigma }^{{}^{\prime }}}{c}_{i\beta \sigma }$$

$$+J^{\prime }\sum _{i,\alpha \ne \beta }{c}_{i\alpha \uparrow }^{†}{c}_{i\alpha \downarrow }^{†}{c}_{i\beta \downarrow }{c}_{i\beta \uparrow },$$

$${\chi }_{l_1{l}_2{l}_3{l}_4}(\mathbf{q},\tau )$$

$${\chi }_{l_1{l}_2{l}_3{l}_4}^0(\mathbf{q},i{\omega }_n)=-\frac{1}{N}\sum _{\mathbf{k}\mu \nu }{a}_{\mu }^{l_4}(\mathbf{k})a_{\mu }^{l_2\ast }(\mathbf{k})a_{\nu }^{l_1}(\mathbf{k}+\mathbf{q})\times$$

$$a_{\nu }^{l_3\ast }(\mathbf{k}+\mathbf{q})\frac{n_F(E_{\mu }(\mathbf{k}))-n_F(E_{\nu }(\mathbf{k}+\mathbf{q}))}{i{\omega }_n+E_{\mu }(\mathbf{k})-E_{\nu }(\mathbf{k}+\mathbf{q})},$$

$${\chi }_s^{RPA}(\mathbf{q})={\chi }^0(\mathbf{q})[1-{\bar{U}}^s{\chi }^0(\mathbf{q}){]}^{-1},$$

$${\chi }_c^{RPA}(\mathbf{q})={\chi }^0(\mathbf{q})[1+{\bar{U}}^c{\chi }^0(\mathbf{q}){]}^{-1},$$

$$\bar{U}^{s/c}=\left(\begin{array}[]{cc}\bar{U}^{s/c}_{A}&0\\ 0&\bar{U}^{s/c}_{B}\end{array}\right),$$

$$\bar{U}^{s}_{A/B,l_{1}l_{2}l_{3}l_{4}}=\left\{\begin{array}[]{rcl}\begin{aligned} &U&{l_{1}=l_{2}=l_{3}=l_{4}},\\ &U^{\prime}&{l_{1}=l_{3}\neq l_{2}=l_{4}},\\ &J&{l_{1}=l_{2}\neq l_{3}=l_{4}},\\ &J^{\prime}&{l_{1}=l_{4}\neq l_{3}=l_{2}},\end{aligned}\end{array}\right.$$

$$\bar{U}^{c}_{A/B,l_{1}l_{2}l_{3}l_{4}}=\left\{\begin{array}[]{rcl}\begin{aligned} &U&{l_{1}=l_{2}=l_{3}=l_{4}},\\ &-U^{\prime}+2J&{l_{1}=l_{3}\neq l_{2}=l_{4}},\\ &2U^{\prime}-J&{l_{1}=l_{2}\neq l_{3}=l_{4}},\\ &J^{\prime}&{l_{1}=l_{4}\neq l_{3}=l_{2}},\end{aligned}\end{array}\right.$$

$${\Gamma }_{ij}(\mathbf{k},{\mathbf{k}}^{\prime })=$$

$$\displaystyle\times\emph{Re}\bigg{[}\Gamma_{l_{1}l_{2}l_{3}l_{4}}(\bm{k},\bm{k}^{\prime},\omega=0)\bigg{]}a^{l_{1}}_{\emph{v}_{j}}(\bm{k}^{\prime})a^{l_{4}}_{\emph{v}_{j}}(-\bm{k}^{\prime}),$$

$${\Gamma }_{l_1{l}_2{l}_3{l}_4}^S(\mathbf{k},{\mathbf{k}}^{\prime },\omega )=$$

$$-$$

$${\Gamma }_{l_1{l}_2{l}_3{l}_4}^T(\mathbf{k},{\mathbf{k}}^{\prime },\omega )=$$

$$-$$

$$\displaystyle\lambda\big{[}\emph{g}(\bm{k})\big{]}=-\frac{\sum_{ij}\oint_{C_{i}}\frac{d\bm{k}_{\|}}{\emph{v}_{\emph{F}}(\bm{k})}\oint_{C_{j}}\frac{d\bm{k}^{\prime}_{\|}}{\emph{v}_{\emph{F}}(\bm{k}^{\prime})}\emph{g}(\bm{k})\Gamma_{ij}(\bm{k},\bm{k}^{\prime})\emph{g}(\bm{k}^{\prime})}{(2\pi)^{2}\sum_{i}\oint_{C_{i}}\frac{d\bm{k}_{\|}}{\emph{v}_{\emph{F}}(\bm{k})}\big{[}\emph{g}(\bm{k})\big{]}^{2}},$$

$${\hat{\Delta }}_n^{\eta }(\mathbf{k})={\Psi }_{\mathbf{k}}^{†}{F}_n^{\eta }(\mathbf{k})[{\Psi }_{-\mathbf{k}}^{†}{]}^T={\Psi }_{\mathbf{k}}^{†}[f_n^{\eta }(\mathbf{k})s_i\otimes {\sigma }_j\otimes {\tau }_l][{\Psi }_{-\mathbf{k}}^{†}{]}^T,$$

$$F^{A_{2g}}(\mathbf{k})$$

$$F^{E_{1g}}(\mathbf{k})$$

$$F^{E_{2g}}(\mathbf{k})$$

$${\text{H}}_0=\sum _{\alpha \beta }\sum _{\mu \nu \sigma }{h}_{\mu \nu }^{\alpha \beta }(k)c_{\alpha \mu \sigma }^{†}{c}_{\beta \nu \sigma },$$

$${\text{h}}_{33}=$$

$${\text{h}}_{44}=$$

$${\text{h}}_{21}^{\ast }=$$

$${\text{h}}_{31}^{\ast }=$$

$$+cos(\frac{\sqrt{3}{k}_x}{2})(2(t_{11}+t_{31})+(t_{31}+3t_{32})cos(k_y))-i(2(t_{11}-t_{31})+(t_{31}+3t_{32})cos(k_y))sin(\frac{\sqrt{3}{k}_x}{2})],$$

$${\text{h}}_{41}^{\ast }=$$

$$-(t_{31}-t_{32})cos(\frac{k_x}{2\sqrt{3}}{)}^{2}sin(k_y)+(t_{31}-t_{32})sin(\frac{k_x}{2\sqrt{3}}{)}^{2}sin(k_y)+i(t_{31}-t_{32})sin(\frac{k_x}{\sqrt{3}})sin(k_y)],$$

$${\text{h}}_{32}^{\ast }=$$

$${\text{h}}_{42}^{\ast }=$$

$$+cos(\frac{\sqrt{3}{k}_x}{2})(2(t_{12}+t_{32})+(3t_{31}+t_{32})cos(k_y))-i(2(t_{12}-t_{32})+(3t_{31}+t_{32})cos(k_y))sin(\frac{\sqrt{3}{k}_x}{2})],$$

$${\text{h}}_{43}^{\ast }=$$

$$t_{11}=-0.036294,t_{12}=-0.050971,$$

$$t_{21}=-0.015141,t_{22}=0.003175,t_{23}=0.012118,$$

$$t_{31}=0.238574,t_{32}=-\mathrm{0.020218.}$$

$$A_1$$

$$E_2$$

$$E_1$$

$$A_1$$

Full text

I Introduction

Two-dimensional (2D) materials, such as graphene and transition-metal dichalcogenides Nov ; Geim , are under extensive contemporary study in condensed matter research. This is because many intriguing quantum phenomena have been realized in them, including spin-valley couplingXu2014 , strong charge-spin correlation Kim , 2D magnetism Kim2 ; Wildes ; Gong ; Huang and even superconductivityLu ; Xi ; Saito ; Costanzo ; Wang . Among them, ternary transition-metal phosphorus trichalcogenide (TMPT) compounds APX3 (A=3d transition metals; X=chalcogens) have attracted enormous attention due to antiferromagnetic (AF) ordering as a hint for significant electronic correlations. For bulk materials, it is experimentally found that they can exhibit diverse AF structures, such as zigzag AF and stripy AFChittari ; Sivadas ; Lee ; Wildes . They can be further exfoliated into few atomic layers Kuo , rendering them extremely suitable for studying AF ordering in the quasi-2D limit. Furthermore, by suppressing AF order with external pressure, superconductivity emerges in iron-based TMPT compounds such as FePSe3, with the highest $T_{c}$ found at about 5.5 K Wang , bearing resemblance to high Tc cuprates and iron based superconductors. All this accumulated evidence strongly suggests that electronic correlations may be of great importance to the family of TMPTs.

The crystal structure of the TMPT family APX3 consists of edge shared AX6 octahedral complexes and P2 dimers. Transition metal atoms are arranged in a hexagonal lattice, as shown in Figs.1(a) and (b). In the octahedral crystal field, five 3d orbitals of transition-metal atoms split into high-energy eg orbitals and low-energy t2g orbitals. For FePX3 with Fe2+ ions (d6), it is an ideal system to study the high-to-low spin-state transition by pressureWang . For the case of NiPX3 with Ni2+ $d^{8}$ filling configuration, t2g bands are fully occupied while eg bands are half filled and dominate the spectral weight near the Fermi level. Therefore, the low energy physics can be described by a two orbital model on the honeycomb lattice, where Dirac cones at K are expected. Ni-based phosphorus trichalcogenides, however, can host additional Dirac cones around the midpoint $\frac{K}{2}$ of $\Gamma K$ near Fermi levelYHGu . The 3d orbital nature renders them ideal candidates for strongly correlated Dirac electron system. According to density functional theory (DFT) calculation and experimental measurements, the ground state of Ni-based phosphorus trichalcogenides has been found to be zigzag AF insulator Flem ; Chittari ; YHGu . Theoretical calculations suggests that charge doping can suppress magnetic order, and superconductivity can eventually be achievedYHGu .

In this article, we investigate the pairing symmetry of Ni-based transition-metal trichalcogenide superconductors near half filling. Based on a two-orbital Hubbard model on the honeycomb lattice and through the use of a random phase approximation (RPA) analysis, we find that an $I$-wave and a chiral $d$-wave superconducting state are the dominant instabilities, and nearly degenerate in terms of pairing propensity for typical electron and hole doping. This is because both instabilities are promoted by the intra Fermi surface nesting (between $\alpha$ FS) and the inter Fermi surface nesting (between $\alpha$ and $\beta$ FS). By further resolving their real-space pairing structure, we find that the $I$-wave ($A_{2g}$) state is mainly attributed to NN and NNN pairing, while the chiral $d$-wave Eg state is broadly extended in real space. Both superconducting orders involve strong inter orbital pairing and exhibit nontrivial topological properties. The chiral $d$-wave state is characterized by a nontrivial Chern number, and as such chiral edge modes. In turn, the nodal $A_{2g}$ state features a non-trivial one dimensional topological invariant, and flat bands at zero energy can appear on the edges. Due to their distinctive physical properties, a variety of experimental measurements can be used to unambiguously distinguish these two pairing states.

The paper is organized as follows. In Sec. II, we present the two-orbital tight-binding model based on eg orbitals (dxz and dyz orbitals) to represent the single-particle description of the Ni-based transition-metal trichalcogenide. Furthermore, the crystal structure and electronic band structure is discussed. In Sec. III, we explain the formalism of the random phase approximation (RPA) approach we employ to predict the pairing symmetries within the scope of multi-orbital Coulomb interactions. In Sec. IV, we calculate the spin susceptibility and pairing symmetry as a function of electron and hole doping starting from half filling. We also analyze the real-space pairing of the obtained pairing states, and calculate the edge states originating from their nontrivial topological properties. Finally, in Sec. V we discuss about the experimental realization of superconductivity conclude that transition-metal trichalcogenide provide a promising platform for topological superconductivity.

II Nickel phosphorous trichalcogenides

The compounds nickel phosphorous trichalcogenides NiPX3 (X=S,Se) crystallizes in a layered hexagonal structure and each layer consists of edge shared MX6 octahedral complexes and P2 dimers. As shown in Fig. 1(a) and (b), the cation Ni is surrounded by six anions S/Se and the anions entity $P_{2}X^{-4}_{6}$ is located at the center of honeycomb lattice. As they can be easily exfoliated to monolayers, we focus on NiPX3 monolayers in the following. The electronic band structure and Density of states for monolayer NiPS3 are displayed in Fig. 1(c). It is evident that the $e_{g}$ and $t_{2g}$ bands is separated by a gap due to the crystal field. The former bands with high energies appear near the Fermi level while the latter locate around $1.4$ eV below the Fermi level. The $e_{g}$ bands ($d_{xz}$, $d_{yz}$ orbitals) are half filled and contribute dominantly to the Fermi surface. Moreover, their bandwidth is quite narrow only about 1.1 eV. The Dirac cone appears at K(K’) is similar to that in graphene but additional Dirac cones appear around $\frac{K}{2}$ ($\frac{K^{\prime}}{2}$), which is protected by the mirror symmetry along $GK$ line. The basic electronic structure can be modelled by a two-orbital tight-binding model on a honeycomb lattice. The corresponding Hamiltonian reads,

[TABLE]

with $\psi^{\dagger}_{\bm{k}\sigma}=(c^{{\dagger}}_{\bm{k}A1\sigma},c^{{\dagger}}_{\bm{k}A2\sigma},c^{{\dagger}}_{\bm{k}B1\sigma},c^{{\dagger}}_{\bm{k}B2\sigma})$. Here $\alpha$,$\beta$ are the sublattice indices (A,B) and $l_{i}$ ($l_{i}=1,2$) is the orbital index ($d_{xz/yz}$). $c^{{\dagger}}_{\alpha\mu\sigma}$ creates a spin $\sigma$ electron with momentum $\bm{k}$ in $\mu$ orbital on $\alpha$ sublattice. The matrix elements of $h^{\alpha\beta}_{\mu\nu}(k)$ are provided in the Appendix. According to our calculationsYHGu , we interestingly find the third nearest neighbor (TNN) hopping is much larger than the NN and second nearest neighbor (SNN) hopping parameters. From the calculation of GGA+$U$YHGu , the NiPS3 favors the zigzag antiferromagnetic state, where the magnetic moments of Ni cations connected by the TNN bonds is antiparallel. We show the orbital resolved band dispersion from the tight-binding model in Fig. 1(d). Orbital mixture can be found along $\Gamma K$ and $KM$ but not $\Gamma M$, due to the presence of two-fold rotational symmetry along $\Gamma M$. The strongest orbital mixture near the Fermi level occurs around the Dirac points ($K$ and $\frac{K}{2}$).

III random phase approximation

In this section, we explain the formalism of the multiorbital RPA approachBickers ; Kemper ; Graser ; Xxwu1 ; Xxwu2 ; Li ; Sante . The adopted onsite Coulomb interaction terms are,

[TABLE]

where $n_{i,\alpha}=n_{i,\alpha,\uparrow}+n_{i,\alpha,\downarrow}$. $U$, $U^{\prime}$, $J$ and $J^{\prime}$ represent the intra- and inter-orbital repulsion, the Hund’s rule and pair-hopping terms. In the following calculations, we use Kanamori relations $U=U^{\prime}+2J$ and $J=J^{\prime}$ as requried by the lattice symmetry.

The multi-orbital susceptibility is defined as,

[TABLE]

In momentum-frequency space, the multi-orbital bare susceptibility is given by

[TABLE]

where $\mu$ and $\nu$ are the band indices, $n_{F}$ is the usual Fermi distribution, $l_{i}$ $(i=1,2,3,4)$ are the orbital indices, $a^{l_{i}}_{\mu}(k)$ is the $l_{i}$ orbital component of the eigenvector for band $\mu$ resulting from the diagonalization of the tight-binding Hamiltonian $H_{0}$ and $E_{\mu}(\bm{k})$ is the corresponding eigenvalue. With interactions, the RPA spin and charge susceptibilities are given by

[TABLE]

where $\bar{U}^{s}$ ($\bar{U}^{c}$) is the spin (charge) interaction matrix

[TABLE]

[TABLE]

[TABLE]

Within RPA approximation, the effective Cooper scattering interaction is,

[TABLE]

where the momenta $\bm{k}$ and $\bm{k}^{\prime}$ is restricted to different FSs with $\bm{k}\in C_{i}$ and $\bm{k}^{\prime}\in C_{j}$. The orbital vertex function $\Gamma_{l_{1}l_{2}l_{3}l_{4}}$ in spin singlet and triplet channels Kemper ; Xxwu2 ; Sante are

[TABLE]

where $\chi^{RPA}_{s}$ and $\chi^{RPA}_{c}$ are the RPA spin and charge susceptibility, respectively. The pairing strength functional for a specific pairing state is given by,

[TABLE]

where $v_{F}(\bm{k})=|\nabla_{k}E_{i}(\bm{k})|$ is the Fermi velocity on a given Fermi surface sheet $C_{i}$. The pairing vertex function in spin singlet and triplet channels are symmetric and antisymmetric parts of the interaction, that is, $\Gamma^{S/T}_{ij}(\bm{k},\bm{k}^{\prime})=\frac{1}{2}[\Gamma_{ij}(\bm{k},\bm{k}^{\prime})\pm\Gamma_{ij}(\bm{k},-\bm{k}^{\prime})]$.

IV results and analysis

According to Ref.YHGu, , superconductivity may emerge after the antiferromagnetic states is suppressed by electron or hole doping. Here, based on weak coupling appoach, we investigate the pairing symmetries for the doped system. In the following, we mainly focus on three typical doping levels $\delta=-0.2,0.2,0.4$ relative to the half filled bands. $\delta>0$ ($\delta<0$) represents the electron (hole) doping. Fig. 2 shows the Fermi surfaces for half-filled case and the above doped cases. At half filling, Dirac cones at K and $K^{\prime}$ contributes two small electron pockets while the Dirac cones around $\frac{K}{2}$ and $\frac{K^{\prime}}{2}$ contribute six small hole pockets, whose area is equal to that of electron pockets as required by charge neutrality. With hole doping, the Fermi surfaces around $K$ shrink to the eletron-hole Lifshitz transition point and then become hole pockets while $\alpha$ Fermi surfaces enlarge and become elliptical, with the long axis along $\Gamma K$. Two of $\alpha$ Fermi surfaces are mainly attributed to $d_{xz}$ orbital while the others are mainly attributed to $d_{yz}$ orbitals. The parts of $\alpha$ Fermi surfaces close to $\Gamma$ point exhibit strong orbital mixture while those away from $\Gamma$ point show weaker orbital mixture. For 0.2 electron doped case, as shown in Fig.2(c), all Fermi surfaces are electron type and the orbital distribution on the Fermi surfaces are similar. With further electron doping, $\delta=0.4$, all $\alpha$ Fermi surfaces enlarge and show a stronger orbital mixture.

In order to study the pairing symmetries, we first show the bare susceptibility $\chi_{0}$ along high-symmetry lines in Fig. 3(a) for the three typical doping levels. For $\delta=0.2$ (red line), there is a prominent plateau around M and a broad peak around $\frac{K}{2}$. The former one is attributed to inter pocket nesting between $\alpha$ and $\beta$ and intra pocket nesting between next NN of $\alpha$ Fermi surfaces. The corresponding nesting vectors $Q_{1}$ and $Q^{\prime}_{1}$ are displayed in Fig.2(c). While the latter peak is mainly contributed by the intra pocket nesting between the NN of $\alpha$ Fermi surfaces ($Q_{2}$ and $Q^{\prime}_{2}$ ), as shown in Fig.2(c). With further increasing electron doping, the slight increase near M point results a double-peak structure and the broad peak enhances with its center shifting toward K. In addition, a slight enhancement appears near $\Gamma$ point. The increase of the ratio $|\chi(K/2)|/|\chi(M)|$ is ascribed to the enlargement of $\alpha$ and $\beta$ pockets. For the 0.2 hole doped case, the basic features are similar to those of electron doped case except that the double peaks merge into a single broad peak around M. We further calculate the RPA spin susceptibility with $U=0.4$ eV, $J/U=0.2$ and the obtained results are displayed in Fig. 3(b), (c) and (d). There are great enhancements for the peak structures mentioned above. All peaks in the susceptibility are far away from $\Gamma$ point which implies the intrinsic antiferromagnetic fluctuations in the system, which is consistent the AF ordering in DFT calculations.

The pairing states can be classified according to the irreducible representations of $D_{3d}$ point group for NiPS3. First we discuss about the case with electron doping. We consider a fixed $U=0.4$ and Fig.4(a) and (c) show the pairing strength eigenvalues for the leading eigenvalues in singlet and triplet channels as a function of $J/U$ for $\delta=0.2$ and $0.4$. In both cases, the pairing strengths of $A_{2g}$ and $E_{g}$ pairing states are close but much stronger than those of triplet parings, which is consistent with the intrinsic antiferromagnetic fluctuations from the susceptibility analysis. For $\delta=0.2$, $A_{2g}$ is slightly favored than $E_{g}$ state for $J/U<0.2$. However, for $\delta=0.4$, the dominant pairing state is $E_{g}$ state for $J/U<0.08$ and $A_{2g}$ state will win for $J/U>0.08$. We further plot the pairing strengths as a function of $U$ with a fixed $J/U=0.2$ in Fig.4(b) and (d). The pairing strengths increase rapidly with increasing $U$ and the dominant pairing states are still $A_{2g}$ and $E_{g}$.

The gap functions for the dominant pairing states are shown in Fig.5 for $\delta=0.2$. A2g state is invariant under $C_{3z}$ and $S_{6z}$ operations and changes sign under $M_{yz}$ and $C_{2x}$ operations. The corresponding pairing has nodes along $k_{x}=0$ , $k_{y}=0$, $k_{x}=\pm\sqrt{3}k_{y}$, $k_{x}=\pm\frac{1}{\sqrt{3}}k_{y}$ lines and can be described by a function $xy(x^{2}-3y^{2})(3x^{2}-y^{2})$, which is an $I$-wave pairing state. The gap function on $\alpha$ FS is much large than that on $\beta$ FS. For the two-fold degenerate $E_{g}$ $d_{xy}$ and $d_{x^{2}-y^{2}}$ states are displayed in Fig.5(b) and (c). For the $d_{xy}$ state, the gap function on each $\alpha$ pocket has a sign change and the resulting nodes are not along high-symmetry lines, different from that of $A_{2g}$ state. Moreover, in contrast to $A_{2g}$ state, the gap function on $\beta$ FS is comparable to that of $\alpha$ FS. For the $d_{x^{2}-y^{2}}$ state, two $\alpha$ pockets on $y$ axis have a small gap size compared with others. Both $A_{2g}$ and $E_{g}$ states satisfy the condition that the superconducting order connected by the nesting vectors ($Q_{1}$, $Q^{\prime}_{1}$, $Q_{2}$ and $Q^{\prime}_{2}$) has a sign change. In $A_{2g}$ state, intra pocket scattering for $\alpha$ FS plays the dominant role. While, in $E_{g}$ state, both intra pocket and inter pocket scattering are important. Furthermore, the two-fold degenerate $E_{g}$ states tend to form the $d+id$ state in order to maximize the condensation energy, and the gap function is shown in Fig.5(d). This is a recurrent theme for other instances of predicted chiral $d$-wave superconductivity for hexagonal lattices Graphene3RG ; GrapheneFRG ; GrapheneFRG2 ; CobaltatesFRG ; PhysRevB.89.020509 .

For $\delta=0.4$, the dominant gap functions are shown in Fig.6. There is a relative enhancement for gap functions on $\beta$ FS in $A_{2g}$ due to the enhancement of $Q^{\prime}_{1}$ nesting. The inter pocket $Q^{\prime}_{1}$ nesting enhancement also promote $E_{g}$ state, which explains the stronger pairing strength for $E_{g}$ for $J/U<0.08$. The corresponding gap functions are shown in Fig.6(b), (c) and (d).

Now we discuss the hole doped case with $\delta=-0.2$. Similar to the electron doped case, we find that $A_{2g}$ and $E_{g}$ pairing states are dominant and their pairing strengths are close, as shown in Fig. 7. The A2g state is the leading pairing for small $J/U$($J/U<0.15$) and the Eg state becomes dominant pairing for large $J/U$ ($J/U>0.15$). The corresponding gap functions of A2g, $d_{x^{2}-y^{2}}$ and $d_{xy}$ are presented in Fig. 8. The $A_{2g}$ state is very similar to the case with $\delta=0.2$, where gap functions almost vanish on $\beta$ FS. For $E_{g}$ state, the noticeable feature is the great enhancement of gap functions on $\beta$ compared with electron doped cases. Therefore $A_{2g}$ and $E_{g}$ pairing state are quite robust in doped monolayer NiPX3.

To further understand the obtained pairing state, we analyze the real-space structure of the obtained pairing states. In multiorbital system, the pairing in orbital space can also transform nontrivially under point group operations. For this two-band honeycomb lattice model, the classification of pairing states in real space have been given in Ref.WuTBG . Generally the pairing state with $\eta$ form factor on $n$-th NN bond can be written as,

[TABLE]

where $\Psi^{\dagger}_{\bm{k}}=(\psi^{\dagger}_{\bm{k}\uparrow},\psi^{\dagger}_{\bm{k}\downarrow})$ and $i,j,k=1,2,3$, $f^{\eta}_{n}(\bm{k})$ is the corresponding lattice harmonics in $\bm{k}$ space and $\bm{s}$, $\bm{\sigma}$, $\bm{\tau}$ are Pauli matrices defined in the spin, sublattice and orbital space. For the $A_{2g}$ state, the pairing matrix can be written as,

[TABLE]

Here $f^{\eta}_{nI/R}$ represents the real/imaginary part of $\eta$ form factor for $n$-th NN bond. Because there is no $A_{2g}$ lattice harmonic, $A_{2g}$ pairing state must involve two-dimensional irreducible representations of lattice harmonics and orbital pairing. For the $E_{g}$ state, the matrices for two pairing states can be written as,

[TABLE]

The lattice harmonics are listed in the Appendix. We fit the obtained gap function from RPA calculations with the above form factors up to the third NN bonds. The $I$-wave state is dominantly contributed by pairing on the NN and NNN bonds. In contrast, $E_{g}$ state in real space is extended and pairing on these bonds as well as onsite pairing will contribute.

Both of the obtained $d$-wave and $I$-wave pairing states can carry topological characters. The chiral $d$-wave state, breaking the time reversal symmetry, belongs to the class C and is characterized by the topological invariant $2Z$. For the nodal superconducting state can be characterized by a nonzero one-dimensional topological invariantSato2011 , which can result dispersionless Andreev bound states (ABS) on surfaces or edges. For the chiral $d$-wave state, we plot the armchair and zigzag edges states with only including onsite $d$-wave state in Fig.9 (a) and (b). As there are four Fermi surfaces in half Brillouin zone, the total Chern number is $4\times 2=8$ per spin channel. In the both case, there are eight edge states, which is consistent with Chern number. With furthering including Rashba spin-orbit coupling and Zeeman coupling, odd Chern number can be achieved, which results edge Majorana modesLu2018 . For the $I$-wave state, Fig. 9(c) and (d) show the zigzag and armchair edge states. There are flat ABS connecting the projections of nodal points and their appearance is characterized by the 1D topological invariant, which is provided in the Appendix. One prominent feature of $I$-wave state is that flat ABS at zero energy can appear at all lattice termination edges, in sharp contrast to the $d$-wave pairing in cuprates. This will induce a large peak at zero energy in the density of states, which can be detected in STM measurements.

V Discussion and Conclusion

In order to achieve superconductivity in NiPX3(X=S,Se), charge doping is required to suppress the antiferromagnetic order. For 2D layered materials, electron or hole carriers can be introduced by gating technology, which has been used to realize superconductivity for semiconducting thin filmsYe2012 . Similar to graphene and monolayer FeSe, carries doping can be introduced by the adsorption of cations on the NiPX3 films or cations intercalation in bulk NiPX3. The charge doping in experiments can be easily realized in NiPX3.

Once superconductivity is realized in Ni-based transition-metal trichalcogenides, $A_{2g}$ ($I$-wave) and $E_{g}$ ($d+id$) pairing states are dominant according to our calculations. $I$-wave state is characterized by its line nodes along $\Gamma-K$. While the chiral $d$-wave state, which has a full gap and breaks the time reversal symmetry, is characterized by a nontrivial topological invariant. For $I$-wave state, the nodal gap structure can be directly detected by the high resolution ARPES. Physical properties related to low energy excitations, such as low temperature specific heat, spin relaxation and penetration depth, should be very similar to the $d$-wave state in cuprates. The time reversal symmetry breaking for the fully gapped $d+id$ state can be verified by muon-spin-rotation/relaxation measurements. Furthermore, at the edges there is a large peak at zero energy in DOS for $I$-wave state, which makes it distinct from the chiral $d$-wave state. Therefore, experimental measurements in the superconducting states can be used to distinguish these two pairing states.

In summary, we have investigated the pairing symmetry for the Ni-based transition-metal trichalcogenide proposed recently. By performing RPA calculations, we find that $I$-wave ($A_{2g}$) state and chiral $d$-wave ($E_{g}$) state are dominant and nearly degenerate for typical electron and hole doping. Both of them are promoted by the intra Fermi surface nesting (between $\alpha$ FS) and the inter Fermi surface nesting (between $\alpha$ and $\beta$ FS). Their nontrivial topological properties are manifested by the presence of edge states. Ni-based transition-metal trichalcogenide provides us a new platform to study the exotic phenomena emerged from the electron-electron correlation.

VI Acknowledgements

This work is supported by the Ministry of Science and Technology of China 973 program (Grant No. 2015CB921300, No. 2017YFA0303100, No.2017YFA0302900), National Science Foundation of China (Grant No. NSFC-11334012), and the Strategic Priority Research Program of CAS (Grant No. XDB07000000). The work in Würzburg is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)-Project-ID 258499086-SFB 1170 and by the Würzburg-Dresden Cluster of Excellence on Complexity and Topology in Quantum Matter – ct.qmat (EXC 2147, Project-ID 39085490).

Appendix A The tight-binding model of NiPS3

The four band tight-binding model is given by

[TABLE]

where ($\alpha$,$\beta$) are the sublattice indices (a,b) and ($\mu$,$\nu$) are the orbital indices ($d_{xz}$, $d_{yz}$). $c^{{\dagger}}_{\alpha\mu\sigma}$ creates a spin $\sigma$ electron in $\mu$ orbital on $\alpha$ sublattice. The matrix elements of $h^{\alpha\beta}_{\mu\nu}(k)$ based on the basis $c^{{\dagger}}_{\alpha\mu}=(c^{{\dagger}}_{A,xz},c^{{\dagger}}_{A,yz},c^{{\dagger}}_{B,xz},c^{{\dagger}}_{B,yz})$ are listed in the follow:

[TABLE]

Here, (1,2,3,4) are orbital indices of ($d^{A}_{xz},d^{A}_{yz},d^{B}_{xz},d^{B}_{yz}$) and $\lambda$ is the chemical potential.

The hopping parameters in the model are

[TABLE]

Appendix B lattice harmonics in honeycomb lattice

The form factor for onsite pairing is $f^{A_{1}}_{0}=1$. The form factors for nearest-neighbor (NN) bond is,

[TABLE]

The form factors for next NN bond is,

[TABLE]

The form factors for third NN bond is,

[TABLE]

Appendix C topological number in $I$-wave state

For time-reversal-invariant superconductor, the topological criterionSato2011 about the zero energy ABS can be written as the following simple summation:

[TABLE]

where the summation is taken for $k_{x}$ satisfying $\varepsilon(k)=0$ with a fixed $k_{y}$. According to the above formula, we obtain $\omega(k_{y})$ and $\omega(k_{x})$ about ABS of $I$-wave state in Tables 1 and 2. These topological numbers $\omega(k_{y})$ and $\omega(k_{x})$ are consistent with the zigzag and armchair edge of $I$-wave superconducting state respectively.