Effect of Zeeman coupling on the Majorana vortex modes in iron-based topological superconductors

TL;DR

This paper investigates how Zeeman coupling influences Majorana vortex modes in iron-based topological superconductors, explaining the observed differences in vortex zero-energy states and their localization properties.

Contribution

It demonstrates the effects of Zeeman coupling on different pairing symmetries and explains the experimental dichotomy of vortex states in FeTeSe superconductors.

Findings

Weak Zeeman fields suppress intra-orbital pairing, localizing vortex states.

Orbital-triplet pairing remains stable but causes delocalized Majorana modes.

Zero-bias peaks disappear with increasing magnetic field due to Zeeman effects.

Abstract

In the superconducting regime of FeTe$_{(1-x)}$Se$_x$, there exist two types of vortices which are distinct by the presence or absence of zero energy states in their core. To understand their origin, we examine the interplay of Zeeman coupling and superconducting pairings in three-dimensional metals with band inversion. Weak Zeeman fields are found to suppress the intra-orbital spin-singlet pairing, known to localize the states at the ends of the vortices on the surface. On the other hand, an orbital-triplet pairing is shown to be stable against Zeeman interactions, but leads to delocalized zero-energy Majorana modes which extend through the vortex. In contrast, the finite-energy vortex modes remain localized at the vortex ends even when the pairing is of orbital-triplet form. Phenomenologically, this manifests as an observed disappearance of zero-bias peaks within the cores of topological vortices upon increase of the applied magnetic field. The presence of magnetic impurities in FeTe$_{(1-x)}$Se$_x$, which are attracted to the vortices, would lead to such Zeeman-induced delocalization of Majorana modes in a fraction of vortices that capture a large enough number of magnetic impurities. Our results provide an explanation to the dichotomy between topological and non-topological vortices recently observed in FeTe$_{(1-x)}$Se$_x$.

Equations

$$\mathcal{H}=\hslash v_F{\tau }_x\mathbf{\sigma }\cdot \mathbf{k}+m_k{\tau }_z,$$

$$\mathcal{H}=\int d^3\mathbf{k}{f}_{\mathbf{k}}^{†}\left[\left(E_{\text{k}}-\mu \right){\nu }_0+{\mathbf{d}}_{\mathbf{k}}\cdot \mathbf{\nu }\right]f_{\mathbf{k}}.$$

$$\frac{{\Delta }_{\alpha }}{2}\int d^3\mathbf{k}{f}_{\mathbf{k}}^{†}{e}^{-i{\varphi }_{\text{k}}}{\nu }_{\alpha }{f}_{-\mathbf{k}}^{†T},$$

$$\mathrm{det}\left|\begin{array}[]{cc}U\bar{\chi}-1&U\chi_{2}\\ V\chi_{2}&V\chi_{1}-1\end{array}\right|=1,\qquad V\chi_{0}=1.$$

$$H_l^{\nu }={\Pi }_z\frac{{\Delta }_0{\lambda }_{\nu }\left(k\right)}{{\xi }_0}+{\Pi }_y\frac{{\Delta }_0}{{\xi }_0}\left(i{\partial }_k\right)+{\Pi }_x{E}_{l,0}^{\nu }(k).$$

$${\Psi }_V^{\nu }\left(l,r\right)\approx {\chi }_{l,k_F}^{\nu }(r){\left(1,1\right)}_{\Pi }^T$$

$$H_V^l$$

$$+\left(\frac{E_l^{+}-E_l^{-}}{2}-\widetilde{ϵ}{\partial }_z^2\right){\eta }_z+{\tilde{v}}_z\left({\partial }_z\right){\eta }_x.$$

$${\mathcal{H}}_0=\int d^3\mathbf{k}/{\left(2\pi \right)}^3{\psi }_{\mathbf{k}}^{†}\left(H_{\mathbf{k}}-\mu \right){\psi }_{\mathbf{k}},$$

$$H_{\mathbf{k}}=\mathbf{\alpha }\cdot \mathbf{k}+m_{\mathbf{k}}\beta$$

$${\psi }_{\mathbf{k}}=\sum _{a=1}^4{\phi }_{\mathbf{k}}^a{f}_{\mathbf{k},a},$$

$${\phi }_{\mathbf{k}}^a=e^{i{\varphi }_{\mathbf{k}}/2}{e}^{-i{\sigma }_z{\varphi }_{\mathbf{k}}/2}{e}^{-i{\sigma }_y{\theta }_{\mathbf{k}}/2}{e}^{-i{\tau }_y{\sigma }_z{\alpha }_{\mathbf{k}}/2}{\hat{e}}_a,$$

$$\tan {\varphi }_{\mathbf{k}}$$

$$\cos {\theta }_{\mathbf{k}}$$

$$\cos {\alpha }_{\mathbf{k}}$$

$${\phi }_{\mathbf{k}}^1$$

$$\displaystyle=\left(\begin{array}[]{c}\cos\frac{\theta_{\mathbf{k}}}{2}\cos\frac{\alpha_{\mathbf{k}}}{2}\\ e^{i\phi_{\mathbf{k}}}\sin\frac{\theta_{\mathbf{k}}}{2}\cos\frac{\alpha_{\mathbf{k}}}{2}\\ \cos\frac{\theta_{\mathbf{k}}}{2}\sin\frac{\alpha_{\mathbf{k}}}{2}\\ e^{i\phi_{\mathbf{k}}}\sin\frac{\theta_{\mathbf{k}}}{2}\sin\frac{\alpha_{\mathbf{k}}}{2}\end{array}\right)$$

$${\phi }_{\mathbf{k}}^2$$

$$\displaystyle=\left(\begin{array}[]{c}-\sin\frac{\theta_{\mathbf{k}}}{2}\cos\frac{\alpha_{\mathbf{k}}}{2}\\ e^{i\phi_{\mathbf{k}}}\cos\frac{\theta_{\mathbf{k}}}{2}\cos\frac{\alpha_{\mathbf{k}}}{2}\\ \sin\frac{\theta_{\mathbf{k}}}{2}\sin\frac{\alpha_{\mathbf{k}}}{2}\\ -e^{i\phi_{\mathbf{k}}}\cos\frac{\theta_{\mathbf{k}}}{2}\sin\frac{\alpha_{\mathbf{k}}}{2}\end{array}\right).$$

$${\mathcal{H}}_0\approx \sum _{a=1,2}{\int }_{\mathbf{k}}\left(E_{\mathbf{k}}-\mu \right)f_{\mathbf{k},a}^{†}{f}_{\mathbf{k},a}.$$

$${\mathcal{H}}_Z\approx \int d^3\mathbf{k}{f}_{\mathbf{k}}^{†}\left[{\mathbf{b}}_{\mathbf{k}}\cdot \mathbf{\nu }\right]f_{\mathbf{k}},$$

$${\left[{\mathcal{H}}_0+{\mathcal{H}}_Z\right]}_{k_z=0}\approx \int d^2\mathbf{k}{f}_{\mathbf{k}}^{†}\left[\left(E_{\mathbf{k}}-\mu \right){\nu }_0-b_{\mathbf{k}}^x{\nu }_x\right]f_{\mathbf{k}}.$$

$${\left[{\mathcal{H}}_0+{\mathcal{H}}_Z\right]}_{k_z=0}\approx \int d^2\mathbf{k}{d}_{\mathbf{k}}^{†}\left[\left(E_{\mathbf{k}}-\mu \right){\nu }_0-b_{\mathbf{k}}^x{\nu }_z\right]d_{\mathbf{k}}.$$

$${\mathcal{H}}_{SC,1}$$

$$=\frac{1}{2}\int d^3\mathbf{r}{\psi }^{†}{\Delta }_{1}i{\sigma }_y{\psi }^{†T}+H.c..$$

$${\mathcal{H}}_{SC,1}$$

$$\approx \frac{{\Delta }_1}{2}\sum _{a,b=1,2}\int d^2\mathbf{k}{f}_{\mathbf{k},a}^{†}\left({\phi }_{\mathbf{k}}^{a†}i{\sigma }_y{\phi }_{-\mathbf{k}}^{b\ast }\right)f_{-\mathbf{k},b}^{†}+H.c..$$

$${\mathcal{H}}_{SC,1}=-\frac{{\Delta }_1}{2}\int d^2\mathbf{k}\left(f_{\mathbf{k}}^{†}{e}^{-i{\varphi }_{\mathbf{k}}}{\nu }_z{f}_{-\mathbf{k}}^{†T}\right)+H.c..$$

$${\left[{\mathcal{H}}_0+{\mathcal{H}}_Z\right]}_{k_z=0}$$

$${\left[{\mathcal{H}}_{SC,1}\right]}_{k_z=0}$$

$${\mathcal{H}}_{SC,0}$$

$$\approx \frac{{\Delta }_0}{2}\sum _{a,b=1,2}\int d^2\mathbf{k}{f}_{\mathbf{k},a}^{†}\left({\phi }_{\mathbf{k}}^{a†}i{\tau }_x{\sigma }_y{\phi }_{-\mathbf{k}}^{b\ast }\right)f_{-\mathbf{k},b}^{†}+H.c..$$

$${\mathcal{H}}_{SC,0}=-\frac{{\Delta }_0}{2}\int d^2\mathbf{k}{e}^{-i{\varphi }_{\mathbf{k}}}\sin {\alpha }_{\mathbf{k}}\left(f_{\mathbf{k}}^{†}{\nu }_0{f}_{-\mathbf{k}}^{†T}\right)+H.c..$$

$${\left[{\mathcal{H}}_0+{\mathcal{H}}_Z\right]}_{k_z=0}$$

$${\left[{\mathcal{H}}_{SC,2}\right]}_{k_z=0}$$

Full text

Introduction: To date, one of the major impediments in the search for Majorana fermions (MFs) is that a requisite topological superconductivity, either intrinsic revtsc; Sato_2017 or induced in a host material via a proximity coupling to a standard $s$-wave superconductor PhysRevLett.100.096407; PhysRevLett.105.077001; Mourik25052012; Nadj-Perge602. Of the available materials that possess topology, superconductivity and magnetism, iron-based superconductors are of recent interest revpnic; RevModPhys.87.855; PhysRevLett.121.076401; PhysRevB.100.014512; PhysRevB.88.134510; PhysRevB.90.024517; PhysRevB.98.144517; PhysRevLett.122.207001; dass. In particular, the iron-based superconductor FeTe 0.55Se0.45 (FTS) has recently been shown to have strong spin-orbit interactions and band inversion that result in a helical, topologically-protected, Dirac cone on the surface Hanaguri; PhysRevB.92.115119; Zhang182; Gangxu2016. The phenomenology of vortices, proliferated in the presence of magnetic fields, is also noteworthy in FTS Wang333; tamagai; Lingyuan; hongjungao2019. The low charge density in the superconducting phase of this system is experimentally advantageous as it results in large Caroli-de Gennes-Matricon (CDM) vortex mode gaps CAROLI1964307 which facilitates the spectral detection of zero-energy vortex modes via scanning tunneling microscopy (STM). Intriguingly, vortices in FTS show two distinct types of behavior: topological, carrying zero energy states consistent with the presence of MF, and trivial vortices that lack the zero energy state but carry finite energy CDMs.

While the evidence of MFs in FTS has been observed, a comprehensive understanding of the salient physics of the vortex composition is lacking. More precisely, the energy spectra of the vortices follow different hierarchies relative to the CDM vortex gap, $\delta=\frac{\Delta^{2}}{\mu}$, where $\Delta$ is the bulk superconducting gap and $\mu$ is the chemical potential. The trivial vortex energy spectrum scales as $(n+1/2)\delta$, (with $n\in\mathbb{Z}$) which does not include the zero mode, whereas the topological vortices follow $n\delta$. Additionally, the percentage of vortices with zero-energy modes decreases as the perpendicular magnetic field increases, despite the fact that the inter-vortex distances are generically larger than the superconducting coherence length Wang333; tamagai; chinkai. It was also shown that the distribution of vortices with and without zero mode has no correlation with the charge disorder on the surface of the material tamagai. These features, on aggregate, suggest that the properties that distinguish the two classes of vortices stem from the bulk properties of individual vortex rather than those of the surface states. In particular the effects of magnetic field, beyond generation of vortices, might crucially affect the properties of superconducting state and the vortices.

Motivated by experimental observations, we examine the effects of Zeeman coupling on the vortex modes in FTS. As the topological properties of FTS are driven by band inversion, we eschew more complex band models and utilize a simple model of a doped 3D time-reversal symmetric (TRS) topological insulator (TI) which has been used as an appropriate toy model to investigate the properties of vortices in FTS mvx; Zhang2019. An essential property of the electronic band structure in TRS TI is the presence of degenerate Fermi surfaces. Due to the strong spin-orbit coupling, Zeeman field splits the the degenerate Fermi surface into two helical Fermi surfaces with opposite helicity (see supplementary material). In this letter, we show that the split Fermi surfaces prefer an orbital-triplet superconducting pairing which delocalizes the MF modes at the ends of the vortices on the surface. To make direct connection with the dichotomy of vortices in FTS we note that the Zeeman field may result from the magnetic impurities along the vortex core PhysRevX.9.011033. Interestingly, such magnetic Fe impurities are known to exists in FTS PhysRevLett.108.107002; Yin2015 and they are attracted to the vortices PhysRevX.9.011033. In addition, increase of magnetic field naturally leads to enhancement of Zeeman coupling which further destabilize the topological vortices, as is experimentally observed. We should also note that neutron scattering measurements have shown the evidence of ferromagnetic clusters of Fe atoms in FTS PhysRevLett.108.107002. Observation of the clusters of vortices, with and without zero energy vortex modes, would further support our theory.

Model Hamiltonian: The two-orbital model of a 3D TRS TI is represented by the tight-binding Hamiltonian $\mathcal{H}_{\mathbf{k}}=\sum_{\mathbf{k}}\psi_{\mathbf{k}}^{\dagger}\left[\tau_{x}\textbf{d}_{\textbf{k}}.\boldsymbol{\sigma}+m_{\textbf{k}}\tau_{z}-\mu\right]\psi_{\mathbf{k}}$ where Pauli matrices $\sigma_{i}$ and $\tau_{i}$ act on spin and orbital space respectively, $d_{k_{i}}=2t\sin(k_{i})$, $m_{\textbf{k}}=M+m_{0}\sum_{i}\cos(k_{i})$ and $M$, $m_{0}$, $t$ are parameters of the model and $\mu$ is the chemical potential. By varying the parameters, this model Hamiltonian could represent both strong and week TRS TIs PhysRevB.81.045120. The tight-binding model is used for the numerical calculations while our analytical results are based on the low-energy effective model

[TABLE]

where $m_{k}=m+\epsilon k^{2}$ is the effective mass term and k is the momentum relative to the center of the Brillouin zone. The trivial (topological) insulator corresponds to $m\epsilon>0$ ($<0$). Without loss of generality, we take $\hbar v_{F}=1$.

With the Hamiltonian defined, we begin our analysis in the metallic phase, when $\mu>|m|$. The model displays two degenerate Fermi surfaces that split by a Zeeman field $\Delta_{Z}$. Since the bulk band structure gap is large compared with superconducting gap, we use the effective Hamiltonian resulting from projecting the Hamiltonian in Eq. (1) into the states at the two Fermi surfaces:

[TABLE]

Here $\nu_{i}$ are the identity or Pauli matrices acting on the the space of two Fermi surfaces. The vector \mathbf{d}_{\mathbf{k}}=\frac{\Delta_{Z}}{2}\left(\begin{array}[]{ccc}-\frac{m_{\mathbf{k}}}{E_{\mathbf{k}}}\frac{k_{x}^{2}+k_{y}^{2}}{\left|\mathbf{k}\right|},&0,&\frac{k_{z}}{k}\end{array}\right) presents the Zeeman field and $f_{\textbf{k}}$ is the Fermion fields $\psi_{\textbf{k}}$ projected onto the Fermi surfaces (see supplementary materials). The two fermi spin-split fermi surfaces are identified by diagonalizing the projected Hamitonian (2) in the $\nu$ space.

Previous analysis PhysRevLett.105.097001 identified two types of superconductivity in doped TIs which are energetically favorable: intra-orbital spin singlet, $\int d^{3}\mathbf{r}\psi^{\dagger}\Delta i\tau_{0}\sigma_{y}\psi^{\dagger T}+H.c.$, and inter-orbital orbital-triplet spin-singlet, $\int d^{2}\mathbf{k}\psi_{\mathbf{k}}^{\dagger}i\tau_{x}\sigma_{y}\psi_{-\mathbf{k}}^{\dagger T}+H.c.$. Henceforth, we will refer to these superconducting pairings as intra-orbital singlet and inter-orbital triplet pairings, respectively.

Upon projection onto the Fermi surfaces the superconducting pairing potentials assume the following form,

[TABLE]

where for the intra-orbital singlet pairing $\alpha=1$ and for inter-orbital triplet pairing $\alpha=0$. Examination of the superconducting pairing term in Eq. (3) in comparison with the kinetic Hamiltonian in Eq. (2) shows that the intra-orbital singlet pairing, $\Delta_{1}$, corresponds to pairing electrons between different Zeeman split fermi surfaces. In contrast, the inter-orbital triplet pairing, $\Delta_{0}$, pairs electrons solely within each of the Zeeman split Fermi surfaces. Thus, the effect of Zeeman coupling is to break the TRS within the model and imbalance the two pairing potentials in the favor of $\Delta_{0}$ which couples the electrons solely within each Zeeman split Fermi surface youngseokkim2016.

To examine the outlined effect of Zeeman coupling on the dominant form of superconducting pairing, we utilize a linear gap equation to determine the critical temperatures of the two superconducting pairings PhysRevLett.105.097001; nakosai2012; hashimoto2016; PhysRevB.90.045130(see supplementary materials). The corresponding $U-V$ model, in which $U$ and $V$ are the intra and inter orbital interaction, leads to the equation :

[TABLE]

Here $\bar{\chi}$, $\chi_{1}$ and $\chi_{2}$ are the superconducting susceptibilities characterizing intra-orbital spin-singlet pairing and $\chi_{0}$ describes the inter-orbital spin triplet pairing. $\bar{\chi}=-\int_{-w_{D}}^{w_{D}}\mathcal{D(\xi)}\tanh\left(\xi/2T\right)/2\xi d\xi$ is the standard $s$-wave susceptibility, $D(\xi)$ is the density of states and $w_{D}$ is the Debye frequency.

By numerically solving the $U-V$ Eq. (S121), we obtain the critical temperature $T_{c}$ for each pairing channel. Fig. 1 shows the resulting phase boundaries that delineate the regions where either $\Delta_{0}$ or $\Delta_{1}$ correspond to higher critical temperature and so is the dominant form of pairing. In Fig. 1, we plot the phase boundary as we vary $\Delta_{Z}$ and the ratio $U/V$. It is evident in Fig. 1 that the inclusion of the Zeeman effect results in the enhancement of the triplet ($\Delta_{0}$) pairing and the suppression of the singlet one ($\Delta_{1}$) at a given chemical potential.

Vortex Modes: Having established the phase diagram of the superconducting pairing, we proceed to study the internal structure of the vortices as we insert a $\pi$-flux-tube into the pairing potential: $\Delta_{\alpha}(r)=|\Delta_{\alpha}(r)|e^{i\theta}$ PhysRevLett.107.097001; chingkaichiu2011.

We fix $k_{z}=0$ as we are interested in points where the topological $\mathbb{Z}_{2}$ index changes and this only occurs at $k_{z}=0$ or $\pi$. Since the vortex modes stem from states close to the Fermi energy, their wave functions can be expressed in terms of a superposition of the TI conduction band eigenstates in cylindrical coordinates $\chi_{l,k}^{\nu}$ (see supplementary material).

Here $l$s are angular momentum quantum numbers and $\nu=\pm$ correspond to the two energy bands of Hamiltonian (1) which are split by Zeeman coupling.

Given the rotational symmetry of the vortex profile, the vortex modes with different $l$s are not hybridized and the vortex Hamiltonian decouples into sectors associated with each $l$. Also for inter-orbital triplet pairing, the vortex does not mixes the $\nu$’s. On the other hand, the translation symmetry in the plane perpendicular to the vortex is broken and different radial momenta $k$, as well as Nambu particle-hole states are mixed by the vortex. The effective vortex Hamiltonian acting in the radial momentum and Nambu particle-hole spaces takes the form of a 1D Jackiw-Rebbi model PhysRevD.13.3398 (see supplementary material)

[TABLE]

The $\Pi$ matrices act on a Nambu space, $\xi_{0}$ is the superconducting coherence length and $\Delta_{0}\lambda_{\nu}\left(k\right)/\xi_{0}=k+\frac{hm_{\mathbf{k}}}{2E^{0}_{\mathbf{k}}}\nu-\mu$. The Jackiw-Rebbi lowest-energy solutions are localized at the Fermi surface where the coefficient $\lambda_{\nu}\left(k_{F}^{\nu}\right)$ changes sign. These states have the form $e^{-\int_{k_{F}^{\nu}-k}^{k_{F}^{\nu}+k}dk^{\prime}\lambda_{\nu}\left(k^{\prime}\right)}\left(1,1\right)^{T}$ in Nambu particle-hole basis and has energy $E_{l,0}^{\nu}(k_{F})$. Notice that vortex-mode wave functions are exponentially localized around Fermi wavevector. Since the BdG Hamiltonian in equation (5) is in the bases of TI conduction band states, the full wave function of the two vortex modes (each associated with one Zeeman split Fermi surface) takes the form

[TABLE]

where $\left(1,1\right)_{\Pi}^{T}$ is the spinor in Nambu particle-hole space. Previously, a similar result was obtained for the intra-orbital spin-singlet case PhysRevLett.107.097001; chingkaichiu2011. In that case, solutions where again centered at the metallic phase Fermi surface, with corresponding energies $E^{\nu}_{l,1}=\frac{\Delta}{k_{F}\xi_{1}}(2\pi l+\pi+\nu\phi_{B})$. $\phi_{B}$ is a Berry-phase-like term which permits zero-modes whenever $\phi_{B}=\pi$. In contrast, for the inter-orbital triplet case, the energies of the vortex modes are $E_{l,0}^{\nu}=\frac{\Delta_{0}}{k_{F}^{\nu}\xi_{0}}\left(2\pi l\right)$. They noticeably lack the Berry phase term present in the intra-orbital singlet case and the $l=0$ states always has zero energy. Therefore, in the inter-orbital triplet case, a zero-energy channel always exists along the vortex. Its presence delocalizes the zero-modes at the end of the vortex, on the sample surface. Delocalization of the zero mode at the ends of the vortex results in a suppression of the zero-bias signal in STM measurements.

In Fig. 2 we verify the analytic results using a 3D lattice model with periodic boundary conditions along the z-direction. Fig. 2 (a) shows the spectrum of the vortex modes for the intra-orbital singlet pairing where the vortex gap closes solely when $\phi_{B}=\pi$. In contrast, Fig. 2 (b) shows that for the inter-orbital triplet pairing, once the chemical potential is in the conduction band the system develops vortex zero modes which remain gapless for all chemical potentials. Thus, increasing the Zeeman coupling results in a shift of the Fermi surfaces that destabilize the intra-orbital singlet pairing in favor of the inter-orbital triplet pairing leading to the omnipresence of a zero mode in the vortex core.

In Fig. 3, we examine the manifestation of the change in superconductive pairing by numerically inserting a vortex in the tight-binding lattice model. We set $\mu=1.1t$, where $t$ is hoping amplitude in the lattice model. For small Zeeman couping, the intra-orbital singlet channel is dominant and the vortex states, both topological and trivial, are localized at the ends of the vortex on the surface. As the Zeeman splitting is increased, we destabilize the intra-orbital singlet pairing channel in favor of the inter-orbital triplet pairing allowing MFs, localized at the ends of the vortex string, to penetrate into the bulk. At sufficiently high Zeeman coupling, the intra-orital singlet pairing is fully suppressed and MFs on the surface delocalize through the vortex modes and extend into the bulk of the superconductor. We may also understand this mechanism, starting from the case where the pairing is solely inter-orbital triplet type where the vortex hosts two zero modes extended through the bulk. Intra-orbital singlet pairing hybridizes these two modes and generate a gap for the vortex modes extending through the bulk and localizes the MFs at the ends of the vortex on the surface. Interestingly, as will be shown below, this mechanism would only delocalize the Majorana zero modes and could not do the same for finite energy vortex modes at the ends of the vortex. It is then well consistent with the experimental results which shows that finite energy vortex modes are intact in all vortices, whereas the MFs are absent in some of the vortices.

Effective 1D Vortex Model: Using the wavefunctions from Eq. (6), we now derive an effective Hamiltonian for the modes along the vortex line connecting the surfaces  PhysRevB.92.064518. To this end, we calculate the matrix elements of the $k_{z}$-dependent terms in Hamiltonian, Eq. (1), within the space of the two vortex zero modes presented in Eq. (6). Without lose of generality we consider $m>0$ and $\epsilon<0$. Defining $\eta_{i}$ as Pauli matrices acting on the space of the two zero modes, the effective vortex Hamiltonian have the general form

[TABLE]

Here, $E^{\nu}_{l}$ are the energies of $l$-th vortex modes from Fermi surface corresponding to $\nu=\pm$. In Eq. (7), both $\tilde{\epsilon}$ and $\tilde{v}_{z}$ are parameters of the $k_{z}$-dependent terms. The effective one dimensional Hamiltonian in Eq. (7) corresponds to the effective vortex Hamiltonian and supports localized states at the two ends if $\left(E^{+}_{l}-E^{-}_{l}\right)\tilde{\epsilon}<0$ RevModPhys.83.1057.

From the definition of $\lambda_{\nu}\left(k_{F}^{\nu}\right)$, we notice $k_{F}^{+}<k_{F}^{-}$ and consequently, $E_{l}^{+}-E_{l}^{-}>0$. Therefore, the condition for the presence of localized states at the end of vortex with finite energy $\left(E^{+}_{l}-E^{-}_{l}\right)\tilde{\epsilon}<0$ is satisfied. The modes with $l\neq 0$ and finite energies of $\frac{E^{+}_{l}+E^{-}_{l}}{2}$ are localized at the ends of the vortex lines. In contrast, for the two zero energy modes ($l=0$) with $\left(E^{+}_{0}-E^{-}_{0}\right)\tilde{\epsilon}=0$. As a result, the MFs can not stay localized at the ends of the vortex when the bulk pairing is of inter-orbital triplet form. As shown in Fig. 4(b), the effective vortex Hamiltonian for the zero energy states is linearly dispersing with momentum along the vortex and, due to their Nambu particle-hole character, protected against back-scattering which further supports the delocalization of zero modes upon formation of inter-orbital triplet pairing.

Conclusion: We have examined the effect of Zeeman coupling on the structure of vortex modes in FTS. We find that the inter-orbital triplet paring and intra-orbital singlet pairing compete as a function of Zeeman coupling, resulting in dramatically different vortex structures. Intra-orbital singlet pairing leads to the presence of Majorana vortex modes localized on the surface at the end of vortex strings. On the other hand, inter-orbital triplet paring supports localized finite-energy trivial vortex modes but destabilize the zero-energy Majorana modes at the end of vortex strings. This delocalization is through the formation of zero modes which extend along the vortex through the bulk. Such extended states have small coupling with STM probe and do not show strong signals, in comparison with the case where the zero modes are localized at the end of vortex, on the sample surface. Our results shed light in the existence of two types of vortices experimentally observed in FTS and support the topological nature of vortex modes in this system.

The main property of the band structure of FTS to facilitate our models is the presence of degenerate Fermi surfaces that are split into two helical Fermi surfaces by the Zeeman field. The size of the superconducting gap in FTS is of the order of $2.5\,\mathrm{meV}$ for hole pocket and $4.2\,\mathrm{meV}$ for electron pocket PhysRevLett.108.107002; Yin2015; Zhang182. The size of the Fe impurities dipole-moments in FTS is of the order of $5\mu_{B}$ PhysRevLett.108.107002; PhysRevX.9.011033. Given the average distance of the Fe impurity atoms, their associated Zeeman coupling is of the order of $\Delta_{Z}\approx 7.84\,\mathrm{meV}$. It is then evident that the Zeeman coupling can affect the form of the superconducting paring. Our results then demonstrate that the nature of the vortices in FTS is inextricably linked to the effect of Zeeman coupling, which determines the form of superconducting pairing. It also indicates that suppression of Zeeman coupling, by reduction of magnetic impurities, stabilizes the vortex MFs in FTS.

Acknowledgments. This research was supported under National Science Foundation Grants EFRI-1542863 and PSC-CUNY Award, jointly funded by The Professional Staff Congress and The City University of New York (PG). PLSL acknowledges support by the Canada First Research Excellence Fund. AG acknowledges support from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 754411. PH was supported by the Department of Physics and the College of Natural Sciences and Mathematics at the University of Houston. M.J.G. acknowledges financial support from the National Science Foundation (NSF) under Grant No. DMR-1720633, CAREER Award ECCS-1351871 and the Office of Naval Research (ONR) under grant N00014-17-1-3012.