Quantum oscillations in strongly correlated topological Kondo insulators

TL;DR

This paper explains quantum oscillations in topological Kondo insulators like SmB6 and YbB12 without a Fermi surface, highlighting the role of correlations and nonlocal hybridization through dynamical mean-field calculations.

Contribution

It demonstrates that correlations and hybridization effects can produce observable quantum oscillations in insulators lacking a Fermi surface, a novel insight into these materials.

Findings

Quantum oscillations can occur without a Fermi surface due to correlations.

Correlations facilitate the observation of quantum oscillations in bulk properties.

The mechanism involves Landau levels approaching the Fermi energy.

Abstract

The observation of quantum oscillations in topological Kondo insulators SmB6 and YbB12 is a recent puzzling experimental discovery. Quantum oscillations observed in the resistivity and the magnetization are usually explained by the existence of the Fermi surface. However, Kondo insulators do not have a Fermi surface and thus should not show quantum oscillations. By performing dynamical mean-field calculations for topologically nontrivial Kondo insulators in a magnetic field, we analyze the effect of correlations on the emergence of quantum oscillations in narrow-gap topological Kondo insulators and demonstrate that the interplay between correlations and nonlocal hybridization, ubiquitously occurring in topological Kondo insulators, can lead to observable quantum oscillations without the necessity of a Fermi surface. Particularly, we show that correlations make it easier to observe quantum oscillations in the magnetization and the resistivity of the bulk material. The fundamental mechanism for these quantum oscillations is a combination of correlation effects and Landau levels coming very close to the Fermi energy. We furthermore demonstrate that quantum oscillations in a three-dimensional system can be understood by analyzing the physics on the two-dimensional planes in the momentum space for which the hybridization in direction of the magnetic field vanishes. We believe that this scenario is relevant to understanding the observation of quantum oscillations in the magnetic torque for SmB6 as well as oscillations in the resistivity and the magnetic torque of YbB12.

Figures

Equations

$$H$$

$$H_{\text{Hyb}}$$

$${\varphi }_{x,y}^x$$

$$H$$

$$\Rightarrow H$$

$${\Psi }_{x,y}=\exp (i{k}_{x}x)\exp (i{k}_{y}y){\varphi }_y\text{with}{\varphi }_{y+N}={\varphi }_y,$$

$$H$$

$$G(\omega {)}_{ii}=\int d\vec{k}{\left(\omega +i0-H(\vec{k})-\Sigma (\omega )\right)}_{ii}^{-1},$$

$${\Delta }_{ii}(\omega )=\omega +i0-G_{ii}^{-1}-{\Sigma }_{ii}(\omega ).$$

$$E_l^c(B)$$

$$E_l^f(B)$$

$$B_l$$

$$E\sim \frac{2\pi }{\partial A_c(E)+\partial A_f(E)}B=\frac{1}{m_c+m_f}B.$$

$$H$$

$$H_0$$

$${ϵ}_k^c$$

$${ϵ}_k^f$$

$$H_{\mathrm{int}}$$

Full text

I introduction

One of the most puzzling recent experimental discoveries in condensed matter physics is the observation of quantum oscillations in insulating materials SmB6 and YbB12Tan et al. (2015); Xiang et al. (2018); Ong (2018); Liu et al. (2018); Hartstein et al. (2017). While firstly these oscillations have been attributed to metallic surface statesLi et al. (2014), there is also strong evidence that they arise from the bulkTan et al. (2015); Xiang et al. (2018); Ong (2018); Liu et al. (2018); Hartstein et al. (2017); Erten et al. (2016). Both materials are strongly correlated $f$ electron systems for which a gap develops because of a hybridization between conduction ($c$) electrons and strongly correlated $f$ electrons, and thus a large resistivity at low temperatures can be measured Fisk et al. (1995); Riseborough (2000). Surprisingly, quantum oscillations in strong magnetic fields have been observed in SmB6 in the magnetic torqueTan et al. (2015) and in YbB12 in the resistivity and the magnetic torqueXiang et al. (2018). These measurements contradict our understanding of quantum oscillations, which is rooted in the existence of a Fermi surface; electron bands, which create the Fermi surface, form Landau levels in a magnetic field. When the magnetic field strength is changed, the energies of these Landau levels change, leading to an oscillatory behavior in most of the observable quantities. However, insulating materials such as SmB6 and YbB12 do not possess a Fermi surface, thus there are no electrons, which can form Landau levels, close to the Fermi energy. Because of this discovery, different theories have been used to explain these observations: quantum oscillations might be observed under special conditions, if the gap is very smallKnolle and Cooper (2015); Pal et al. (2016); Pal (2017a, b, 2019); Ram and Kumar (2017); Riseborough and Fisk (2017). Other theories explain these quantum oscillations by composite-excitonsKnolle and Cooper (2017); Chowdhury et al. (2018) or Majorana-fermionsErten et al. (2017), which form a Fermi surface. However, the existence of fermionic charge-neutral excitations is highly controversial.Xu et al. (2016) Furthermore, it has been pointed out that quantum oscillations can be understood using non-hermitian properties of the materialShen and Fu (2018). Besides these works, in a study for narrow-gap topological insulators, Zhang et al.Zhang et al. (2016) have shown in a noninteracting continuum model that the gap of a topological insulator closes at a critical magnetic field, which has been further analyzed in other works.Grubinskas and Fritz (2018) They have shown that this gap closing is accompanied by oscillatory behavior in observable quantities. In this way, there are a variety of theories based on different assumptions, which might explain these quantum oscillations. However, a conclusive answer has not been found yet, although this problem is of uttermost importance: Quantum oscillations are viewed as an accurate method for the experimental determination of the Fermi surface of materials, and the measurement of quantum oscillations in an insulator contradict the existing theories.

The only examples of insulating material showing quantum oscillations in magnetic fields are SmB6 and YbB12, which are both good candidates for topological Kondo insulators.Takimoto (2011); Lu et al. (2013); Alexandrov et al. (2013); Dzero et al. (2012, 2010); Weng et al. (2014); Dzero et al. (2016); Peters et al. (2016) Thus, it is natural to ask, whether nontrivial topology, strong correlations, or a combination of both is essential to observe quantum oscillations in insulating materials.

We here answer this question by analyzing correlation effects on quantum oscillations appearing in the bulk of a two-dimensional (2D) topological Kondo insulator. (i) We confirm in our work that the gap closing due to a momentum-dependent hybridization as described by Zhang et al.Zhang et al. (2016) for a noninteracting continuum model exists in strongly correlated lattice models. Such a momentum-dependent hybridization is ubiquitously found in topological Kondo insulators. (ii) We show that not only the gap-width is reduced by the renormalization of the band structure, but also the slope of the gap-closing with increasing magnetic field strength decreases. Thus, the critical field strength, where the gap closes and the insulator changes into a metal, is nearly unchanged. (iii) We demonstrate that the amplitude of the quantum oscillations is enhanced by correlations due to the finite life-time of quasi-particles. We believe that this together with the renormalization of the band structure is an important fact for the experimental detection and might explain why quantum oscillations in insulating materials have been so far only observed in strongly interacting topological materials. (iv) Finally, we demonstrate that the quantum oscillations occurring in the bulk of a three-dimensional (3D) system are determined by the momenta where the hybridization in the direction of the magnetic field vanishes, and that the physics at the Fermi energy of the 3D bulk is essentially the same as in 2D. This validates the relevance of our study to 3D materials such as SmB6 and YbB12.

The rest of this paper is organized as follows: In the next section, Sec. II, we introduce the model and methods used to analyze the quantum oscillations in a topological Kondo insulator. This is followed by a section about the gap closing in the noninteracting model, Sec. III. The main results are shown in Sec. IV, where we analyze quantum oscillations due to the interplay of strong correlations and nontrivial topology. In Sec. V, we show results for a 3D model and explain the relation and relevance of the 2D model to 3D systems. Finally, in Sec. VI, we discuss the obtained results and the relevance for SmB6 as well as for YbB12 and conclude the paper.

II Model and method

II.1 Model

Because both materials, SmB6 and YbB12, have different band structures, we believe that the details of the band structure may not play a major role for explaining these quantum oscillations. To obtain generic properties, we thus study a model of a two-dimensional topological Kondo insulator which captures the essence of the interplay between correlations and nontrivial topology.

Our model includes one $f$-electron- and one $c$-electron-band, which hybridize via a nonlocal hybridization. This results in a topologically nontrivial gap at the Fermi energy. We furthermore include a strong local Coulomb interaction in the $f$-electron band. The model is thus a periodic Anderson model with a momentum dependent hybridization, and can be written as

[TABLE]

with

[TABLE]

The band structure corresponds to a 2D tight binding model with only nearest neighbor hopping. The dispersion reads $\epsilon^{c/f}_{\vec{k}}=2t_{c/f}(\cos k_{x}+\cos k_{y})$. We choose the hopping of the $c$ electrons, $t_{c}=t$, as unit of energy throughout this paper and set the hybridization strength to $V/t=0.1$ and $t_{f}/t=0.2$. $H_{\text{Hyb}}$ describes the hybridization between $f$ and $c$ electrons, which results in a topologically nontrivial gap at the Fermi energy. Besides a hybridization arising from spin-orbit interaction, we need to change the $f$ electron number away from $n_{f}=1$ in order to obtain a topological insulator. In the absence of a magnetic field, the analyzed model has only time-reversal symmetry and thus can be a topological insulator of class AII where systems may have a $Z_{2}$ topological invariant in two and three dimensions.Chiu et al. (2016) Because our model has furthermore inversion symmetry, the topological invariant can be easily calculated in the noninteracting model by the product of the eigenvalues of the parity operator for the occupied states at the time-reversal invariant momenta in the Brillouin zone.Fu and Kane (2007) We find that this product is negative, corresponding to a topologically nontrivial model. We note that the $Z_{2}$ invariant for correlated systems can be defined by the single-particle Green’s function as long as it is nonsingular (i.e., there is neither gap-closing nor a divergence of the self-energy).Wang and Zhang (2012) We have confirmed that the $Z_{2}$ invariant remains nontrivial even in the strongly correlated region. Besides verifying these conditions, we also have directly calculated the surface spectrum without magnetic field and have confirmed the existence of metallic surface states at $\vec{k}=(0,0)$. We thus confirm that the analyzed model is a topological Kondo insulator. In Fig. 1, we show a typical band structure for the valence fluctuating regime with $n_{f}=1.5$ and $n_{c}=0.5$. Clearly visible is a flat $f$ electron band and a wide $c$ electron band. Furthermore, at the Fermi energy $E/t=0$, a gap exists.

II.2 Peierls phase on a lattice

Throughout this paper, we consider a magnetic field $\vec{B}=B(0,0,1)$ in $z$-direction. When a magnetic field is applied to electrons on a lattice, the Hamiltonian must be modified using the Peierls substitutionPeierls (1933), which takes into account that electrons moving in a closed path acquire a phase proportional to the magnetic flux through the area enclosed by the path. We here describe a system with a vector potential $\vec{A}=B(-y,0,0)$ which leads to a magnetic field in $z$-direction with strength $B$. To incorporate a vector potential into a tight binding model, we include the appropriate Perierls phases

[TABLE]

where $A_{x}(x,y)$ and $A_{y}(x,y)$ are given by the vector potential at a lattice site $(x,y)$ in $x$ and $y$ directions, respectively. Thus, the hopping in $x$ ($y$) direction must be modified in a magnetic field by an additional factor $\exp(i\phi^{x}_{x,y})$ [$\exp(i\phi^{y}_{x,y})$].

For a homogeneous magnetic field with vector potential $A=B(-y,0,0)$, we only need to modify the hopping in $x$ direction by a phase factor which depends on the $y$ coordinate of the lattice site, reading

[TABLE]

where the first (second) equation describes the system without (with) magnetic field. We thus need to take into account that the wavefunctions on lattice sites with different $y$-index are different. However, if $B=2\pi\frac{M}{N}$ with some integers $M$ and $N$, the wavefunctions on lattice sites with $y$ and $y+N$ are equal, because the phase factors are equal. We thus make an ansatz for the wavefunction for every lattice site $y=1\ldots N$

[TABLE]

and enlarge the unit cell to the magnetic unit cell including $N$ different lattice sites along the $y$-direction.

Our original Hamiltonian can be written as

[TABLE]

where $\vec{k}=(k_{x},k_{y})$.

This Hamiltonian with periodic boundary conditions enables us to calculate properties for magnetic field strengths which can be written as $B=2\pi\frac{M}{N}$ where $M$ and $N$ are integers. For a tight binding model with a single band on a square lattice, the energy dispersion yields the famous Hofstadter butterfly spectrum. For our model consisting of $4$ different bands (including the spins of the $c$ and $f$ electrons), we obtain an energy diagram which is the overlap of two Hofstadter butterflies, see Fig. 2.

II.3 Boundary conditions

As explained above, for a given magnetic field $B=2\pi\frac{M}{N}$ ($M$ and $N$ are integers), the unit cell should be enlarged to $N$ lattice sites. As a consequence, only calculations for magnetic fields where the magnetic field is a rational number $\frac{M}{N}$ of $2\pi$ can be exactly simulated. Furthermore, smoothly varying the magnetic field strength is not possible, because the magnetic unit cell changes. Finally, if the magnetic unit cell becomes too large, numerical calculations become impossible due to limited computer resources.

We thus choose here a different approach. Instead of simulating a bulk system for which the system size corresponds to the magnetic unit cell, we simulate an open boundary system consisting of $80$ lattice sites. Because the lattice has open boundaries (there are no periodic boundary conditions), the magnetic flux through the whole lattice is not necessarily a multiple of $\frac{2\pi}{80}$, but can be any value. We note here that one can expect deviations from the bulk calculations particularly when the magnetic field strength is very weak, implying very large magnetic unit cells. Details and comparisons between calculations with different boundary conditions are shown in the appendix. There are no qualitative differences for the observation of quantum oscillations between calculations with different boundary conditions.

II.4 Correlations

We incorporate correlation effects via the real-space dynamical mean field theory (RDMFT), which is an extension of the dynamical mean field theoryGeorges et al. (1996) to inhomogeneous situations. The inhomogeneity arises here from the Peierls phases of the magnetic field leading to a large magnetic unit cell.

To perform the real-space DMFT calculations, we calculate the local Green’s function via

[TABLE]

where $i$ is the index of the site, $\omega$ the frequency, $H(\vec{k})$ the one-particle part of the Hamiltonian of the finite slab or the magnetic unit cell including all Peierls phases, and $\Sigma(\omega)$ a matrix containing the self-energies of all lattice sites. In the first iteration of the real-space DMFT, this self-energy can be set to zero. For the system consisting of $80$ layers, we thus find $80$ local Green’s functions. For a periodic system with magnetic unit cell of $N$ sites, we have $N$ different Green’s functions. From these local Green’s functions, we set up independent impurity models by determining the hybridization functionsBulla et al. (2008)

[TABLE]

These impurity models are solved using the numerical renormalization group (NRG)Bulla et al. (2008) from which the self-energy for each lattice site is calculated. These self-energies are then used to calculate new local Green’s functions of all lattice sites. This is iterated until self-consistency is reached.

The usage of NRG enables us to calculate self-energies on the real-frequency axis with high precision around the Fermi energyPeters et al. (2006). NRG uses a logarithmic discretization of bath states to iteratively diagonalize the impurity Hamiltonian. This logarithmic discretization results in a poor resolution away from the Fermi energy but a high resolution around the Fermi energy. Because the observed quantum oscillations are caused by Landau levels close to the Fermi energy, it is expected that NRG yields accurate results. To confirm that our results are not influenced by the discretization, we have performed test calculations in a magnetic field with different discretization parameters and have found no discernible difference. All calculations are done for $T/t=10^{-4}$ which is far below the Kondo temperature and the gap width of the model. We note that strong correlations can lead to the emergence of long-range ordered magnetic phases in the periodic Anderson model besides the appearance of a ferromagnetic polarization due to the magnetic field. Possible phases are antiferromagnetic phases or charge ordered phases, as has been shown in previous papersPeters et al. (2018). Such phase transitions can also be triggered by including the magnetic field, particularly because the gap width is suppressed by the magnetic field. As we are interested here in the paramagnetic phase only showing the polarization due to the magnetic field, we suppress such metamagnetic transitions.

III Gap closing mechanism in the noninteracting model

In this section, we firstly analyze the noninteracting model to demonstrate that the gap closes in strong magnetic fields. The gap closing mechanism in the noninteracting model for strong magnetic fields is equivalent to the situation described in the work of Zhang et al.Zhang et al. (2016) in a noninteracting continuum model. In Fig. 3 (top), we show the energy diagram for $n_{f}=1.5$ in the noninteracting model. In this plot, energy levels approaching the Fermi energy with nearly linear magnetic field dependence are clearly visible. These energy levels can be understood as Landau levels of the $c$ and $f$ electrons. We can write these as

[TABLE]

where we use $D_{c}$ and $D_{f}$ as fitting parameters for $c$ and $f$ electrons, and $A_{c,f}(E)$ is the area of the Fermi surface at energy $E$ for the $c$ and $f$ electrons. Due to the energy dependence of the derivative $\frac{\partial A^{c,f}(E)}{\partial E}$, the dispersion of the Landau levels is bent. In the bottom panel of Fig. 3, we include these Landau levels into the energy diagram. In this figure, we do not include a hybridization between these $c$ and $f$ electron Landau levels. Thus, these Landau levels do not form a hybridization gap at the Fermi energy. Figure 3 (bottom) shows a good agreement, away from the Fermi energy between the calculated energy diagram of the lattice model and these Landau levels.

To understand the formation and the closing of the hybridization gap, we need to include a hybridization between the $c$ and $f$ electron Landau levels. In contrast to $f$ electron systems with local hybridization, the hybridization originating in the spin-orbit interaction is an odd function of the momentum such as $\sigma_{i}\sin(k_{i})$, which leads to a hybridization of Landau levels $l$ with $l+1$ or $l$ with $l-1$.Zhang et al. (2016) The hybridization between Landau levels with different index can explain why the gap closes at strong magnetic fields. In Fig. 4, we use the Landau levels from Fig. 3 and include a hybridization between Landau levels with different index as described above.

Assuming that the gap opens when the energetic distance between the $c$ and the $f$ electron Landau levels is of the size of the hybridization, i.e. $V\approx E_{l}^{c}(B)-E_{l\pm 1}^{f}(B)$, we can derive

[TABLE]

Solving this equation for $l$ and inserting it into the energy equation of the Landau levels, we obtain the energy depending on the magnetic field where the hybdrization gaps open. The dependence of the energy on the magnetic field reads

[TABLE]

Thus, the slope of the gap closing in the energy diagram of the lattice model is proportional to the inverse of the mass of the electrons.

IV quantum oscillations in the strongly correlated model

We now analyze the effect of strong correlations on the above described gap closing and the visibility of quantum oscillations. Figure 5 shows the comparison of the energy level structure between the interacting and the noninteracting model in an applied magnetic field. The particle number of the $f$ electrons is fixed to $n_{f}=1.5$ corresponding to a valence fluctuating regime. We note that in this parameter regime, renormalization effects on the band structure are small even for an interaction strength $U/t=7$, which corresponds to $U/t_{f}=35$.

The energy level structures of the models are obtained from the maximum position of the peaks in the local Green’s function in the bulk. Clearly visible in Fig. 5 is the gap at $B/t=0$ around the Fermi energy $E/t=0$ and the Landau levels periodically approaching the Fermi energy. By increasing the magnetic field strength, Landau levels from above and below the Fermi energy approach $E/t=0$ and finally close the gap at $B_{c}/t\approx 4.5$. The mechanism of the gap closing is the same as described for the noninteracting continuum model.

Thus, Landau levels cross the Fermi energy for very strong magnetic fields. However, in SmB6 and YbB12 quantum oscillations are observed even in the insulating regime, i.e. before the magnetic breakdown.

We therefore show (a) the magnetization, (b) the density of states (DOS), and (c) the resistivity in Fig. 6. The magnetization of the $f$ electrons increases approximately linearly until the magnetic breakdown. A precise analysis of the magnetization by subtracting the linear part, however, reveals that it includes oscillations shown in Fig. 6(a). These oscillations occur well below the magnetic breakdown. Much more importantly, these oscillations are strongly enhanced by the correlations. While for the noninteracting model, the oscillations can be observed for $B/t>1.5$, in the interacting model they are visible for $B/t>0.5$. Directly in the DOS, which is small compared to the DOS at the magnetic breakdown, we can also see several small oscillations in the interacting model, see Fig. 6(b). Finally, we show the resistivity in Fig. 6(c) calculated by the Kubo formula within the DMFT approximation, which neglects vertex corrections. To use the vector potential $A=B(-y,0,0)$, we have introduced different layers in the $y$-direction in our real-space DMFT calculation. In the $x$-direction, the lattice still has periodic boundary conditions. We can thus calculate the conductivity in the $x$-direction using the Kubo-formula in the same way as done for superlattices.Peters et al. (2013)

We only show the results for the interacting system, because the resistivity of the noninteracting system strongly depends on an artificial broadening which must be included to obtain a finite resistivity. In the interacting system, such an artificial broadening is unnecessary because of the self-energy calculated by the DMFT. We see that the resistivity is large, as can be expected for an insulating material. Because of the gap closing, the resistivity decreases with increasing magnetic field. Furthermore, oscillations are clearly visible. We note that the numerical error can be expected to be constant for all magnetic field strengths. Thus, the oscillations observed at high magnetic fields cannot be attributed to numerical errors, as demonstrated by the smooth behavior of the observables at low magnetic fields. We thus conclude that even in these calculations which do not exhibit a strong renormalization effect because of the particle number, $n_{f}=1.5$, we nevertheless can observe oscillations in the magnetization and the resistivity which are clearly enhanced, compared to the noninteracting system. We note that the resistivity drops by two orders of magnitude to values below $10^{3}$ at the magnetic breakdown, $U/t=4.5$. This confirms that the quantum oscillations occur in the insulating regime.

Although the calculations in Figs. 5 and 6 are performed for strong interaction, there is only a weak renormalization because correlation effects are suppressed away from $n_{f}=1$ in our model. To better understand the effect of strong interactions and renormalization, we show in Figs. 7 and 8 calculations in the Kondo regime with particle number $n_{f}=1.04$.

For this particle number, the interaction strength $U/t=1.5$ already leads to a considerable renormalization as shown in Fig. 7. Increasing the interaction strength to $U/t=2$, we observe that the gap at zero magnetic field is smaller than $1/3$ of the noninteracting band gap. Irrespective of the interaction strength, we see that Landau levels approach the Fermi energy and close the gap at a critical magnetic field strength, $B_{c}/t\approx 2.4$. The renormalization has thereby two important effects. First, the gap at zero field becomes smaller. Second, the slope of the gap closing, which strongly depends on the mass of the particles involved, also becomes smaller. Thus, the critical field strength of the gap closing is essentially unchanged from the noninteracting case.

Figure 8 shows the magnetization and the resistivity in the Kondo regime over a wide range of magnetic fields including the insulating and the metallic regimes. As can be confirmed in Fig. 7, Landau levels cross the Fermi energy for magnetic field strengths larger than $B/t\approx 2.4$. Thus, the system shows normal metallic behavior for these magnetic field strengths. However, already for magnetic field strengths $2<B/t<2.4$, the gap becomes very small. This is confirmed by the resistivity which strongly decreases around $B/t\approx 2$. However, we note that the values of the resistivity for $B/t>2.4$ when a Landau level crosses the Fermi energy are another order of magnitude lower than the resistivity for magnetic fields $2<B/t<2.4$. We thus define three regions in our results: An insulating region with high resistivity, $B/t<2$, a crossover region with intermediate resistivity, $2<B/t<2.4$, and a metallic region $B/t>2.4$. Already in the insulating region, we observe small oscillatory behavior in the calculated properties. We note that by an analysis as done in Fig. 6 the number of visible oscillations increases. Landau levels are still energetically well separated from the Fermi energy in this regime and thus do not have big influence on the physical properties of the system. In the crossover region, when Landau levels come very close to the Fermi energy but do not yet cross it, Landau levels have a strong impact on the physical properties. We observe quantum oscillations with large amplitude in the resistivity and the magnetization of the system. The amplitude of these oscillations is much larger than in the noninteracting system. As before, we do not show the resistivity for the noninteracting system, because it becomes only finite when inserting an artificial and arbitrary life-time of the particles. In the metallic region for magnetic fields larger than the critical field strength, the amplitude of the quantum oscillations further increases.

It is remarkable that quantum oscillations with large amplitude can be observed in the crossover region although Landau levels do not reach the Fermi energy. The reason for this phenomenon is the finite-life time of the quasiparticles in the Landau levels induced by the correlation effects (imaginary part of the self-energy). We furthermore note that in the experiments for SmB6 and YbB12 the resistivity decreases before quantum oscillations are observed. This could be evidence for the narrowing of the gap with increasing the magnetic field strength as also found in our analysis.

Although these oscillations occur before the gap closes, Landau levels of the $c$ and $f$ electrons approaching the Fermi energy are responsible for these quantum oscillations. As can be seen in Figs. 5 and 7, Landau levels come very close to the Fermi energy. In Fig. 9, we show again the quantum oscillations in the DOS, the magnetization, and the resistivity in the Kondo regime, $n_{f}=1.04$, for $U/t=1.5$ plotted as $1/B$. Furthermore, we include into this figure the delta-peaks of the $c$ and $f$ electron Landau levels. These peaks correspond to the magnetic field strengths where the unhybridized $c$ and $f$ electron Landau levels, described by Eq. (7b), cross the Fermi energy. Although there is no perfect agreement between the quantum oscillations in the observed quantities and the peaks of the Landau levels, we see that the frequency of these Landau levels can explain the frequencies seen in the quantum oscillations of the DOS, the magnetization, and the resistivity. This proves that we can understand the quantum oscillations in strong magnetic fields before the magnetic breakdown as correlated Landau level approaching the Fermi energy. Correlations, which results in renormalization and broadening of the Landau levels, thereby strongly enhance the amplitude of the quantum oscillations in the observed quantities.

V three-dimensional system

Up to now, we have analyzed a 2D model of a topological Kondo insulator, where the magnetic field is perpendicular to the lattice. However, SmB6 and YbB12 are 3D materials. Electron hopping and hybridization in the direction of the magnetic field are not influenced by the Landau quantization. Thus, the hybridization in the direction of the magnetic field can open a gap between the $c$- and $f$-electron bands. As a consequence, the gap-closing mechanism due to the nonlocal hybridization and the quantum oscillations due to Landau levels approaching the Fermi energy as described above are influenced and might even not occur at all because of the hybridization in the direction of the magnetic field.

However, we here show that the above-mentioned physics remains essentially intact and the 3D system can be understood in the same way as the 2D model. For this purpose we use a 3D model of a topological Kondo insulator.Peters et al. (2018) The Hamiltonian reads

[TABLE]

The operator $c^{\dagger}_{k,\sigma,o}$ creates an electron with momentum $k$, spin direction $\sigma$ in orbital $o\in\{c,f\}$. $\epsilon^{o}_{k}$ describes the energy depending on the momentum for each orbital. The energies have been chosen in a way that there are band inversions between $c$-electrons and $f$-electrons at $(\pi,0,0)$, $(0,\pi,0)$, and $(0,0,\pi)$ in the Brillouin zone, which resembles qualitatively the band structure of SmB6. We include nearest neighbor, next-nearest neighbor and next-next-nearest neighbor hopping on a cubic lattice. Due to the hybridization, $V$, between the $c$-electron band and the $f$-electron band, a gap opens in the bulk spectrum. $\sigma^{x}$, $\sigma^{y}$, $\sigma^{z}$ are the Pauli matrices. The operator $n_{i,\sigma,c}$ and $n_{i,\sigma,f}$ are local density operators on lattice site $i$ for the $c$-electrons and $f$-electrons, respectively. Finally, $H_{\mathrm{int}}$ describes a repulsive local density-density interaction in the $f$-electron band, which is necessary to describe the Kondo effect in strongly interacting $f$-electron systems. We have used the same model to analyze the interplay between magnetism and topology in a topological Kondo insulator in our previous work.Peters et al. (2018). We again solve this model by means of real-space DMFT. Compared to the 2D model, the additional dimension shows up when calculating the local Green’s functions which stipulate the input of the impurity models. We use again the vector potential $\vec{A}=B(-y,0,0)$ for our calculations.

In Figs. 10 and 11, we show results for the valence-fluctuating regime, $n_{f}=1.6$, for $U/t=6$ and $V/t=0.2$. In Fig. 10, we again extract the Landau level position by analyzing the peaks in the Green’s function of the bulk system. Figure 10(a) shows the Landau level structure for the local bulk Green’s function, where we have integrated over $k_{z}$. We clearly see that Landau levels approach the Fermi energy and close the gap at a critical magnetic field strength, $B/t\approx 0.3$. As explained above, this might be surprising as the hybridization in $z$-direction acts between Landau levels with the same index and is not influenced by the magnetic field at all.

To understand the Landau level structure in Fig. 10(a), we show the level structure for separate momenta $k_{z}$ in (b)-(d). Here it becomes clear that depending on $k_{z}$ a gap at the Fermi energy exists or not. This can be understood by the fact that in a topological insulator the hybridization is momentum dependent and has nodes for which the hybridization vanishes. In our 3D model, the hybridization in $z$-direction is proportional to $\sin k_{z}$ and thus vanishes for $k_{z}=0$ and $k_{z}=\pm\pi$. For these momenta, only Landau levels with different index hybridize and the above-described physics for the 2D system holds true. It can be clearly seen that for $k_{z}=0$ [Fig. 10(b)] and $k_{z}=\pm\pi$ [Fig. 10(d)] Landau levels approach the Fermi energy and close the gap in the same way as in the valence-fluctuating regime in the 2D model. On the other hand, for $k_{z}=\pi/2$ [Fig. 10(c)] the hybridization in $z$-direction does not vanish and we observe a gap at the Fermi energy. Finally, the local Green’s function as shown in Fig. 10(a) is the integral/sum over all $k_{z}$. Thus, the physics at the Fermi energy is determined by the momenta where the hybridization in the direction of the magnetic field vanishes, because for these momenta Landau levels approach the Fermi energy and the gap is closed in the same way as in the 2D system. We note that the resolution of the figure is lower than for the 2D system, as the calculations are numerically more demanding. This leads to a low resolution of the structure: not all the points found in the level structure in Figs. 10 (b)-(d) can be seen in Fig. 10(a).

Having established the existence of the gap closing in strong magnetic fields, which is similar to the 2D model, it is not surprising to observe quantum oscillations in the magnetization before the gap closes, which is shown in Fig. 11. To enhance the clarity of the figure, we have subtracted a quadratic background. We clearly observe oscillatory behavior before the gap closes at $B/t=0.3$.

VI discussion and conclusions

Let us discuss these numerical results in more detail. Our calculations have shown that quantum oscillations can be observed in the DOS, the magnetization, and the resistivity of the bulk of a topological Kondo insulator for magnetic fields smaller than the magnetic breakdown. We note that topological surface states do not contribute to the oscillations shown here. However, we believe that metallic surface states will contribute to the experimental observed quantum oscillations. Thus, a computational analysis of how topological surface states contribute to the observed quantum oscillations is an interesting question, which is left as a future project.

The frequencies of the bulk quantum oscillations observed here, when plotting these quantities over $1/B$, agree well with the frequencies generated by unhybridized $c$- and $f$-electrons. Thus, the oscillations can be reproduced by taking into account the Landau levels of the light $c$ electrons and those of the heavy $f$ electrons. The Fermi surface which causes these oscillations is small, because the electrons must be treated as unhybridized. However, although the Fermi surface of the $f$ electrons is small, it is important to realize that the $f$ electrons are renormalized and heavy; the mass $\frac{1}{2\pi}\frac{\partial A_{f}(E)}{\partial E}=m_{f}^{*}$ is much larger than that of the $c$ electrons.

These results naturally lead us to propose the notion of a virtual Fermi surface, which is created by the unhybridized $c$ and $f$ electrons. This virtual Fermi surface can be observed at high magnetic fields. Because of a hybridization between $c$ and $f$ electrons, which is an odd function of the momentum such as $\sigma_{i}\sin(k_{i})$ and thus leads to a coupling of Landau levels with different index, the effect of the hybridization becomes invisible when the cyclotron frequency is of the order of the hybridization strength. The closing of the gap in strong magnetic fields and the observation of quantum oscillations thus provide evidence for a hybridization with odd momentum dependence, which ubiquitously appears in topological insulators. To observe the gap-closing and the related quantum oscillations in an insulator, this kind of hybridization is a necessary ingredient. For 3D materials, the gap closes exactly for the momenta for which the hybridization in direction of the magnetic field between $c$ and $f$ electrons vanishes. The physics at the Fermi energy can be understood by these 2D momentum planes.

Furthermore, we have demonstrated by direct calculation that strong correlations enhance the amplitude and thus the visibility of the oscillations in observable quantities such as the magnetization and the resistivity for magnetic fields smaller than the magnetic breakdown. There are two reasons for this. First, because of the renormalization, the slope of the gap closing and the gap width are reduced. If we assume that experiments can detect quantum oscillations of Landau levels which are slightly away (but not too far) from the Fermi energy, then the renormalization will enlarge the range of magnetic fields for which Landau levels are observable. This effect can be seen in Fig. 7 comparing between the noninteracting and the interacting energy level structure. Second, because of the self-energy arising from the strong correlations, quasiparticle bands away from the Fermi energy are broadened, and can thus influence observable quantities at the Fermi energy. To demonstrate this effect, we show in Fig. 12, the DOS and the imaginary part of the self-energy for $U/t=1.5$ and $B/t=2$ in the Kondo regime, $n_{f}=1.04$. In the DOS, we see several energetically separated peaks, which correspond to the Landau levels close to the Fermi energy. These Landau levels are broadened and thus the DOS at the Fermi energy is increased, making it possible to observe quantum oscillations at the Fermi energy. The broadening of the Landau levels is thereby given by the imaginary part of the self-energy shown in Fig. 12(b). This self-energy is self-consistently calculated by DMFT and describes the correlations (which leads to a finite life-time) of the $f$ electrons. We see that although the imaginary part of the self-energy becomes smaller when approaching the Fermi energy, it is still finite and results in the broadening of the Landau levels. We note that under certain conditions, the broadening and the renormalizing of quasiparticles bands due to the self-energy can turn even a band insulator into a metal Garg et al. (2006); Hoang (2010); Sentef et al. (2009); Bouadim et al. (2007). In this work, however, the self-energy of the $f$ electrons broadens quasiparticle bands close to the Fermi energy, resulting in a slight increase of the density of states at the Fermi energy for small magnetic fields. When quasiparticle bands eventually cross the Fermi energy at large magnetic fields, the DOS increases by several orders of magnitude. Thus, the resistivity is high for small magnetic fields consistent with an insulator. Furthermore, we note that if the imaginary part becomes too large (the life-time becomes too short), Landau levels in the DOS merge to broad bands and quantum oscillations would not be observable anymore.

We believe that this scenario can explain the observations in SmB6, which is a good candidate for a topological Kondo insulator, thus including strong correlations and an odd momentum-dependent hybridization. Oscillations are observed for strong magnetic fields but before the magnetic breakdown. Because the material is still insulating in these fields, the oscillations can be understood as strongly correlated Landau levels approaching but not crossing the Fermi energy. Furthermore, the experimentally observed oscillations show characteristics of the metallic rare-earth hexaborids,Tan et al. (2015) which do not form a hybridization gap at the Fermi energy. Thus, the conclusion that the oscillations originate in the unhybridized $c$ and $f$ electrons with virtual Fermi surface agrees well with the experimental observations in SmB6. On the other hand, the case of YbB12 seems to be more difficult, as there seems to be no direct correspondence between the observed oscillations in metallic LuB12. However, comparing the band structures of YbB12 and LuB12 Liu et al. (2018), it seems possible that the Boron bands are energetically shifted in these compounds, which would explain different quantum oscillations. Furthermore, because SmB6 and YbB12 are topologically nontrivial, it can be expected that metallic surface states contribute to the experimentally observed quantum oscillations. Thus, it remains an interesting problem to distinguish quantum oscillations due to the metallic surface states from those due to the insulating bulk in the experimental data.

In summary, we have shown that quantum oscillations can be observed in topological Kondo insulators for magnetic fields before the magnetic breakdown. While it is difficult to observe quantum oscillations in the noninteracting model, we have demonstrated that oscillations are strongly enhanced in the correlated model. Thus, strong correlations are essential for the experimental observation of quantum oscillations in an insulator. We have further shown that the quantum oscillations can be explained by a virtual Fermi surface made of unhybridized light $c$ and heavy $f$ electrons.

Acknowledgements.

This work is partly supported by JSPS KAKENHI Grant No. 25220711, JP15H05855, JP16K05501, 18K03511, 18H05842, and No. 18H04316 (JPhysics) and CREST, JST No. JPMJCR1673. Computer simulations were performed on the ”Hokusai” supercomputer in RIKEN and the supercomputer of the Institute for Solid State Physics (ISSP) in Japan.

Appendix A Comparison between periodic and open boundary conditions

To demonstrate that our approach yields correct results for the magnetic field strengths for which we observe the gap closing, we show in Fig. 13 a comparison between the local density of states in the middle of the slab calculations (open boundary conditions) and the periodic boundary conditions. While the upper panels show the $c$ electron bands for two magnetic field strengths, the lower panels show the $f$ electron band. The local DOS of the $c$ electron band clearly demonstrates the existence of narrow bands, showing up as peaks in the DOS. These narrow bands correspond to the Landau levels. With increasing magnetic field strength the distance between the Landau levels grows. The $f$ electron band, on the other hand, includes rather broad peaks, especially away from the Fermi energy. The emergence of broad peaks, instead of very flat bands (narrow peaks), is due to the strong correlations in the $f$ electron band. Strong correlations lead to a finite lifetime of the particles away from the Fermi energy, resulting in broadened bands. Comparing the spectral function calculated using periodic boundary conditions with those calculate with open boundary conditions, we clearly see that there is very good agreement between both calculations. This demonstrates that our calculations with open boundary conditions yield correct results.