Terahertz Faraday and Kerr rotation spectroscopy of Bi$_{1-x}$Sb$_x$ films in high magnetic fields up to 30 Tesla

TL;DR

This study uses terahertz Faraday and Kerr rotation spectroscopy in high magnetic fields to distinguish bulk and surface carrier contributions in Bi$_{1-x}$Sb$_x$ films, revealing their electronic structure and topological phase transition.

Contribution

It provides a comprehensive experimental and theoretical analysis of carrier dynamics in Bi$_{1-x}$Sb$_x$ films under high magnetic fields, uniquely identifying surface and bulk contributions related to topological properties.

Findings

Semimetallic and topological insulator films show distinct Faraday and Kerr spectra.

Theoretical models accurately reproduce experimental spectra based on band parameters.

High magnetic field terahertz spectroscopy can differentiate topological from nontopological carriers.

Abstract

We report results of terahertz Faraday and Kerr rotation spectroscopy measurements on thin films of $\text{Bi}_{1-x}\text{Sb}_{x}$, an alloy system that exhibits a semimetal-to-topological-insulator transition as the Sb composition $x$ increases. By using a single-shot time-domain terahertz spectroscopy setup combined with a table-top pulsed mini-coil magnet, we conducted measurements in magnetic fields up to 30~T, observing distinctly different behaviors between semimetallic ($x < 0.07$) and topological insulator ($x > 0.07$) samples. Faraday and Kerr rotation spectra for the semimetallic films showed a pronounced dip that blue-shifted with the magnetic field, whereas spectra for the topological insulator films were positive and featureless, increasing in amplitude with increasing magnetic field and eventually saturating at high fields ($>$20~T). Ellipticity spectra for the semimetallic films showed resonances, whereas the topological insulator films showed no detectable ellipticity. To explain these observations, we developed a theoretical model based on realistic band parameters and the Kubo formula for calculating the optical conductivity of Landau-quantized charge carriers. Our calculations quantitatively reproduced all experimental features, establishing that the Faraday and Kerr signals in the semimetallic films predominantly arise from bulk hole cyclotron resonances while the signals in the topological insulator films represent combined effects of surface carriers originating from multiple electron and hole pockets. These results demonstrate that the use of high magnetic fields in terahertz magnetopolarimetry, combined with detailed electronic structure and conductivity calculations, allows us to unambiguously identify and quantitatively determine unique contributions from different species of carriers of topological and nontopological nature in Bi$_{1-x}$Sb$_x$.

Tables

Table 1. Table 1: Characteristics of the five Bi 1 − x ​ Sb x subscript Bi 1 𝑥 subscript Sb 𝑥 \text{Bi}_{1-x}\text{Sb}_{x} thin film samples studied. The Sb content ( x 𝑥 x ), film thickness ( t 𝑡 t ), and crystal orientation of the films are shown. The experimental methods used to determine the parameters are indicated in the parentheses. SM = semimetal. TI = topological insulator.

Sample No.	Nominal $x$	Actual $x$	Thickness $t$ (nm)	Orientation	Character
1	0	0	68 (AFM)	(RHEED)	SM
2	0.04	0.03 (XRF)	60 (XRF)	(XRD)	SM
3	0.08		70 (AFM)	N/A	TI
4	0.10	0.136 (XRF)	54 (XRF)	and (XRD) (ARPES)	TI
5	0.15	$x>0.136$	77 (AFM)	(RHEED)	TI

Equations

$$E_{\pm }(\omega )=\frac{1}{\sqrt{2}}[E_x(\omega )\pm i{E}_y(\omega )].$$

$$\theta (\omega )=$$

$$\eta (\omega )=$$

$${\sigma }_{\pm }(\omega )=\frac{(1+n_{\text{Si}})(1-t_{\pm })}{Z_{0}d{t}_{\pm }},$$

$${\sigma }_{xx}=$$

$${\sigma }_{xy}=$$

$${\mathcal{H}}_h$$

$${\mathcal{H}}_e$$

$${\sigma }_{\alpha \beta }(\omega )=\frac{i\hslash }{S}\sum _{m,n}\frac{f_m-f_n}{E_m-E_n}\frac{⟨{\Psi }_m|{\hat{j}}_{\alpha }|{\Psi }_n⟩⟨{\Psi }_n|{\hat{j}}_{\beta }|{\Psi }_m⟩}{\hslash \omega +E_m-E_n+i\gamma },$$

$$E_{h,n}$$

$$E_{e,n}$$

$${\Psi }_{n,\downarrow }$$

$${\Psi }_{n,\uparrow }$$

$${\mathcal{H}}_e^{\bar{\Gamma }}$$

$${\mathcal{H}}_{h1}$$

$${\mathcal{H}}_{e1}^{\bar{\text{M}}}$$

$${\mathcal{H}}_e$$

$$|{\Psi }_n⟩=\left(\begin{array}{c}{A}_n|n-1⟩\\ B_n|n⟩\end{array}\right),$$

$$\left(\begin{array}{cc}0& \frac{\sqrt{2}\hslash v_{\text{D}}}{{\ell }_B}\sqrt{n}\\ \frac{\sqrt{2}\hslash v_{\text{D}}}{{\ell }_B}\sqrt{n}& 0\end{array}\right)\left(\begin{array}{c}{A}_n\\ B_n\end{array}\right)=E_n\left(\begin{array}{c}{A}_n\\ B_n\end{array}\right).$$

$$E_n=\text{sgn}(n)\sqrt{\frac{2{\hslash }^2{v}_{\text{D}}^2}{{\ell }^2}|n|}.n=0,1,2,\cdots$$

$$A_n$$

$$B_n$$

$$\hat{\mathbf{j}}=\frac{ie}{\hslash }[{\mathcal{H}}_e,\hat{\mathbf{x}}],$$

$$⟨{\Psi }_m|{\hat{j}}_x|{\Psi }_n⟩$$

$$⟨{\Psi }_m|{\hat{j}}_y|{\Psi }_n⟩$$

$${\sigma }_{xx}^{(e,\bar{\Gamma })}$$

$${\sigma }_{xy}^{(e,\bar{\Gamma })}$$

$${\mathcal{H}}_{h1}$$

$${\hat{\tilde{j}}}_x$$

$${\hat{\tilde{j}}}_y$$

$$⟨m|{\hat{\tilde{j}}}_x|n⟩$$

$$=-\frac{e\hslash }{\sqrt{2}M{\ell }_B}(\sqrt{n}{\delta }_{m,n-1}+\sqrt{n+1}{\delta }_{m,n+1})$$

$$⟨m|{\hat{\tilde{j}}}_y|n⟩$$

$$=\frac{ie\hslash }{\sqrt{2}M{\ell }_B}(\sqrt{n+1}{\delta }_{m,n+1}-\sqrt{n}{\delta }_{m,n-1}),$$

$${\tilde{\sigma }}_{xx}^{(h1)}(\omega )$$

$${\tilde{\sigma }}_{xy}^{(h1)}(\omega )$$

$${\sigma }_{xx}^{(h1)}(\omega )$$

$${\sigma }_{yy}^{(h1)}(\omega )$$

Full text

I Introduction

One of the most appealing methods proposed for probing topologically protected surface states in a three-dimensional (3D) topological insulator (TI) is observing the topological magnetoelectric (TME) effect [1, 2, 3, 4, 5, 6, 7, 8]. When time-reversal symmetry is broken by a magnetic field, a gap appears in the surface Dirac state(s), and the Hall conductivity of the sample surface is half-integer quantized, which in turn leads to a quantized bulk magnetoelectric response. This phenomenon can be described through an electromagnetic Lagrangian analogous to the theory of axion electrodynamics in particle physics [9]. Experimentally, the most common and established experimental techniques for studying 3D TIs are angle-resolved photoemission spectroscopy (ARPES) [10, 11, 12, 13, 14, 15, 16, 17, 18] and electronic transport [19, 20, 21, 22, 23, 24, 25] measurements, both of which are not suited for detecting the TME effect. Specifically, a magnetic field cannot be applied in ARPES experiments, and there are limitations due to nontopological edge states in transport experiments.

Magneto-optical spectroscopy at low photon energies, such as the microwave and far-infrared spectral ranges, has been recognized to be an ideal probe for the TME effect [4, 3, 5]. By using electromagnetic radiation whose photon energy is comparable to or smaller than the magnetic-field-induced surface gap, previous terahertz (THz) magneto-optical spectroscopy experiments have detected changes of the surface optical conductivity induced by an applied small or moderate magnetic field [26, 27, 28, 29, 30, 31, 32]. Applying a much stronger magnetic field on the sample is, generally speaking, expected to be more advantageous because the field can induce a gap much larger than the photon energy and bring the surface carrier states toward the quantum limit [33].

In this article, we describe results of THz Faraday and Kerr rotation spectroscopy measurements on thin films of $\text{Bi}_{1-x}\text{Sb}_{x}$. This alloy system is known to show a semimetal (SM)-to-TI transition as a function of $x$, and its phase diagram (Fig. 1) is well established [34, 35, 36, 10, 13], although some controversy remains as to whether surface states in the SM region are topologically nontrivial [37]. We performed measurements using a single-shot THz time-domain polarimetry setup combined with a 30-T pulsed magnet system [38, 39, 40]. The wide tunability of the band structure of $\text{Bi}_{1-x}\text{Sb}_{x}$ with $x$ makes this material system suitable for identifying and distinguishing the uniquely different magneto-optical responses of samples in the topologically trivial and nontrivial phases.

A THz beam normally incident on the sample surface in the Faraday geometry exhibited polarization rotations due to the field-induced Hall conductivities of the surface and/or bulk carriers. Faraday rotation spectra for the SM films had a pronounced dip, which blue-shifted with the magnetic field, while the Faraday rotations in the TI films were positive and spectrally featureless, increasing and then saturating with increasing magnetic field. Using a theoretical model incorporating realistic band parameters and the Kubo formula for calculating the optical conductivity, we found that the optical Hall signals in the SM (TI) samples can be attributed to carriers in the bulk (surface) bands. The model suggested that the magneto-optical signal from the SM films was dominated by the cyclotron resonance of bulk high-mobility holes, while that from the TI film resulted from the summed contributions of multiple electron and hole pockets associated with the surface bands.

II Samples and Methods

II.1 $\text{Bi}_{1-x}\text{Sb}_{x}$ films

We studied $\text{Bi}_{1-x}\text{Sb}_{x}$ films on silicon substrates grown by molecular beam epitaxy using the methods described in Refs. 14, 41. Five samples with different $x$ values were studied; see Table 1. The nominal $x$ values were 0, 0.04, 0.08, 0.1, and 0.15, respectively, while the thickness, $t$, was nominally 40 nm for all films. According to Fig. 1, Samples 1 and 2 were in the SM regime while Samples 3–5 were in the TI regime. We used a combination of structural and chemical characterization methods to precisely determine the actual values of $x$, $t$, and crystal orientation of the films; in-situ reflection high-energy electron diffraction (RHEED) patterns determined the crystal orientation, while ex-situ x-ray diffraction (XRD), x-ray fluorescence (XRF), and atomic force microscopy (AFM) experiments provided information on the crystal structure, chemical composition, and film thickness, respectively. The obtained parameters of the samples are summarized in Table. 1.

For Sample 1 ($x=0$), RHEED determined that the film orientation was $\left<001\right>$. No Sb was incorporated in this sample, so that chemical analysis was not needed. AFM determined that $t$ = 68 nm. For Sample 2 (nominal $x$ = 0.04), XRF and XRD measurements were performed. An obtained XRF spectrum was fit with a model built in the measurement software, using $x$ and $t$ as adjustable parameters, and the parameters that gave the best fit were $x=0.03$ and $t$ = 60 nm. XRD showed a dominating diffraction peak due to the (001) plane of the film, confirming that the crystal orientation was $\left<001\right>$.

For Sample 4 (nominal $x$ = 0.10), fitting analysis on an XRF spectrum allowed us to determine $x=0.136$ and $t$ = 54 nm. An XRD curve showed diffraction peaks from the (001) and (012) planes, suggesting that the film possibly had some spatial inhomogeneity in terms of orientation; however, the ARPES data shown in Fig. 2(a) confirmed that the film area on which our magneto-optical measurements were performed was dominated by the $\left<001\right>$ orientation. Transport measurements were also performed on the film; see Fig. 2(b) for the resistance-temperature ($R$-$T$) characteristic. The increasing $R$ with decreasing $T$ in the $120$ K $<T<$ $250$ K region can be attributed to the decreasing number of thermal carriers in the insulating bulk, while the subsequent decrease of $R$ in the $T<120$ K region is likely due to surface metallicity; such behavior has been observed in $R$-$T$ curves measured for TI systems with minimal bulk doping [24, 25].

For Samples 3 (nominal $x$ = 0.08) and 5 (nominal $x$ = 0.15), we found that the uncertainties given by model fits to their XRF spectra were too large to determine both $x$ and $t$ accurately. However, as the amount of Sb incorporation was progressively increased in growing the five films, from Sample 1 to Sample 5, it is certain that Sample 3 had an $x$ value that is between those of Samples 2 and 4 while Sample 5 had $x>0.136$.

II.2 THz polarimetry

Figure 3 schematically shows the THz polarimetry technique we used to probe the magneto-optical response of the $\text{Bi}_{1-x}\text{Sb}_{x}$ films; similar techniques have previously been used to study systems other than TIs [42, 43, 44]. Our laser system was a Ti:sapphire regenerative amplifier (1 kHz, 150 fs, 775 nm, Clark-MXR, inc.), which generated and detected THz probe pulses with ZnTe crystals. The incident THz beam was linearly polarized in the $x$ direction, but both the $x$ and $y$ components of the electric field ($E_{x}$ and $E_{y}$) of the transmitted field were measured by electro-optic sampling in the time domain to quantify the THz polarization state that was affected by the carrier Hall effect. Because the THz waveforms were both amplitude- and phase-resolved, we were able to detect polarization rotations as small as 1 mrad as well as the corresponding ellipticity change. In addition, the pulse-based technique allowed us to measure both Faraday and Kerr rotations using only the transmission geometry.

As shown in the upper right schematic in Fig. 3, the THz pulse that directly passes through the film and the substrate gives the Faraday rotation. However, a portion of the pulse experiences additional reflection events at the vacuum-substrate interface and the film-vacuum interface, and can still be measured as a back reflection pulse in the same alignment geometry; the back reflection pulse appears later in time than the main pulse that is directly transmitted due to its additional path inside the substrate, but it contains both Faraday and Kerr rotation signals. The Kerr rotation signal can be isolated out by subtracting the Faraday rotation from the total polarization rotation of the back reflection pulse.

Below, we show the process of determining the polarization state of a pulse. We start with measurements of the $E_{x}(t)$ and $E_{y}(t)$ of a pulse in the time domain. Since the $E_{y}(t)$ signal is usually much smaller than $E_{x}(t)$, we changed the polarity of the magnetic field $B$ and took the average of the subtraction $[E_{y}(+B)-E_{y}(-B)]/2$ as the real $E_{y}(t)$ signal that excludes any effect of imperfect linear polarization of the incident beam. We Fourier-transformed $E_{x}(t)$ and $E_{y}(t)$ into complex-valued $E_{x}(\omega)$ and $E_{y}(\omega)$, which are functions of frequency $\omega$. We then defined the modes in the circular polarization basis,

[TABLE]

Then, the rotation of the polarization plane and change of ellipticity can be quantified, respectively, as

[TABLE]

In addition, the real and imaginary parts of the complex-valued longitudinal and Hall conductivity of the film can be determined if a reference pulse signal $E_{\text{ref}}$ is collected as a THz pulse passed through the bare substrate without the film. We first calculated the complex-valued conductivity in the circular basis

[TABLE]

where $n_{\text{Si}}=3.5$ is the refractive index of the silicon substrate, $Z_{0}=377\ \Omega$ is the vacuum impedance, $d$ is the thickness of the film, and $t_{\pm}=E_{\pm}(\omega)/E_{\text{ref,}\pm}$. Then, the longitudinal and Hall conductivities of the film can be obtained, respectively, as

[TABLE]

Just like the complex-valued longitudinal and Hall conductivity ($\sigma_{xx}$ and $\sigma_{xy}$), the combination of the polarization rotation angle and ellipticity change ($\theta$ and $\eta$) contains complete information on the THz magneto-optical response of the film, with the only difference that a reference signal is not needed in determining $\theta$ and $\eta$.

Electronic properties of a sample, irrespective of whether it is a SM or TI, contain a mixture of contributions from electrons and holes. Sometimes one carrier type can be dominant, and determining the carrier type gives valuable information on the origin of the optical conductivity signal. We can use the sign of the Hall conductivity to determine whether the dominant carriers are electrons or holes. We performed THz transmission measurements on a standard GaAs two-dimensional electron gas (2DEG) sample at 3 T, where the electron cyclotron resonance peak appeared in the middle of the THz spectrum, and calculated its optical Hall conductivity. As shown in Fig. 4, both the real and imaginary parts of the Hall conductivity are positive at the cyclotron resonance peak. We used this sign as a reference to determine the dominant carrier type in the $\text{Bi}_{1-x}\text{Sb}_{x}$ samples.

II.3 Single-shot THz spectroscopy in pulsed high magnetic fields

In addition to using a standard delay-stage-based step-scan THz setup combined with a 10-T superconducting magnet [45, 46, 47, 48, 49, 50, 51, 52], we used a unique single-shot THz setup to perform measurements in pulsed magnetic fields up to 30 T [40]. Pulsed magnets typically generate higher magnetic fields with a smaller magnet and cryostat size than DC magnets, but the challenge is that the peak magnetic field only lasts for a short time (approximately 400 $\mu$s in our case [38]), so that the time for a delay stage to step through the THz waveform is insufficient. We developed a single-shot THz time-domain detector, which is capable of measuring the full THz time-domain waveform using just one laser pulse within the 400-$\mu$s-long time window during which the sample is experiencing the peak of the pulsed magnetic field.

Single-shot THz detection relied on a reflective echelon mirror tilting the pulse front of the optical gate beam by forming time-delayed beamlets, which encoded time delay information across the beam intensity profile [53, 54, 55, 56]; see the inset in Fig. 5. The temporal and spatial overlap of the gate beam and the THz beam was achieved at the electro-optical crystal. A combination of a quarter-wave plate, a Wollaston prism, and a silicon complementary metal-oxide-semiconductor (CMOS) camera was used to detect the THz-induced change of the gate beam polarization. The CMOS camera captured the full image of the gate beam, which contained information about the THz electric field at various time delays. The difference between the two images spatially separated on the CMOS camera by the Wollaston prism gave the time-domain THz electric field signal.

III Results

III.1 Semimetallic samples: Samples 1 and 2

Figures 6(a) and (b) display Faraday rotation ($\theta_{\mathrm{F}}$) and Faraday ellipticity ($\eta_{\mathrm{F}}$) spectra, respectively, for Sample 1 ($x$ = 0) at $T$ = 2 K in $B$ up to 10 T. The curves are intentionally offset vertically for clarity, and the zero baselines are shown as dashed colored lines. A resonance feature that shifts higher in frequency with increasing $B$ is clearly observed. We then calculated the real and imaginary parts of the optical Hall conductivity ($\sigma_{xy}$) of the sample by taking into account the reference signal; see Eq. (6). Figure 7 shows the calculated $\sigma_{xy}$ spectra. The signs of the real and imaginary parts of $\sigma_{xy}$ are both opposite to that obtained from the standard 2DEG sample shown in Fig. 4. This indicates that holes are the major contributors to the magneto-optical signal.

Figures 8(a) and (b) display $\theta_{\mathrm{F}}$ and $\eta_{\mathrm{F}}$ spectra, respectively, for Sample 2 at $T$ = 2 K in $B$ up to 10 T. The major resonance feature that shifts with $B$ in a similar manner to that in Sample 1 is observed, except that the linewidth is broader. This suggests that the major contributors to the magneto-optical signal have the same carrier origin as in the Bi film, and the Sb incorporation reduces the carrier mobility. Figures 9(a) and (b) show Kerr rotation ($\theta_{\mathrm{K}}$) and Kerr ellipticity ($\eta_{\mathrm{K}}$) spectra, respectively, obtained for the same sample under the same conditions as in Fig. 8. The resonance feature induces a drastic response in the $\theta_{\mathrm{K}}$ spectra; an abrupt $\pi$ phase shift of the rotation angle appears at the resonance frequency at each $B$. The $\eta_{\mathrm{K}}$ spectra, on the other hand, do not show a clear trend, likely due to an insufficient signal-to-noise ratio.

We also performed measurements on Sample 2 in $B$ up to 30 T at $T$ = 21 K, using the single-shot THz detection setup described in Section II.3. Figures 10(a) and (b) show, respectively, the extracted $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$ versus $B$ for a fixed THz frequency (0.7 THz). Resonance behavior is again clearly observed in both $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$. $\theta_{\mathrm{F}}$ shows a dip, while $\theta_{\mathrm{K}}$ shows a $\pi$ phase shift at the resonance magnetic field. At higher $B$, both quantities decrease in magnitude and tend to zero.

III.2 Topological insulator sample: Sample 4

Faraday rotation, Kerr rotation, Faraday ellipticity, and Kerr ellipticity spectra for Sample 4 (nominal $x$ = 0.1 at $T$ = 2 K in $B$ up to 10 T are shown in Figs. 11(a)-(d). No curve is intentionally offset. The Faraday and Kerr rotations are featureless as a function of frequency, but both clearly show an increasing trend as $B$ increases. $\theta_{\mathrm{F}}$ is smaller than $\theta_{\mathrm{K}}$ for a given $B$, but its signal-to-noise ratio is higher because the back reflection pulse from which $\theta_{\mathrm{K}}$ is derived has a smaller amplitude than the main pulse.

Neither $\eta_{\mathrm{F}}$ nor $\eta_{\mathrm{K}}$ shows any signal that can be clearly distinguished from the noise floor for all $B$, suggesting that the carriers in the TI film induce pure rotations without any ellipticity change in the THz probe light. This behavior can be directly observed in the time-domain THz waveform data, as shown in Fig. 11(e). The incident THz pulse is polarized in the $x$ direction. It is clear that the amplitude of the transmitted $E_{x}$ does not vary much between 1 T and 10 T; at both magnetic fields, the main pulse appears at 3.5 ps, together with a smaller back reflection pulse at 12.5 ps. However, the $E_{y}$ waveform at 10 T is much larger than that at 1 T, suggesting a much larger polarization rotation; this is consistent with Figs. 11(a) and (b). In addition, as marked by the vertical dashed lines at the pulse peaks, the phase of $E_{y}$ also matches well with $E_{x}$ for both the main and back reflection pulses. This suggests that both pulses remain linearly polarized. Only the polarization plane is rotated with respect to the incident beam, and the ellipticity does not change.

We further performed measurements in $B$ up to 30 T on the same sample at $T$ = 21 K. Figures 12(a) and (b) show, respectively, $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$ versus THz frequency and $B$. The data obtained using the 10-T magnet system (Fig. 11) is also included in Fig. 12, agreeing with the data obtained using the 30 T system. We extracted the evolutions of $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$ with $B$ for a fixed THz frequency (0.7 THz), and the results are plotted in Figs. 12(c) and (d), respectively. We observe that $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$ increase with increasing $B$ until 15 T, above which both quantities saturate with further increasing $B$.

III.3 All samples

Figure 13 displays $\theta_{\mathrm{F}}$ versus $B$ at $T$ = 2 K for a fixed THz frequency (0.7 THz) for all samples of $\text{Bi}_{1-x}\text{Sb}_{x}$ films. As $x$ increases, the system moves from the SM to the TI regime in the phase diagram (Fig. 1). We can see a clear trend of magneto-optical response accompanying this transition. From the Bi and $\text{Bi}_{0.96}\text{Sb}_{0.04}$ films in which a bulk hole cyclotron resonance appears as negatively valued dips within 10 T, to the $\text{Bi}_{0.9}\text{Sb}_{0.1}$ film where the low-mobility surface electrons lead to an increasing positive $\theta_{\mathrm{F}}$ with increasing $B$, the change of $\theta_{\mathrm{F}}$ curves with increasing $x$ is monotonic and smooth.

IV Discussion

IV.1 Theory of magneto-optical response of Bi1-xSbx in the semimetallic regime

In order to understand the THz magneto-optical response of Bi1-xSbx films in the SM regime, we developed a detailed theoretical model. We took into account the bulk bands of Bi, which allowed us to determine the origin of the experimentally observed magneto-optical signal for Sample 1. The reason for choosing the Bi sample instead of $\text{Bi}_{0.96}\text{Sb}_{0.04}$ for this analysis is because the Bi sample showed a much sharper resonance feature.

The bulk band structure we considered is schematically depicted in Fig. 14 [57, 58]. There is an indirect negative band gap between the valence band maximum at the T point and the conduction band minima at the three equivalent L points (later we refer to these as a single point, L). For a $(001)$ Bi film, the hole pocket at the T point has an isotropic in-plane effective mass and a parabolic dispersion relation, while the bands at the L point host Dirac electrons with a hyperbolic dispersion, but a small gap $2\Delta$ exists at the L point.

The Hamiltonian for the T-point holes and the L-point electrons are

[TABLE]

where $E_{h}^{0}=38.5\ \text{meV}$ is the T-point band edge offset, $k$ represents the wave vector, $M_{c}=0.0677m_{0}$ and $M_{z}=0.721m_{0}$ are, respectively, the in-plane and out-of-plane hole effective masses, $m_{0}$ is the free electron mass, $\Delta=7.65\ \text{meV}$ is half of the L-point gap, and $v$ and $v_{z}$ are the in-plane and out-of-plane electron Dirac velocities, respectively. Because our film has a finite thickness, we considered the quantum confinement effect on $k_{z}$ along the growth direction for both the hole and electron pockets. As schematically shown in Fig. 14, many hole subbands described by discrete $k_{z}$’s with quantum number $N_{h}$ are above the Fermi energy $E_{\text{F}}$, while only two electron subbands (described by quantum number $N_{e}$) are filled at the L point; the electrons are more strongly influenced by quantum confinement than the holes because of the lighter electron mass.

The key for the magneto-optical response of carriers is to calculate the optical conductivity tensor given by the Kubo formula:

[TABLE]

where $\alpha$ and $\beta$ take choices between $x$ and $y$, $S$ is the sample area, $f_{m}$ ($f_{n}$), $E_{m}$ ($E_{n}$), and $|\Psi_{m}\rangle$ ($|\Psi_{n}\rangle$) are, respectively, the occupation factor calculated by the Fermi-Dirac distribution function, energy, and eigenfunction of the $m$th ($n$th) eigenstate of the system Hamiltonian, $\hat{j}_{\alpha}$ and $\hat{j}_{\beta}$ are current operators, and $\gamma$ is the scattering rate responsible for transition line broadening. We calculated the conductivity tensors contributed by each pocket and later added all their contributions.

In a magnetic field, the Landau-level eigenenergies for the hole and electron pockets are

[TABLE]

where $s_{h}=\pm 1/2$ and $s_{e}=\pm 1/2$ are, respectively, the hole and electron spin quantum number, $n$ represents the Landau-level index, $g=62.6$ is the hole $g$ factor, $\mu_{\text{B}}$ is the Bohr magneton, $m_{c}$ and $m_{z}$ are, respectively, the in-plane and out-of-plane electron effective masses. The calculated spin-resolved Landau-level energies for electrons and holes are plotted in Fig. 15(a) and (b), respectively. The results are displayed up to $N_{e}=2$ and $n=2$ ($N_{h}=10$ and $n=2$) for electrons (holes), where $N_{e}$ and $N_{h}$ are the electron and hole subband index, respectively.

Then we calculated the matrix elements of the current operators, $\langle\Psi_{m}|\hat{j}_{\alpha}|\Psi_{n}\rangle$, for both electron and hole pockets. For the electron pocket, the situation is complicated because the eigenspinor of the Dirac Hamiltonian above takes separate forms for spin-up and spin-down electrons

[TABLE]

where $\downarrow$ and $\uparrow$ denote electron spin, and $A_{n,\downarrow}$, $B_{n,\downarrow}$, $C_{n,\downarrow}$, $D_{n,\downarrow}$, $A_{n,\uparrow}$, $B_{n,\uparrow}$, $C_{n,\uparrow}$, and $D_{n,\uparrow}$ are the coefficients to be determined by the eigenvalue equation. Therefore, there are eight matrix elements of the current operator describing possible transitions between Landau levels across different spin channels. These are $\langle\Psi_{m,\downarrow}|\hat{j}_{x}|\Psi_{n,\downarrow}\rangle$, $\langle\Psi_{m,\uparrow}|\hat{j}_{x}|\Psi_{n,\uparrow}\rangle$, $\langle\Psi_{m,\downarrow}|\hat{j}_{x}|\Psi_{n,\uparrow}\rangle$, $\langle\Psi_{m,\uparrow}|\hat{j}_{x}|\Psi_{n,\downarrow}\rangle$, $\langle\Psi_{m,\downarrow}|\hat{j}_{y}|\Psi_{n,\downarrow}\rangle$, $\langle\Psi_{m,\uparrow}|\hat{j}_{y}|\Psi_{n,\uparrow}\rangle$, $\langle\Psi_{m,\downarrow}|\hat{j}_{y}|\Psi_{n,\uparrow}\rangle$, and $\langle\Psi_{m,\uparrow}|\hat{j}_{y}|\Psi_{n,\downarrow}\rangle$.

We calculated optical conductivity tensors by plugging the matrix elements of the current operator above into the Kubo formula. The total conductivity of each carrier pocket was obtained by summing up transitions between all possible states. Each state has five indices, i.e., the Landau-level index, spin index, band index describing valence or conduction band, subband index, and the pocket index due to the existence of multiple equivalent pockets. As an example, we show the transitions considered for the hole pocket within the $N_{h}$th subband and the electron pocket within the $N_{e}$th subband in Fig. 16. The hole pocket contains only the spin-conserving inter-Landau-level transitions. The electron pocket contains intraband inter-Landau-level transitions and interband magneto-optical transitions across different spin channels and Landau-level indices. Note that intersubband transitions are not allowed in either the hole or electron pocket due to the polarization selection rule.

Finally, we summed up the conductivities of the electron and hole pockets to obtain the total optical conductivity tensor of the Bi film. We adjusted three parameters to try to match the theoretically calculated magneto-optical response with the experiments. They are the Fermi energy $E_{\text{F}}$, the scattering rate of holes at the T point $\gamma^{\text{T}}_{h}$, and the scattering rate of electrons at the L point $\gamma^{\text{L}}_{e}$. The optimized parameters that achieve the best agreement between theory and experiment are $E_{\text{\text{F}}}=24\ \text{meV}$, $\gamma^{\text{T}}_{h}=0.75\ \text{meV}$, and $\gamma^{\text{L}}_{e}=0.9\ \text{meV}$. The calculated $\theta_{\mathrm{F}}$ and $\eta_{\mathrm{F}}$ spectra for the Bi film sample at 2 K up to 10 T are shown in Fig. 17; the curves show good agreement with the experimental data in Fig. 6.

We provide some comments on the optimized parameters. First, the extracted Fermi energy $E_{\text{F}}=24\ \text{meV}$ is lower than the value (28 meV) used in previous studies [57, 58]. The reason can be that the bands are modified compared to the true bulk Bi due to the band quantizations resulting from the finite thickness of the film. Second, $\gamma^{\text{T}}_{h}=0.75\ \text{meV}$ and $\gamma^{\text{L}}_{e}=0.9\ \text{meV}$ are both much smaller than the bandwidth of our THz setup. This therefore allows cyclotron resonance signals to be observed. From the calculated results in Fig. 17, we found that the dip feature that moves to higher frequency with increasing magnetic field is mainly due to hole cyclotron resonance. The electron cyclotron resonance feature, on the other hand, appears very close in frequency with the hole cyclotron resonance feature, but it is inhomogeneously broadened because cyclotron transition frequencies are different for different electron subbands. The effect of electron cyclotron resonance can still be identified in the calculated results in Fig. 17. We plotted the $\theta_{\mathrm{F}}$ and $\eta_{\mathrm{F}}$ spectra calculated by only taking into account the hole pocket in Fig. 18. It can be easily seen that the hole pocket alone cannot explain the asymmetric lineshapes of $\theta_{\mathrm{F}}$ and $\eta_{\mathrm{F}}$ in Fig. 17 and in the experimental data in Fig. 6.

IV.2 Theory of magneto-optical response of Bi1-xSbx in the topological insulator regime

We used a surface band model to analyze the experimental observations made in Sample 4, a TI sample [36, 17]; see Fig. 19. The $(001)$ surface of Bi0.9Sb0.1 has two spin-polarized surface bands, $S_{1}$ and $S_{2}$. The dispersion relation along the $\bar{\Gamma}$-$\bar{\text{M}}$ line has the following features. The bottom of the electron pocket formed by the $S_{1}$ band is located at the $\bar{\Gamma}$ point; the band has a linear dispersion, and its spin texture is similar to that of typical TI surface states with spin-momentum locking. Away from the $\bar{\Gamma}$ point, the $S_{1}$ band bends up and then down to intersect with the Fermi surface again, forming an anisotropic electron pocket near the $\bar{\text{M}}$ point (but not enclosing it); later we refer to it as the $\bar{\text{M}}$-point electron pocket. On the other hand, the $S_{2}$ surface band forms an anisotropic hole pocket in the middle of the $\bar{\Gamma}$-$\bar{\text{M}}$ line. Both the $\bar{\text{M}}$-point electron pocket and the hole pocket have six replicas in the first Brillouin zone, which are related to each other by a six-fold rotational symmetry with respect to the $\bar{\Gamma}$ point. We denote the $\bar{\text{M}}$-point electron pocket and the hole point along the $\bar{\Gamma}$-$\bar{\text{M}}$ line to be $e1$ and $h1$, respectively, and their replicas as $e2\cdots,e6$, and $h2\cdots,h6$.

The model Hamiltonians for the $\bar{\Gamma}$-point electron pocket, the $h1$ hole pocket, and the $e1$ electron pocket are

[TABLE]

where $v_{\text{D}}=2600\ \text{meV\AA}$ is the Dirac velocity, $p$ represents the carrier momentum, $E^{0}_{h}=170\ \text{meV}$ and $E^{0}_{e}=129.4\ \text{meV}$ are the band-edge energies of the hole pocket and the $\bar{\text{M}}$-point electron pocket, respectively, measured from the Dirac point, $p_{h0}=\hbar\times 0.25\ \text{meV/\AA}$ and $p_{e0}=\hbar\times 0.67\ \text{meV/\AA}$ are the displacements of band centers of the $h1$ and $e1$ pockets, respectively, and in-plane anisotropic effective masses can be described as $m^{h}_{x}=1.2m_{0}$, $m^{h}_{y}=0.2m_{0}$, $m^{e}_{x}=1.5m_{0}$, and $m^{e}_{y}=0.05m_{0}$, where $m_{0}$ is the free electron mass. The surface band dispersions calculated by applying these parameters are shown in Fig. 19(a). The Fermi surfaces of the pockets within one quadrant of the first Brillouin zone are shown in Fig. 19(b); the Fermi level is obtained from a fit to the Fermi surface of the $\bar{\Gamma}$-point electron pocket measured by ARPES, as shown in Fig. 2(a).

First, in a magnetic field, the Hamiltonian for the $\bar{\Gamma}$-point electron pocket is given by

[TABLE]

where $l_{B}=\sqrt{\hbar/|e|B}$ is the magnetic length, and $\hat{a}$ ($\hat{a}^{\dagger}$) is the Landau-level raising (lowering) operator. Its eigenstate spinor takes the form

[TABLE]

where $n$ is the Landau-level index, and $A_{n}$ and $B_{n}$ are coefficients to be determined by the following eigenvalue problem:

[TABLE]

The Landau-level eigenenergies are obtained as

[TABLE]

See Fig. 20 for the calculated eigenenergies versus magnetic field up to the $n=10$th Landau level. The corresponding coefficients are found to be

[TABLE]

Given that the current operator is given by

[TABLE]

its matrix elements can be calculated as

[TABLE]

where $\delta_{m-1,n}$ and $\delta_{m,n-1}$ are Kronecker’s $\delta$. By plugging the matrix elements above into the Kubo formula, we obtained the longitudinal and Hall conductivities for the $\bar{\Gamma}$-point electron pocket

[TABLE]

where $\gamma^{\bar{\Gamma}}_{e}$ represents the scattering rate of electrons in the $\bar{\Gamma}$-point electron pocket.

Second, the Hamiltonian for the hole pocket in a magnetic field is

[TABLE]

where $\omega^{h}_{c}$ is the hole cyclotron frequency. The form is simple because the pocket can be assumed to have a parabolic band dispersion. Landau-level energies depend linearly on the magnetic field; see Fig. 20.

Regarding the treatment of effective mass anisotropy for the hole pocket, it has been found that the system can be transformed into an isotropic model, which is easier for calculation, by performing a scaling procedure on the spatial metric [57]. We scaled the current operator as

[TABLE]

where $\eta=(m^{h}_{x}/m^{h}_{y})^{1/4}$ is the scaling factor. Then the matrix elements of $\hat{\tilde{j}}_{\alpha}$ can be written as

[TABLE]

where $M=\sqrt{m^{h}_{x}m^{h}_{y}}$. Substituting the matrix elements above into the Kubo formula gives the scaled conductivity tensor

[TABLE]

where $\gamma_{h}$ represents the scattering rate of the hole pocket. The actual conductivity tensor is related to the scaled conductivity tensor by

[TABLE]

The optical conductivity tensor for the rest of hole pockets $h2,\ h3,\ \cdots,\ h6$ can be obtained by six-fold rotations. The total conductivity as a sum of contributions from all hole pockets is obtained as

[TABLE]

Finally, the procedure for calculating the conductivity tensor for the $\bar{\text{M}}$-point electron pocket, $\sigma^{(e,\bar{\text{M}})}_{xx}$ and $\sigma^{(e,\bar{\text{M}})}_{xy}$, is similar to that of the hole pocket.

We summed up the contributions from all pockets to obtain the total conductivity tensor elements, $\sigma^{\text{tot}}_{xx}$ and $\sigma^{\text{tot}}_{xy}$, and calculated the Faraday and Kerr rotations following the process discussed in Section II.2. All band parameters are either given in the literature or can be obtained through fits to predetermined band structures. The three free parameters we can tune to match the theoretical results with experimental data are the $\bar{\Gamma}$-point electron scattering rate $\gamma^{\bar{\Gamma}}_{e}$, the hole scattering rate $\gamma_{h}$, and the $\bar{\text{M}}$-point electron scattering rate $\gamma^{\bar{\text{M}}}_{e}$. We found that $\gamma^{\bar{\Gamma}}_{e}=10\ \text{meV}$, $\gamma_{h}=60\ \text{meV}$, and $\gamma^{\bar{\text{M}}}_{e}=10.5\ \text{meV}$ give the best fit with the experimental data, as shown by the polarization rotation curves at 0.7 THz in Figs. 12(c) and (d). Maps of calculated Faraday and Kerr rotations as a function of THz frequency and magnetic field using the optimized parameters are shown in Fig. 21. These results can be compared to the experimental maps shown in Figs. 12(a) and (b), and there is agreement between the experimental and theoretical results.

The three optimized scattering rates, $\gamma^{\bar{\Gamma}}_{e}$, $\gamma_{h}$, and $\gamma^{\bar{\text{M}}}_{e}$, are all larger than the bandwidth of our THz probe, suggesting that the surface carriers do not have high enough mobility for their cyclotron resonance peaks to appear in our measurements. In addition, $\gamma_{h}$ is much larger than $\gamma^{\bar{\Gamma}}_{e}$ and $\gamma^{\bar{\text{M}}}_{e}$. This observation is not surprising as we examine the linewidths of surface bands measured by ARPES in previous studies [17], but its effect is that the contribution of surface holes in the THz polarization rotation signal is negligibly small compared to that of the surface electrons.

We provide some additional comments on the saturation behavior of Faraday and Kerr rotations in the $B>15\ \text{T}$ region in Figs. 12(c) and (d). It might be tempting to explain these features as the quantized optical Hall effect observed in several recent studies on other TI systems [31, 44, 32]. However, as shown in the calculation of $\theta_{\mathrm{F}}$ and $\theta_{\mathrm{K}}$ in a much wider magnetic field range in Fig. 22, we found that the saturation behavior observed within the $30$ T magnetic field range arises from the summation of the broadened cyclotron resonance signals contributed by the two electron pockets. Theoretical calculations predict a major quantum Hall plateau in the $60\ \text{T}<B<110\ \text{T}$ range as the filling factors of the two electron pockets are both small, but experimental observation might still be challenging because the short carrier localization length in optical experiments (compared to DC experiments) significantly shrinks the quantum Hall plateaus [59]. When $B>150\ \text{T}$, the filling factors of the carrier pockets with parabolic dispersions tend to zero, but the Dirac electron pocket at the $\bar{\Gamma}$ point gives a finite signal due to the nontrivial Berry’s phase created by electrons circling around in momentum space.

V Summary

In summary, we performed THz Faraday and Kerr rotation spectroscopy measurements on $\text{Bi}_{1-x}\text{Sb}_{x}$ thin films. This alloy system exhibits a semimetal-to-topological-insulator transition as a function of $x$. By using single-shot time-domain THz spectroscopy combined with a 30-T table-top pulsed magnet, we observed distinctly different behaviors between semimetallic ($x<0.07$) and topological insulator ($x>0.07$) samples. We were able to distinguish the origins of the magneto-optical responses of these films by comparing experimental data with predictions from our theoretical models. We found that a surface (bulk) band model including some material parameters established in previous studies can completely explain the THz Hall signal of all samples. The combined effort of the THz polarimetry experiments performed in high magnetic fields and the detailed theoretical analysis can be applied to other topological materials to investigate surface and bulk carrier contributions to the optical conductivity.

Acknowledgements.

This research was primarily supported by the National Science Foundation through the Center for Dynamics and Control of Materials: an NSF MRSEC under Cooperative Agreement No. DMR-1720595. I.K. acknowledges support from the Japan Society for the Promotion of Science (JSPS) through the Bilateral Joint Research Project. I.K. and J.T. thank the Ministry of Education, Culture, Sports, Science and Technology (MEXT)/JSPS for support through KAKENHI Grant Nos. 16H06010, 17H06124, and 18H04288. G.A.F. gratefully acknowledges support from a Simons Fellowship.