Transverse Mode Instability in High-Power Yb-Doped Double-Clad Fiber Amplifiers: A Three-Layer Optical–Thermal Analysis Based on Stimulated Thermal Rayleigh Scattering

TL;DR

This paper analyzes transverse mode instability in high-power fiber lasers using a detailed optical-thermal model to improve laser stability.

Contribution

A three-layer optical-thermal model is introduced to accurately predict transverse mode instability in Yb-doped fiber amplifiers.

Findings

The inner cladding affects optical and thermal mode structures, altering the STRS coupling.

The model enables accurate calculation of transverse eigenvalues relevant to TMI.

The formulation supports improved design of high-power fiber lasers with better modal stability.

Abstract

Transverse mode instability (TMI) in high-power ytterbium-doped double-clad fiber lasers is widely interpreted as being a consequence of a thermo-optic nonlinear phenomenon driven by stimulated thermal Rayleigh scattering. This work presents a coupled optical–thermal model for a continuous-wave forward-pumped (λp=976nm) fiber amplifier emitting at λs=1064nm over an optimal length of 12 m. The formulation explicitly resolves the three radial regions of a double-clad fiber, avoiding single-clad approximations. Modal fields are computed using the weakly guiding approximation (WGA) in the core combined with the semi-WGA at the cladding interfaces, enabling accurate calculation of higher-order modes of penetration into the inner cladding and of the transverse eigenvalues U01 and Umn relevant to TMI. Within this framework, the nonlinear stimulated thermal Rayleigh scattering coupling coefficient is evaluated, including gain saturation and the thermal eigenmodes of the multi-layer geometry. The results show that the inner cladding modifies both the optical and thermal mode structures, altering the optical–thermal overlap between LP01 and higher-order modes and changing the effective strength of STRS, directly influencing the predicted TMI threshold. The proposed formulation provides a quantitative and physically consistent tool for analyzing thermo–optic dynamics in Yb-double-clad fiber amplifiers and supports the design of next-generation high-power fiber lasers with improved modal stability.

Figures

Full text

1. Introduction

High-power ytterbium-doped double-clad fiber lasers (YDCFLs) have enabled continuous-wave operation well above the kilowatt level and are widely employed in industrial and scientific applications [1]. This architecture is recognized for its high efficiency, robustness, and power scalability, and it has become the leading choice for high-brightness systems [2,3,4,5,6]. Despite these advantages, YDCFLs are ultimately limited by thermally driven effects, among which transverse mode instability (TMI) remains the primary obstacle to further power scaling in large-mode-area (LMA) fibers. TMI manifests as an abrupt transfer of energy from the fundamental mode $[eqn]$ (the linearly polarized mode of radial order zero and azimuthal order one) to higher-order modes (HOMs), typically accompanied by kilohertz-scale oscillations in modal content [7,8,9,10,11].

A broad consensus that TMI is driven by the nonlinear mechanism of stimulated thermal Rayleigh scattering (STRS) has emerged in the literature. In this process, interference between guided modes generates a longitudinally modulated irradiance pattern, which, through quantum-defect heating, induces a dynamic refractive-index perturbation in the doped core [7,12,13,14,15,16]. Radial heat diffusion introduces a phase delay between the irradiance modulation and the corresponding refractive-index response, forming a traveling thermo-optic grating that enables coherent modal coupling and produces the characteristic threshold behavior of TMI. Recent refinements have incorporated gain saturation, improved thermal-source descriptions, and rigorous treatments of radial thermal modes [17].

In a substantial proportion of prior work, researchers modeled double-clad fibers (DCF) using an effective two-region approximation, an approach equivalent to treating the waveguide as a single-clad fiber (SCF) comprising only a core and a single cladding [13,18,19,20,21,22,23]. This approach has been instrumental in elucidating many aspects of STRS and TMI. However, it cannot fully reveal the role of the inner cladding, which modifies the transverse eigenvalue structure and may allow HOMs to penetrate beyond the core under high-power operation [11,24,25]. These effects alter optical overlap integrals and influence susceptibility to STRS-driven coupling.

More advanced modal descriptions rely on the weakly guiding approximation (WGA) within the core region, combined with semi-weakly guiding approximation (semi-WGA) boundary conditions to account for the refractive-index contrast at the core and inner- and outer-cladding interfaces [24,26,27,28]. These multi-layer formulations yield accurate propagation constants and field distributions, enabling realistic computation of optical–thermal overlap integrals and STRS coupling coefficients in double-clad fibers.

In this extended version of our previous conference work [29], we report the development of a quantitative model that describes the influence of STRS on TMI in double-clad fibers. This model combines WGA-based core solutions with semi-WGA boundary conditions at both cladding interfaces, together with thermal diffusion induced by the quantum defect and the STRS formalism governing coupling between $[eqn]$ and HOMs. This framework enables a physically consistent evaluation of how the multi-layer geometry modifies the optical–thermal modal structure and influences the TMI threshold. We then apply the model to analyze a high-power forward-pumped fiber amplifier operating at $[eqn]$ and emitting at $[eqn]$ in a continuous-wave (CW) steady-state regime over an active fiber length of 12 m. The system is based on a large-mode-area ytterbium-doped double-clad fiber (Yb-DCF), with a radial geometry defined by a core radius $[eqn]$ , an inner cladding radius $[eqn]$ , and an outer cladding radius $[eqn]$ . The insights provided in this analysis can aid in the design of next-generation high-power fiber lasers with improved modal stability.

2. Theoretical Model

Describing transverse mode instability in ytterbium-doped fiber amplifiers requires joint modeling of the guided optical fields, heat generation and diffusion in the core, and the thermo-optic coupling between modes induced by STRS. We explicitly consider a three-layer fiber—a core, inner cladding, and outer cladding—with refractive indices $[eqn]$ , respectively, a characteristic structure of double-clad fibers employed in high-power systems. This geometry, illustrated in Figure 1, captures the role of the inner cladding in modifying modal confinement and the thermal eigenmodes relevant to STRS while retaining the nearly uniform thermal diffusion properties of the silica regions. Although the thermal conductivities of the core and inner cladding are nearly identical, the presence of the intermediate layer plays an important role in the harmonic thermal response relevant to STRS. The additional boundary condition at $[eqn]$ modifies the thermal eigenvalues $[eqn]$ and therefore the phase-lagged temperature component that drives thermo-optic coupling. Thus, even though steady-state heat diffusion remains almost uniform in the glass regions, the multi-layer geometry has a significant impact on the dynamic thermal modes involved in TMI.

2.1. Modal Formulation in Three-Layer Optical Fibers

The modal analysis employed in this work follows the vector formulation reported by Tsao et al. [27], which describes electromagnetic propagation through the Debye potential and rigorously enforces the boundary conditions of the tangential components of the electric and magnetic fields at the interfaces $[eqn]$ and $[eqn]$ . This approach remains valid even when the refractive-index contrast between layers is high, a typical condition in double-clad fibers.

From Maxwell’s equations and the separation of variables in cylindrical coordinates, the normalized parameters can be obtained [24,26,27]:

$$[eqn]$$

where $[eqn]$ is the wavenumber in a vacuum. The parameters $[eqn]$ , $[eqn]$ , and $[eqn]$ characterize propagation in the core, penetration into the inner cladding, and evanescent decay in the outer cladding, respectively.

The propagation constant $[eqn]$ is obtained by numerically solving the characteristic (eigenvalue) equations arising from the enforcement of the electromagnetic boundary conditions at $[eqn]$ and $[eqn]$ . Once found, the resulting profiles may be treated in the LP form, provided that the weak-guidance approximation is applied only at the core–inner-cladding interface, where $[eqn]$ . At $[eqn]$ , the larger index contrast between $[eqn]$ and $[eqn]$ requires the use of the full boundary conditions of the vector model. This combination enables an accurate description of modal behavior throughout the structure, which is essential for evaluating the optical and thermal overlap terms involved in STRS.

These modal solutions also determine the transverse eigenvalues $[eqn]$ and $[eqn]$ , which are entered into the asymptotic TMI-threshold model (Equation (31)). Because STRS depends on the spatial interference between modes and on the projection of the thermal source onto the thermal eigenmodes, an accurate description of the field penetration into the inner cladding is essential for predicting the strength of thermo-optic coupling.

The resulting fields are then normalized to generate the radial profiles $[eqn]$ used in the optical and thermal integrals. The normalization follows the procedure adopted by Dong [13] over the entire cross-section $[eqn]$ :

$$[eqn]$$ $$[eqn]$$

With this normalization, the scalar field used in the integrals of interference, heat generation, and thermo-optic coupling is expressed as

$$[eqn]$$

where c is the speed of light in a vacuum, $[eqn]$ is the permittivity of free space, and $[eqn]$ is the effective refractive index of the LP_mn_ mode.

2.2. Transverse Fields in the Three-Layer Structure (DCF)

Determining the $[eqn]$ modes requires computing the normalized parameters $[eqn]$ , $[eqn]$ , and $[eqn]$ obtained from the same characteristic (dispersion) equations that ensure continuity of the tangential components of the electric and magnetic fields at the interfaces $[eqn]$ and $[eqn]$ .

At $[eqn]$ , due to the small index contrast $[eqn]$ , the weak-guidance approximation remains valid. At the interface $[eqn]$ , however, the contrast $[eqn]$ is significant, making the WGA inadequate and justifying the use of the semi-weak guidance approximation, as described by Tan et al. [24]:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ denote the Bessel, Neumann, and modified Bessel functions, respectively. The coefficients A, $[eqn]$ , $[eqn]$ , and C are determined by enforcing continuity of the tangential components of the electromagnetic fields at the interfaces $[eqn]$ and $[eqn]$ , resulting in four coupled equations. Although the thermal conductivity of the silica layers is nearly identical, the presence of the inner cladding introduces an additional thermal interface in the harmonic heat equation. This modifies the thermal eigenvalues $[eqn]$ associated with STRS and slightly changes the radial shape of the thermal modes. As a result, the effective phase delay between irradiance modulation and temperature response differs from that of a two-layer SCF, contributing to the nonlinear coupling coefficient $[eqn]$ , even though the steady-state heat flow is nearly uniform within the silica regions.

One of the characteristic (dispersion) equations applied at the interface $[eqn]$ can be expressed as follows [24]:

$$[eqn]$$

This represents one of the continuity conditions that define the characteristic equation of the mode in the three-layer structure.

The presence of the inner cladding allows greater radial penetration of the fields and redistributes the optical power between the core and the inner cladding. This modifies the spatial overlap between $[eqn]$ and higher-order modes, as well as the overlap with the thermally induced index perturbation and the gain profile. As a consequence, the real part of the nonlinear coupling coefficient $[eqn]$ and the associated TMI threshold become functions of the full three-layer optical–thermal structure.

2.3. Modal Superposition and Interference Pattern

Considering the normalized modal fields $[eqn]$ , the total optical field within the core region $[eqn]$ can be written as [13]

$$[eqn]$$

where $[eqn]$ is the power carried by the $[eqn]$ mode, $[eqn]$ is its optical frequency, and $[eqn]$ is the corresponding propagation constant.

The interference between the fundamental mode $[eqn]$ and the $[eqn]$ modes yields the irradiance pattern

$$[eqn]$$

where

$$[eqn]$$

The term $[eqn]$ represents a “traveling irradiance grating” arising from modal beating. Its spatial period is determined by the modal beat length:

$$[eqn]$$

This space–time modulation of the optical irradiance is the fundamental driver of STRS [7,13]. Through quantum-defect heating, irradiance beating produces a spatially and temporally oscillating thermal source. The harmonic thermal response of the medium then generates a delayed temperature modulation, which induces a refractive-index perturbation capable of coherently coupling the optical modes. Thus, Equation (10) constitutes the starting point for the thermo-optic dynamics leading to TMI.

Smith & Smith’s Moving-Grating Hypothesis

Smith & Smith [7] demonstrated that a phase shift between the irradiance pattern and the induced thermo-optic grating is a necessary condition for coherent power transfer between modes. This phase shift arises whenever the irradiance modulation travels along the fiber due to a slight difference in optical frequency between the interfering modes.

By identifying that

$$[eqn]$$

we can express the axial velocity of the moving irradiance grating as follows [7]:

$$[eqn]$$

where $[eqn]$ and $[eqn]$ are the effective indices of the $[eqn]$ and $[eqn]$ modes, respectively. For $[eqn]$ , the condition $[eqn]$ results in a grating traveling in the positive z direction, while $[eqn]$ reverses the direction of motion.

In the thermal regime, heat diffusion introduces a characteristic delay in the medium’s thermo-optic response. Consequently, the temperature grating does not perfectly follow the moving optical pattern a phase lag. This broken symmetry leads to a nonreciprocal nonlinear coupling term between the interacting modes. Dong [13] formalized this mechanism by showing that modal interference generates a traveling optical intensity wave of the form $[eqn]$ , while the harmonic solution of the heat equation produces a temperature perturbation proportional to $[eqn]$ with a complex-valued amplitude, whose imaginary part encapsulates the diffusion-induced phase delay.

This phase-lagged thermal response is precisely the origin of the nonlinear coefficient $[eqn]$ responsible for the exponential growth of the higher-order mode above the instability threshold.

2.4. Heat Generation and Quantum Defect

The traveling interference pattern discussed in the previous section modulates the local heat deposition in the doped core. The heating rate is determined by the ytterbium ions’ Stokes shift, and the fraction converted into heat is quantified by the quantum defect [30,31]:

$$[eqn]$$

which is positive for Yb-doped fiber lasers operating at $[eqn]$ .

Interference between $[eqn]$ and $[eqn]$ produces an irradiance modulation with axial periodicity $[eqn]$ and temporal frequency $[eqn]$ . The harmonic thermal source is written as [13]

$$[eqn]$$

with radial amplitude

$$[eqn]$$

and $[eqn]$ for $[eqn]$ . Here, $[eqn]$ denotes the saturated effective gain that determines the fraction of absorbed pump power contributing to the harmonic component of the heating relevant to STRS, while $[eqn]$ and $[eqn]$ are the normalized modal profiles of the interfering modes.

This oscillatory heating term excites the radial thermal eigenmodes of the fiber, generating a temperature perturbation with the same axial and azimuthal dependencies. The delayed thermo-optic grating arising from this temperature response constitutes the fundamental physical mechanism of STRS-driven modal coupling [7,13,32].

2.5. Harmonic Heat Equation and Thermal Modes

The temperature perturbation $[eqn]$ generated by $[eqn]$ obeys the heat equation [7,13,32,33]:

$$[eqn]$$

where $[eqn]$ is the density, C is the specific heat, and $[eqn]$ is the thermal conductivity of the glass. Since $[eqn]$ contains the factor $[eqn]$ , we seek a time-harmonic steady-state solution with the same axial and azimuthal dependence:

$$[eqn]$$

where $[eqn]$ are the radial thermal modes of the three-layer structure.

These modes are obtained from the static heat equation subject to the boundary conditions described below, following the multi-layer thermal formulation widely adopted in high-power double-clad fiber amplifiers [34,35]. At the fiber axis $[eqn]$ , cylindrical symmetry is imposed:

$$[eqn]$$

At the material interfaces $[eqn]$ and $[eqn]$ , continuity of both temperature and heat flux is enforced:

$$[eqn]$$

where the subscript denotes the fiber region (1 for the core, 2 for the inner cladding, and 3 for the outer cladding), and $[eqn]$ is the corresponding thermal conductivity of each region. At the outer-cladding boundary $[eqn]$ , heat removal by the environment is modeled through a convective boundary condition,

$$[eqn]$$

where h is the effective convective heat-transfer coefficient and $[eqn]$ is the coolant temperature.

The corresponding eigenvalues $[eqn]$ incorporate the influence of the inner cladding on radial diffusion and govern the phase-lagged thermal response relevant to STRS.

Projecting the heat equation onto $[eqn]$ and using radial orthogonality yield

$$[eqn]$$

It is convenient to introduce the thermal damping parameter,

$$[eqn]$$

which makes the frequency dependence explicit in the coupling expressions that enter the nonlinear coefficient $[eqn]$ .

2.6. Thermo-Optic Perturbation

The temperature variation induces a refractive-index perturbation through the thermo-optic coefficient $[eqn]$ :

$$[eqn]$$

Since $[eqn]$ inherits the dependence $[eqn]$ , the index perturbation forms a traveling thermo-optic grating that mediates stimulated thermal Rayleigh scattering, providing the link between modal interference and coherent modal coupling.

2.7. Nonlinear Modal Coupling Coefficient

The thermo-optic perturbation $[eqn]$ induces coherent coupling between the $[eqn]$ and $[eqn]$ modes. This interaction is described by the complex nonlinear coefficient $[eqn]$ , which, for the three-layer DCF, takes the following form [13,15,31,33,36]:

$$[eqn]$$

The complex frequency response is

$$[eqn]$$

whose real part peaks at $[eqn]$ , while the imaginary part captures the diffusion-induced phase delay between the irradiance and thermal gratings. This phase asymmetry underlies the nonreciprocal character of STRS-driven coupling.

2.8. Coupled Equations for Power Evolution

In the quasi-stationary regime, the axial evolution of the modal powers is described by the coupled equations [13,17,31,33,37]:

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ are the saturated modal gains, and $[eqn]$ is the loss of the higher-order mode.

Under thermo-optic coupling, the HOM may experience net amplification over extended portions of the length of the fiber. When this occurs, the coupled equations predict an exponential growth in HOM power, which characterizes the onset of TMI.

2.9. Microscopic STRS Threshold and Connection with TMI

STRS provides the microscopic mechanism that triggers the coherent transfer of power between $[eqn]$ and higher-order modes [7,13,33]. The threshold occurs when the effective gain of the $[eqn]$ mode, driven by the real part of the coupling coefficient $[eqn]$ , becomes positive over a significant portion of the fiber [17].

In the high-gain regime, where the fundamental mode dominates the propagation and the initial HOM power is very small, integration of the coupled equations leads to a simple analytical expression for the threshold power [13]:

$$[eqn]$$

where $[eqn]$ represents the fraction of HOM power at the fiber output used to define the threshold (typically $[eqn]$ – $[eqn]$ ). In this regime, $[eqn]$ appears as the dominant parameter setting the threshold, while the ratio $[eqn]$ controls the sensitivity to the initial HOM fraction.

This expression shows that, in large-mode-area fibers, the threshold depends primarily on the thermo-optic efficiency encapsulated in $[eqn]$ , becoming weakly influenced by the total gain when the product $[eqn]$ is sufficiently large [13,33]. Under moderate-gain conditions, HOM losses and gain nonuniformities increase the threshold; here, such effects are incorporated directly through numerical integration of the propagation equations.

The STRS threshold therefore defines the microscopic scale at which the higher-order mode begins to grow exponentially due to thermal coupling. TMI emerges when this growth manifests macroscopically at the amplifier output, altering the beam profile and degrading its quality [21,32].

2.10. Analytical Estimate of the TMI Threshold

The macroscopic TMI threshold is reached when small transverse perturbations begin to grow exponentially and transfer power from the fundamental mode to HOMs. Among the available analytical approaches, the formulation of Zervas [38,39] provides a compact expression that relates the thermal threshold to the optical, thermal, and geometrical parameters of the fiber.

In the regime typical of high-power amplifiers, the reduced threshold expression can be written as follows [38,39]:

$$[eqn]$$

where $[eqn]$ is the thermal conductivity of silica; $[eqn]$ and $[eqn]$ are the transverse wavenumbers of the fundamental and coupled modes, respectively; and $[eqn]$ is the diameter of the core.

The quantity $[eqn]$ represents an effective loss of the fundamental mode, incorporating both intrinsic losses and the dissipative term associated with the quantum defect. According to the formulation developed by Zervas [38], this loss is written in terms of a saturated gain of the fundamental mode:

$$[eqn]$$

where $[eqn]$ is the background loss (including photodarkening), and $[eqn]$ is the quantum defect. In the context of modern STRS modeling, $[eqn]$ must be associated with the effective STRS gain introduced by Dong rather than the small-signal gain $[eqn]$ . Dong showed that the heating relevant to STRS can be written as follows [13,36]:

$$[eqn]$$

where $[eqn]$ is the effective STRS gain, which is given by

$$[eqn]$$

$[eqn]$ is the normalized inversion obtained from the rate equations, and $[eqn]$ is the local conversion efficiency for the signal. To reconcile the expression developed by Zervas with this formalism, in this work, we set

$$[eqn]$$

That is, $[eqn]$ is the effective STRS gain weighted by the modal profile of the fundamental mode in the core. The additional term $[eqn]$ in Equation (32) therefore reflects the fraction of the amplified power that is converted into heat by the combination of the quantum defect and the saturated effective gain that governs STRS and the onset of instability.

Equation (31) correctly captures the dependence of the TMI threshold on mode size, the thermo-optic response of the glass, and the thermal efficiency of the material. Although it poses quantitative limitations for more complex fiber geometries, it remains a useful design reference, enabling direct interpretation of how thermal and optical parameters influence the emergence of instability.

3. Results and Discussion

Numerical simulations of a Yb-DCF amplifier were performed in MATLAB (R2016a) using the geometrical, optical, thermal, and active-medium parameters summarized in Table 1. For comparison purposes, an SCF configuration was also considered using the same geometrical dimensions $[eqn]$ .

The small-signal pump absorption coefficient at $[eqn]$ follows directly from the pump propagation equation in the limit $[eqn]$ , $[eqn]$ [40,41], yielding $[eqn]$ for the parameters listed in Table 1. The pump power $[eqn]$ represents the net optical power coupled with the inner cladding at the fiber input end; insertion losses of pump diodes and coupling optics are therefore not explicitly modeled. The system analyzed in this work corresponds to a forward-pumped fiber amplifier, and no optical resonant cavity is involved. In the present forward-pumped amplifier configuration, no counter-propagating pump component is considered. The active fiber length is $[eqn]$ , and the amplifier operates at fixed pump and signal wavelengths ( $[eqn]$ , and $[eqn]$ ), so spectral scanning is outside the scope of the present continuous-wave (CW) model.

The model integrates modal propagation, quantum-defect heating, and the thermal–optic interaction described by STRS in a three-layer Yb-DCF under forward pumping.

This framework provides a self-consistent description of the axial evolution of modal power, the radial temperature distribution, and the nonlinear coupling between $[eqn]$ and higher-order modes. A central aspect of the analysis is the role of the inner cladding: its presence modifies the optical distribution of the modes and the thermal diffusion pathways, thereby altering the overlap integrals that govern STRS-induced modal coupling.

The following subsections examine how the multi-layer geometry affects the temperature field generated by the quantum defect, the axial dependence of the coupling coefficient $[eqn]$ , and the resulting modal power transfer. For completeness, the results are also compared with a single-clad fiber with an identical core radius and numerical aperture. This contrast highlights how the transverse architecture—through its influence on modal eigenvalues, confinement, and thermal redistribution—determines the conditions for the onset and evolution of transverse mode instability.

3.1. Pump and Signal Power Evolution

Figure 2 shows the axial evolution of the pump and signal power along the Yb-DCF that forms the active medium of the amplifier schematically illustrated in Figure 1. The pump is injected only at the fiber input end ( $[eqn]$ ) with the launched power $[eqn]$ defined in Table 1. The amplifier is seeded at $[eqn]$ with an input signal power $[eqn]$ , and the output signal power is given by the forward signal at the fiber end, $[eqn]$ .

As the pump is absorbed by the active medium, the forward pump component $[eqn]$ decreases monotonically along the fiber, while the forward signal component $[eqn]$ increases monotonically due to stimulated emission under gain-saturated conditions. In the absence of counter-propagating pump and signal waves, the resulting axial inversion profile is solely determined by the longitudinal evolution of the forward pump and signal power, as shown in Figure 3.

This axial inversion profile defines the spatial distribution of quantum-defect heating along the fiber, which constitutes the driving mechanism for STRS and the thermally induced modal coupling discussed in the following sections.

For the operating conditions considered here, the forward signal power increases from the injected seed level $[eqn]$ to an output value of $[eqn]$ at the fiber end. The corresponding optical efficiency of the forward-pumped fiber amplifier is defined as follows:

$$[eqn]$$

For a launched pump power of $[eqn]$ , this yields $[eqn]$ , which remains below the quantum-defect limit given by $[eqn]$ .

3.2. Modal Field Distribution in SCF and DCF

Figure 4 compares the radial distributions of the $[eqn]$ and $[eqn]$ modes in single-clad and double-clad fibers. In the SCF, both modes remain tightly confined to the core, as expected from a two-layer step-index structure in which no intermediate lower-index region is present. The absence of an inner cladding prevents significant field penetration beyond the core, resulting in higher modal concentration in the doped gain region.

In the DCF, the presence of the intermediate inner cladding modifies the modal structure, allowing penetration of the guided modes into the lower-index region. This redistribution spreads the optical power over a larger transverse area and reduces the fraction confined to the core. Such changes in modal localization are expected to influence both the effective refractive-index separations and the physical parameters that appear in threshold models for modal instability.

The quantitative impact of this modal restructuring on TMI is evaluated in the next subsection through the transverse eigenvalues and the corresponding instability thresholds.

3.3. Modal Eigenvalues and TMI Threshold

Identification of the dominant coupling channel is directly related to the transverse eigenvalues of the modes involved. For the double-clad fiber, the eigenvalues were computed using both the rigorous three-layer formulation developed by Tsao et al. [27] and the semi–weakly guiding model developed by Tan et al. [24]. These formalisms incorporate the influence of the inner cladding in different ways, resulting in distinct predictions for $[eqn]$ and $[eqn]$ . For the Tan semi–WGA formalism, the modal solutions were obtained under Condition 1 ( $[eqn]$ ) [24]. As discussed in Ref. [24], because $[eqn]$ in typical DCF structures, the core-guided Condition 2 ( $[eqn]$ ) can hardly be achieved, and the analytical development proceeds under Condition 1. In the same framework, WGA is applied only at the core–inner-cladding interface, whereas the inner–outer cladding interface retains the $[eqn]$ dependence (semi-WGA), since $[eqn]$ is not a valid assumption for most DCFs. For completeness, enforcing Condition 2 in our solver yields $[eqn]$ and $[eqn]$ , illustrating the expected SCF-like limit when $[eqn]$ and confirming that the conventional core-guided regime is recovered when required. Since the asymptotic threshold expression Equation (31) depends explicitly on these eigenvalues, the choice of modal model directly affects the predicted instability behavior.

For comparison, a single-clad fiber with the same core radius and numerical aperture was analyzed using the classical scalar step-index dispersion relation developed by Gloge [18]. The resulting eigenvalues $[eqn]$ and $[eqn]$ reflect the stronger optical confinement characteristic of a two-layer structure.

Table 2 summarizes the modal eigenvalues and the corresponding threshold power obtained from Equation (31), assuming $[eqn]$ nm, $[eqn]$ m, and $[eqn]$ .

The results reveal a clear ordering of instability thresholds. When we use the rigorous eigenvalues developed by Tsao et al. [27], the $[eqn]$ – $[eqn]$ channel yields the lowest threshold (≈1.05 kW), while the $[eqn]$ – $[eqn]$ pair plays a secondary role with a significantly higher threshold. Tan’s semi-WGA model predicts that $[eqn]$ [24], which is not compatible with the ordering assumptions implicit in Equation (31). Therefore, within Equation (31), the $[eqn]$ – $[eqn]$ channel is the only applicable contribution in this case, yielding a threshold of ≈1.59 kW.

Although the SCF exhibits stronger optical confinement, its predicted threshold (≈2.18 kW) is higher than that of the DCF. In the asymptotic framework of Equation (31), this reflects the combined influence of (i) the transverse optical eigenvalues $[eqn]$ and $[eqn]$ , which differ substantially between two- and three-layer structures, and (ii) the thermal eigenvalues $[eqn]$ associated with radial heat diffusion. For the present parameter regime, the contrast between SCF and DCF thresholds is dominated by the optical eigenvalues, while the thermal response provides a secondary yet consistent contribution to the overall STRS-driven coupling.

Overall, these comparisons demonstrate that the dominant TMI channel and the resulting threshold depend sensitively on both the modal formalism and the multi-layer waveguide geometry through their joint impact on the optical and thermal eigenvalue structure entered into Equation (31). Accurate prediction of TMI therefore requires internal consistency between the optical modal analysis, the thermal diffusion model, and the STRS-based asymptotic threshold.

3.4. Radial and Axial Temperature Profiles in the Yb-DCF

Figure 5 shows the thermal behavior along the active fiber (Yb-DCF) in a forward-pumped Yb-DCF amplifier operating in the CW steady state, with $[eqn]$ and a seed power $[eqn]$ . At the fiber input ( $[eqn]$ ), the pump power is maximum, and heat deposition is strongest, resulting in a core/center temperature of $[eqn]$ . As shown in Figure 5a, the temperature decreases monotonically with an increase in z due to pump absorption.

At the fiber output ( $[eqn]$ ), the remaining pump power is $[eqn]$ , while the forward signal reaches $[eqn]$ . The strong pump depletion and efficient optical conversion explain the progressive reduction in heat generation toward the fiber end, consistent with Figure 5a.

Table 3 lists representative axial core-center temperatures $[eqn]$ extracted from the curves shown in Figure 5a, allowing a direct quantitative comparison between the formulation in question and the results reported by Fan et al. [34]. The quantitative agreement along the fiber axis confirms the consistency of the thermal implementation, while the higher peak temperature in the present case reflects the higher pump power.

Beyond the quantitative agreement along the fiber axis, the variations observed in the radial and axial profiles further characterize the thermal regime associated with the higher pump power. As observed in Figure 5a, the axial temperature variation at $[eqn]$ increases from approximately 324 K in the 1.22 kW case (Fan et al.) to about 457 K in the present 2 kW regime. Similarly, from Figure 5b, the radial temperature difference between the core center and the cladding boundary at $[eqn]$ increases from approximately 20 K to about 38 K. These results indicate a rise in the thermal load in both the transverse and longitudinal directions, while the monotonic radial and axial behavior of the temperature distribution is maintained.

In silica-based double-clad fibers, the core and inner cladding exhibit similar thermal conductivities, whereas the outer polymer jacket has a significantly lower value [30,35]. Thus, within the glass region, the temperature profile is primarily determined by heat deposition in the doped core and by the thermal boundary condition at $[eqn]$ . The behavior observed in Figure 5 therefore follows the axial pump-absorption pattern rather than strong thermal contrasts between the silica layers. The predicted temperature levels and thermal gradients are consistent with experimentally reported values for high-power Yb-doped fiber amplifiers operating under comparable cooling conditions [34], supporting the physical consistency of the thermal model adopted in this work. With respect to TMI, the inner cladding affects the coupling mechanism through the thermal eigenvalues and modes associated with the three-layer geometry, a finding consistent with the formulation in the previous section. For the parameter regime considered here, however, the dominant contrast between SCF and DCF thresholds originates from the transverse optical eigenvalues $[eqn]$ and $[eqn]$ entered into Equation (31), while the thermal response provides the thermo-optic background that contributes to the STRS-driven coupling coefficient and the corresponding instability threshold.

3.5. STRS-Induced Modal Coupling

Figure 6 shows the axial evolution of the modal power $[eqn]$ and $[eqn]$ along the fiber length ( $[eqn]$ ). The curves were obtained by integrating the coupled differential equations, Equations (28) and (29), using the nonlinear thermo-optic coupling coefficient $[eqn]$ , which incorporates the radial thermal modes and eigenvalues.

From the input end up to approximately $[eqn]$ , the $[eqn]$ mode remains dominant and varies smoothly, whereas the $[eqn]$ mode, initially seeded at an extremely low noise floor, grows rapidly. In the simulations, a seed ratio of $[eqn]$ was adopted, consistent with the deterministic STRS simulations reported by Dong [13]. Smith and Smith [42] showed that higher-order modes are inevitably seeded by quantum noise (with approximately one photon per beat cycle) as well as by scattering at fiber imperfections and the pump or signal. More recent modeling studies have further indicated that amplitude noise of quantum or relative-intensity-noise origin provides a realistic physical source for the $[eqn]$ seed participating in STRS [43]. The adopted seed level therefore represents a physically realistic background perturbation that initiates the STRS process. Additionally, because the instability threshold depends only logarithmically on the seed ratio, variations of over several orders of magnitude produce only weak changes in the predicted threshold. The modal fraction,

$$[eqn]$$

therefore remains close to zero near the fiber input and increases significantly as z approaches $[eqn]$ .

As the total power increases, the thermo-optic modulation of the refractive index forms a traveling thermal grating that enhances modal coupling. The instability threshold may be estimated directly from Equation (30). Introducing the seed ratio $[eqn]$ transforms the expression into

$$[eqn]$$

where $[eqn]$ defines the fractional power criterion used to identify the onset of thermal instability.

For representative values $[eqn]$ and $[eqn]$ , consistent with the parameters of the case shown in Figure 6, Equation (38) yields $[eqn]$ . This value is in good agreement with the numerical threshold $[eqn]$ reported by Dong for the same amplifier configuration [13].

It is important to note that the threshold defined by Equation (30) does not correspond to the condition $[eqn]$ but rather to the point at which the modal fraction reaches $[eqn]$ , following Dong’s convention [13]. Accordingly, TMI is identified by the rapid rise in $[eqn]$ and the associated decrease in $[eqn]$ , even when $[eqn]$ still carries only a small fraction of the total power.

While Figure 6 illustrates the spatial build-up of the STRS-driven modal coupling along the fiber length between the $[eqn]$ and $[eqn]$ modes, Figure 7 represents its corresponding time-domain signature at the output of the amplifier. The apparent irregular power exchange between the fundamental mode $[eqn]$ (FM) and the higher-order mode $[eqn]$ (HOM) is therefore a direct consequence of the axial thermo-optic coupling mediated by the traveling thermal grating. Such temporal modal power fluctuations directly translate into output-beam instability over time, which constitutes a hallmark of TMI in high-power fiber amplifiers, in agreement with the experimental observations reported by Jauregui et al. [10].

Above the threshold, the STRS dynamics become nonlinear. Energy may flow intermittently back into the fundamental mode, giving rise to an oscillatory behavior along the fiber. This effect is consistent with the “re-seeding” mechanism described by Dong [13], in which the traveling thermal grating continuously modulates the coupling direction according to the evolving modal power ratio. The resulting sequence of oscillations observed in the simulation reflects this dynamic feedback process.

3.6. Modal Coupling and TMI Evolution

Figure 8 shows the axial evolution of modal power along the Yb-DCF, obtained from the complete amplifier model, assuming a seed power of $[eqn]$ at $[eqn]$ . The dashed curve corresponds to the reference case without modal coupling: the fundamental mode $[eqn]$ grows monotonically over the 12 m solely under optical gain, indicating purely single-mode operation.

When thermo-optic coupling induced by STRS is included, the dynamics change qualitatively. Part of the power is transferred from $[eqn]$ to a higher-order mode $[eqn]$ , as indicated by the coupled curves in Figure 8. This process is evidenced by the sharp drop in $[eqn]$ at $[eqn]$ , where the fundamental mode reaches $[eqn]$ . From this point on, $[eqn]$ grows rapidly and becomes the dominant mode.

Physically, this behavior corresponds to the onset of TMI: the thermal grating generated by quantum-defect heating produces a nonlinear coupling coefficient with sufficient magnitude and phase delay to invert the local gain balance between the modes. The initial noise component of $[eqn]$ is coherently amplified, and power is progressively transferred from $[eqn]$ to $[eqn]$ . The axial location of the threshold and the associated power level agree with the analytical prediction of Equation (30), reinforcing the interpretation of TMI as a thermo-optic instability driven by STRS.

The threshold behavior is qualitatively consistent with that reported for fiber amplifiers in [7]. In their model, the frequency-shifted interference between modes creates a moving irradiance pattern whose phase-lagged thermal response produces coherent modal coupling. The local small-signal gain $[eqn]$ of the higher-order mode is integrated to give $[eqn]$ , resulting in $[eqn]$ , with maximum gain occurring in the first half of the fiber (see Figures 7–11 of [7]).

In both models, the instability threshold appears as an abrupt modal transition triggered by thermally driven refractive-index modulation. In the present amplifier, this occurs at the point where the uncoupled $[eqn]$ solution would otherwise continue to grow toward its nominal output value of $[eqn]$ . Beyond the local threshold at $[eqn]$ , the higher-order mode becomes dominant, degrading beam quality and representing the macroscopic manifestation of TMI in continuous-wave high-power operation.

3.7. Modal Profiles, STRS-Driven Superposition, and Transverse Instability

Figure 9 shows the normalized transverse profiles of the $[eqn]$ and $[eqn]$ modes in the SCF. While $[eqn]$ exhibits a circularly symmetric central maximum, $[eqn]$ presents two characteristic lateral lobes separated by a nodal line. These transverse structures form the basis for the interference patterns that later drive thermal gratings through STRS.

These two-dimensional maps complement the radial profiles previously shown in Figure 4, providing the full transverse structure of the guided modes that participate in STRS and, ultimately, TMI.

Figure 10 shows the transverse normalized intensity pattern resulting from the superposition of $[eqn]$ and $[eqn]$ during TMI in the SCF. Unlike the isolated modes in Figure 9, the pattern here arises from a small $[eqn]$ component generated through the thermal–optic coupling induced by STRS. The strong optical confinement of both modes within the core produces high-contrast interference fringes, which modulate the heat generation associated with the quantum defect. This modulation reinforces the thermo-optic grating responsible for driving power transfer from $[eqn]$ to $[eqn]$ .

Figure 11 shows the modes $[eqn]$ and $[eqn]$ , again corresponding to the modes shown in Figure 4 in the DCF. Because the optical modes extend radially into the inner cladding, the interference pattern becomes broader and less concentrated near the fiber axis compared with the SCF. This spatial restructuring modifies the distribution of the thermo-optic source term associated with quantum-defect heating, altering the strength and shape of the resulting thermal grating. Although the thermal conductivities of the silica core and inner cladding are similar, the three-layer geometry changes the effective overlap between the optical interference pattern and the radial thermal modes. As a consequence, the STRS-induced coupling becomes more efficient in the DCF than in the SCF, consistent with the lower thresholds predicted from the transverse eigenvalues in the preceding sections.

Finally, Figure 12 shows the axial evolution of the transverse intensity patterns in the DCF during modal instability. These patterns result from the STRS-driven coherent superposition of the $[eqn]$ and $[eqn]$ electric fields. The periodic distortion of the intensity distribution, synchronized with the increase in $[eqn]$ , is a hallmark of STRS-driven modal coupling. The migration of optical power within the core region reflects the onset of TMI, providing a direct visualization of the underlying instability mechanism.

Taken together, these visualizations demonstrate that the onset and strength of STRS—and therefore robustness against TMI—are not governed solely by optical confinement. In the SCF, strong confinement produces high-contrast interference fringes, but the corresponding modal eigenvalues lead to a higher instability threshold. In the DCF, the three-layer geometry redistributes the modal fields and modifies their overlap with the thermal modes, resulting in a more effective STRS coupling mechanism and a lower TMI threshold. These observations are in agreement with the eigenvalue analysis and threshold predictions presented in the preceding sections.

3.8. Consistent Optical–Thermal Predictions of the TMI Threshold in SCF and DCF

The instability threshold was computed based on Equation (31) using the transverse eigenvalues for both the SCF and the DCF within the asymptotic STRS-based framework originally developed for high-power fiber amplifiers [38]. For the SCF, the eigenvalues obtained from the scalar step-index dispersion relation yield $[eqn]$ and $[eqn]$ , corresponding to strong modal confinement in the core. For the DCF, the dispersion relations developed by Tsao et al. [27] predict the values listed in Table 1, reflecting the modified modal structure resulting from the three-layer geometry.

Assuming $[eqn]$ m and $[eqn]$ nm, the resulting quantum defect is $[eqn]$ for a pump wavelength of $[eqn]$ nm. Under these operating conditions, the predicted instability thresholds are $[eqn]$ kW for the SCF and $[eqn]$ kW for the DCF. Although the SCF supports a more tightly confined optical mode, the DCF exhibits a lower threshold. This apparent contradiction highlights that, within the asymptotic STRS model for amplifiers, the TMI threshold is not only governed by modal confinement but also by the combined effect of quantum-defect heating and the transverse eigenvalue structure. In the present model, this behavior is explained by the different transverse eigenvalues of the two waveguides: the three-layer structure of the DCF yields a smaller modal factor $[eqn]$ in Equation (31), leading to a reduced asymptotic threshold by lowering the critical signal power required for STRS-driven mode coupling in the amplifier regime, despite its weaker optical confinement.

Figure 13 shows the dependence of the threshold on the quantum defect.

In connection with the spectral behavior shown in Figure 13, it should be noted that the behavior of $[eqn]$ as a function of $[eqn]$ is derived from the asymptotic TMI threshold formalism [38], in which the wavelength dependence enters primarily through the $[eqn]$ parameter and the transverse optical eigenvalue structure governing modal coupling. In contrast, the semi-analytical model reported by Tao et al. [36] includes wavelength-dependent gain saturation and spatial hole burning, which can produce non-monotonic spectral features with a maximum near 1030 nm. These effects were intentionally excluded in the present formulation. Despite the different modeling assumptions, the overall decrease in the TMI threshold toward longer signal wavelengths shown in Figure 13 remains qualitatively consistent with the trend reported in [36].

To confirm the consistency of these results, the threshold was also evaluated as a function of core diameter $[eqn]$ while maintaining the same operating wavelengths $[eqn]$ nm and $[eqn]$ nm. As shown in Figure 14, the thresholds obtained at $[eqn]$ m match those derived from the quantum-defect parameterization. This agreement is expected since both formulations originate from the same asymptotic expression (31) and depend on the same transverse eigenvalue structure. In addition, to benchmark the present SCF predictions against published amplifier data, discrete threshold values extracted from the Zervas model are summarized in Table 4.

The quantum defect governs the longitudinal heat load, while the transverse eigenvalues determine how this heat distribution couples the $[eqn]$ mode with higher-order modes through STRS. Because the DCF exhibits a modified thermal diffusion profile associated with its three-layer geometry, the resulting temperature distribution interacts differently with the modal intensity patterns. Within the asymptotic framework proposed by Zervas for fiber amplifiers [38], this difference manifests primarily through the dependence of the threshold on the transverse eigenvalue structure appearing in Equation (31), while the local thermo-optic term remains governed by the thermal and saturation parameters of the model, such as $[eqn]$ and $[eqn]$ . Consequently, the lower threshold obtained for the DCF in both parameterizations is not incidental: it emerges from a unified optical–thermal mechanism linking quantum-defect heating, transverse modal structure, thermal diffusion, and the nonlinear coupling coefficient $[eqn]$ .

Table 4 compares the SCF TMI threshold values predicted by the amplifier-based model employed in the present work with those reported by Zervas [38], considering $[eqn]$ and a saturated gain coefficient $[eqn]$ . The remaining fiber and amplifier parameters not explicitly specified in [38] were adopted from the simulation parameter set listed in Table 1. Small discrepancies are therefore expected due to differences in thermal boundary conditions and material parameters.

Figure 15 shows that the instability threshold’s dependence on the core numerical aperture (NA) and the normalized frequency parameter V is directly related to the modal structure supported by the fiber. Since the asymptotic threshold expression derived from the Zervas model [38] depends explicitly on the transverse eigenvalues $[eqn]$ and $[eqn]$ , variations in NA (and therefore V) modify the modal confinement and the eigenvalue separation governing the $[eqn]$ – $[eqn]$ coupling responsible for STRS-driven TMI. In the present case, the dominant channel corresponds to the $[eqn]$ – $[eqn]$ pair, and the resulting NA- and V-dependent trends are qualitatively consistent with the numerical and experimental observations reported by Tao et al. [44], although the specific fiber and amplifier parameters considered here are not identical.

4. Conclusions

In this work, we developed and applied a coupled optical–thermal formulation for the analysis of TMI in high-power continuous-wave double-clad ytterbium-doped fiber amplifiers, explicitly accounting for the multi-layer radial geometry. The model incorporated modal profiles obtained from dispersion relations, radial thermal modes, gain saturation, and the nonlinear coupling coefficient $[eqn]$ associated with stimulated thermal Rayleigh scattering (STRS). The interaction between the fundamental $[eqn]$ mode and higher-order modes was described through coupled power equations, reproducing both modal transfer and the onset of the thermal instability threshold.

The results show that replacing a two-layer SCF with a three-layer DCF alters the transverse modal eigenvalues and the associated optical–thermal overlap integrals. For the geometry investigated, the eigenvalue structure imposed by the inner cladding leads to stronger STRS-induced coupling and, consequently, a lower TMI threshold for the DCF, even though the silica-related thermal parameters are essentially the same in both fibers. A quantitative comparison with previously reported SCF results demonstrated good agreement with the literature over a wide range of core diameters, validating the present model and confirming the systematic reduction in the TMI threshold in DCFs. Eigenvalue analysis further confirmed that the dominant instability channel depends sensitively on the modal structure: the rigorous Tsao formalism predicted preferential coupling between $[eqn]$ and $[eqn]$ , whereas the formulation of Tan yielded a physically meaningful threshold only for the $[eqn]$ – $[eqn]$ pair because of the relative ordering of the transverse eigenvalues. These differences highlight the importance of maintaining internal consistency between optical modal analysis and the thermal diffusion model when evaluating thermally induced modal instabilities.

In summary, the observed instability is a manifestation of thermally driven modal coupling in high-power fiber amplifiers: quantum-defect heating generates a delayed refractive-index modulation that forms a traveling thermal grating responsible for coherent intermodal power transfer. This mechanism provides a consistent physical interpretation of the STRS-based modal coupling described by Dong and by Smith & Smith while remaining compatible with the asymptotic thermal-instability threshold scaling predicted by the analytical model developed by Zervas. The threshold analysis for variations in the core NA showed that $[eqn]$ adheres to an NA- and V-dependent behavior determined by changes in the transverse eigenvalues governing $[eqn]$ – $[eqn]$ coupling. As a direct implication of this modal requirement, in the single-mode regime (e.g., $[eqn]$ m), where no guided HOM exists, an FM–HOM TMI threshold associated with $[eqn]$ – $[eqn]$ power transfer is not defined within the adopted formulation. The numerical results further show that the axial build-up of STRS-induced coupling naturally gives rise to intermittent temporal energy exchange between transverse modes at the amplifier output, a finding that is in agreement with experimental observations of beam fluctuations reported in the literature. The formulation presented here quantifies the impact of multi-layer fiber geometry on thermo–optic coupling and provides a foundation for designing high-power fiber lasers with improved modal stability.