Secure analysis in cognitive satellite-terrestrial networks with untrusted LEO relaying and friendly jamming

TL;DR

This paper explores how to secure data transmission in satellite networks by using a jammer to prevent eavesdropping through untrusted relays.

Contribution

The study introduces a friendly jammer to enhance secrecy rates in cognitive satellite-terrestrial networks with untrusted LEO relays.

Findings

Increasing interference threshold initially boosts secrecy rate but saturates at maximum transmit power.

Removing maximum power constraints allows optimizing secrecy rate via interference threshold adjustment.

Lighter shadowing improves secrecy rates despite enhancing the untrusted relay's channel.

Abstract

This study examines physical layer security in a cognitive satellite-terrestrial network where a ground user (GU) transmits confidential data to a Geostationary Earth Orbit (GEO) satellite via an untrusted Low Earth Orbit (LEO) relay adopting the Amplify-and-Forward (AF) relay strategy. To counter eavesdropping risks from the relaying LEO satellite, a friendly LEO jammer emits artificial noise. The secrecy rate is analyzed under cognitive radio constraints, considering interference thresholds, relay gain, and GU-LEO link shadowing. Numerical results show that increasing the interference threshold initially improves the secrecy rate but saturates when the maximum transmit power constraint at the GU is reached. When this maximum power constraint is removed, the interference threshold can be adjusted to optimize the secrecy rate. Additionally, we observe that lighter shadowing conditions generally enhance the secrecy rate, even though they improve the untrusted relay’s channel. These findings offer valuable insights into optimizing security in cognitive satellite-terrestrial networks.

Figures

System model.A cognitive satellite-terrestrial communication system model.

Secrecy rate versus Ith with PM = 10 dB, GAF = 1, m = 2, m0 = 20, b = 0.158, and Ω=1.29.The impact of the interference threshold Ith on the secrecy rate under different artificial noise power levels.

Secrecy rate versus Ith with PM→∞, GAF=1, σZ2=5, m0 = 20, b = 0.158, and Ω=1.29.

Secrecy rate versus Ith with PM = 10 dB, m = 2, m0 = 20, b = 0.158, and Ω=1.29.

Secrecy rate versus Ith with PM = 10 dB, GAF=1, m = 2 and σZ2=5.

Equations

Full text

Introduction

The integration of satellite and terrestrial networks has become an essential component in modern communication systems, particularly as demand for seamless, high-speed global coverage continues to grow [1–3]. Low Earth Orbit (LEO) satellites, due to their proximity to the Earth, offer low-latency and high-bandwidth capabilities, position them as ideal relay candidates in hybrid satellite-terrestrial networks [4]. LEO satellites have been also used as relay nodes to improve the connectivity between ground users (GUs) and Geostationary Earth Orbit (GEO) satellites, enabling efficient long-distance communications [5, 6].

Recently, several studies (e.g., [7]) have investigated cognitive radio (CR) based hybrid satellite–terrestrial networks, in which the GU accesses the spectrum originally licensed to terrestrial primary users (PUs) to establish communication with satellites. This spectrum-sharing strategy is implemented to address the scarcity of available spectrum in satellite-terrestrial systems, allowing the GU to opportunistically access the same frequency bands as terrestrial PUs. However, this introduces additional challenges, as the GU must limit its transmit power to avoid causing interference to PUs, such as terrestrial base stations and mobile users. Failure to control the interference can degrade the communication quality for the PUs. Consequently, power control becomes a critical component in ensuring delivery performance in both the primary and secondary networks, balancing the need for effective communication between the GU and the satellite while protecting the quality of service for the terrestrial PUs. This power control constraint also makes it challenging for the GU to establish a direct link to the GEO satellite, especially in scenarios with heavy obstructions. In such cases, LEO satellites can effectively serve as relays, forwarding messages from the GU to the GEO satellite [6]. Amplify-and-Forward (AF) relaying is particularly relevant in satellite systems due to its simplicity and efficiency in scenarios where the relay may not have the computational resources required for more complex strategies like Decode-and-Forward (DF) relaying [8].

However, with the increasing reliance on LEO satellites as relays, security concerns become even more pronounced when these relay satellites are considered untrusted. An untrusted LEO satellite may not only forward the GU’s signal to the GEO satellite but may also attempt to eavesdrop on the transmitted information, posing a direct threat to the confidentiality of the communication. The challenge is further complicated by the fact that the untrusted LEO relay may have access to the codebook used by the GU, enabling it to decode the relayed messages. Moreover, any physical layer technique that aims to improve the connection between the GU and the GEO satellite, such as increasing the transmit power or optimizing signal transmission, will simultaneously enhance the connection between the GU and the LEO relay. This makes it challenging to secure the communication without inadvertently strengthening the eavesdropper’s channel. Nevertheless, physical layer security techniques can still ensure secure communication by exploiting the inherent randomness of wireless channels [9, 10], even if the untrusted relay has access to critical information, such as the codebook [11].

In recent years, there has been a growing body of research on physical layer security in terrestrial-satellite systems. For instance, in [12], the secrecy performance was investigated in a scenario involving randomly located vehicles distributed across different roads, where a satellite aims to securely serve one vehicle while preventing information leakage to another located elsewhere. In [13], the authors proposed a spatial cooperation strategy, wherein symbol mapping, beam steering, and antenna array partitioning were jointly optimized to confine the insecure region and suppress sidelobe radiation. The study in [14] considered the effect of terrestrial interference at the satellite and employed stochastic geometry to model the spatial distribution of LEO satellites, where multiple unintended LEO satellites attempt to eavesdrop on a legitimate communication link between a terrestrial IoT device and its intended LEO receiver. Moreover, artificial noise has also been investigated as a means to enhance communication secrecy. For example, [15] analyzed the security–reliability tradeoff in satellite–terrestrial relay networks assisted by a friendly jammer, where a terrestrial relay assists communication between a satellite and a ground user, while an external jammer broadcasts artificial noise to interfere with multiple nearby eavesdroppers.

More recently, physical layer security in cognitive satellite–terrestrial networks with artificial noise has also drawn attention. For instance, [16] studied the secrecy performance in the presence of artificial noise and non-colluding eavesdroppers. However, this work focused only on the secondary terrestrial network, while the security of the satellite–terrestrial network was ignored. Notably, scenarios involving cognitive satellite–terrestrial networks with untrusted LEO relays have received little attention and remain an open research direction. One of the key challenges lies in the analytical complexity introduced by terrestrial–satellite wireless channels. In most existing works (e.g., [12, 15]), the derived expressions for performance metrics such as secrecy rate or secrecy outage probability are highly intricate, which not only makes numerical evaluation computationally intensive but also hinders the extraction of useful design insights from the resulting formulas.

In this paper, we focus on analyzing the secrecy rate in a CR hybrid satellite-terrestrial network where an untrusted LEO satellite serves as an AF relay between a GU and a GEO satellite. In order to mitigate the eavesdropping threat posed by the untrusted LEO relay, a friendly LEO jammer transmits artificial noise to disrupt the relay’s ability to intercept the communication. We derive closed-form expressions for the secrecy rate and investigate how different system parameters, such as channel fading, power control, and jamming, affect the physical layer security of the system. Our analysis is conducted within a CR framework, where the GU’s transmit power is constrained by interference limits at terrestrial PUs. The key contributions of this paper are outlined as follows:

We develop an analytical model for the secrecy rate in a CR satellite-terrestrial network with an untrusted LEO AF relay and a friendly LEO jammer, considering various system parameters such as channel fading and power constraints imposed by CR.We derive an exact closed-form expression for the secrecy rate, providing insights into the security performance of the system under untrusted AF relaying.We validate the analytical results through Monte Carlo simulations and explore how power limitations imposed by CR, and the impact of jamming and shadowing, affect the secrecy rate.

System model

In this paper, we consider a cognitive satellite-terrestrial communication system, as shown in Fig 1, where a GU transmits confidential data to a GEO satellite through an untrusted LEO satellite, which operates as an AF relay. The DF protocol is not considered in this work, as it requires the relay to decode the legitimate user’s message before forwarding [17]. In the untrusted relay scenario studied here, this would lead to full exposure of confidential information, making DF unsuitable. The system operates under a CR framework, meaning that the GU uses the same spectrum as the terrestrial primary networks. In such an environment, the transmit power of the GU is constrained to ensure that the interference to the terrestrial primary users (PUs), remains below a specific threshold. A direct connection between the GU and the GEO satellite is not feasible due to heavy obstructions, such as buildings or terrain, causing significant shadowing. Additionally, the power control imposed by the underlay CR framework further challenges the feasibility of a direct GU-to-GEO link, as the GU must limit its transmit power to avoid causing interference to terrestrial PUs.

System model.A cognitive satellite-terrestrial communication system model.

The LEO relay is not fully trusted to forward the message securely, and it may potentially eavesdrop on the transmitted messages. To mitigate this risk, we introduce a cooperative jamming strategy where another LEO satellite serves as a friendly jammer. This jammer is located outside the field of view of the GU, so it cannot act as a possible relay for forwarding the messages from the GU to the GEO satellite. However, it can still transmit interference to the untrusted LEO relay, effectively degrading its ability to eavesdrop on the transmitted messages.

The message transmission from the GU to the GEO satellite occurs over two-time slots. In the first time slot, the GU transmits its signal to the untrusted LEO satellite, while a friendly LEO jammer simultaneously sends artificial noise to the untrusted LEO satellite to degrade its eavesdropping capability. In the second time slot, the untrusted LEO satellite forwards the signal received from the GU in the first time slot to the GEO satellite, applying the AF relaying strategy.

We consider a widely used underlay CR framework, where the transmit power of the GU (denoted by PU) is constrained to ensure that the interference at the PU remains below a predefined threshold, $[eqn]$ . The channel gain between the GU and the PU is represented by hT, and the interference constraint is expressed as $[eqn]$ Additionally, we account for a maximum power constraint, denoted by PM, at the GU. Therefore, the transmit power at the GU in this underlay CR network is given by (cf. [18, 19])

$$[eqn]$$

In contrast to the GU’s power control imposed by the CR framework, the friendly LEO jammer operates without such power limitations, as it is far away from the terrestrial primary networks.

Communication links

In the following, we provide a detailed description of the channel models for the terrestrial-satellite link, the inter-satellite link, and the terrestrial link, respectively.

GU-to-LEO link.

The GU transmits a signal to the LEO satellite over a wireless link subject to Rician-Shadowed fading. The channel gain for this link is denoted as hL, and follows a Rician-Shadowed distribution characterized by three parameters: the parameter m0 reflecting the average obstruction of the LOS component, the average power of the scattered component 2b, and the average power of the LOS component, denoted by Ω. The probability density function (PDF) of the Rician-Shadowed fading channel power gain $[eqn]$ is given by the following expression [20]

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ are functions of the Rician-Shadowed fading parameters m0, b, and Ω, and $[eqn]$ is the confluent hypergeometric function of the first kind. For simplification, we consider that m0 is a positive integer. Then, the PDF and the cumulative distribution function (CDF) of $[eqn]$ are respectively of the form [12, 21]

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ , and $[eqn]$ denotes the upper incomplete Gamma function.

Satellite-to-satellite links.

The LEO satellite forwards the amplified signal to the GEO satellite. The LEO-to-GEO link is considered to be subject only to large-scale path loss, and the channel gain for this link is denoted as LG, representing the path loss. Satellite-to-satellite links typically do not experience small-scale fading due to the absence of obstacles in space and the direct LoS nature of the communications, which eliminates the effects of multipath propagation. The signal received at the GEO satellite is given by

$$[eqn]$$

where xL is the amplified signal sent from the LEO satellite, and nG is the Additive White Gaussian Noise (AWGN) at the GEO satellite with the variance of $[eqn]$ . Similar to the LEO-to-GEO link, we assume that the link between the LEO jammer and the untrusted LEO relay experiences only large-scale path loss (denoted by LJ), with no small-scale fading effects.

GU-to-PU link.

For the GU-to-PU link, we assume a Nakagami-m fading channel. The PDF and CDF of $[eqn]$ , with a mean of $[eqn]$ , are expressed as follows

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ represents the Gamma function, and $[eqn]$ denotes the lower incomplete Gamma function.

Untrusted AF relaying

The untrusted LEO satellite employs the AF relaying strategy, in which it amplifies the received signal from the GU and forwards it to the GEO satellite. The amplification gain at the LEO satellite is denoted by $[eqn]$ , and the signal received at the GEO satellite after relaying is

$$[eqn]$$

where xU is the transmitted signal from the GU, $[eqn]$ is the AF gain, and nL is the AWGN with the variance of $[eqn]$ at the LEO satellite. In the above, $[eqn]$ and LJ represent the artificial noise and the transmit power of the friendly LEO jammer respectively. Notably, the artificial noise z is known to the GEO satellite, whereas the untrusted LEO relay has no prior knowledge of it.

Since the LEO relay is untrusted, the GU does not perform channel training with the LEO satellite, meaning that the LEO satellite lacks knowledge of the instantaneous channel state information (CSI) for the GU-to-LEO link. This approach prevents the untrusted LEO relay from adapting its gain to degrade the SNR at the GEO satellite. In this scenario, a fixed amplification factor is considered, as the LEO satellite does not have the instantaneous CSI of the first hop. The fixed AF relay gain ensures that the LEO relay maintains a consistent level of amplification regardless of the instantaneous channel conditions between the GU and the LEO satellite. By fixing the relay gain, the system maintains control over potential security risks posed by the untrusted relay while enabling the LEO satellite to forward the signal to the GEO satellite.

Since the GU does not engage in channel training with the LEO satellite, and due to the absence of a direct transmission path between the GU and the GEO satellite caused by significant LoS obstructions, the GEO satellite lacks access to the instantaneous CSI for both the GU-LEO link and the GU-terrestrial PU link. As a result, the GEO satellite demodulates the received signal based solely on the statistical properties of the GU-LEO channel and the GU-PU channel. This reliance on statistical knowledge complicates the signal decoding process, as the GEO cannot utilize instantaneous CSI to optimize its decoding strategy. Instead, the GEO must rely on the channel’s average behavior to decode the signal. Given this scenario and removing the artificial noise, the signal for decoding xU is of the form

$$[eqn]$$

which can be rewritten as

$$[eqn]$$

where $[eqn]$ denotes the expectation operator. The end-to-end SNR at the GEO satellite, denoted as $[eqn]$ , can be expressed as

$$[eqn]$$

where $[eqn]$ denotes the variance of the inside random variable.

To decode the signal sent from the GU based on the statistical CSI knowledge, the received signal at the untrusted LEO satellite can be rewritten as

$$[eqn]$$

The SNR for decoding xU at the LEO satellite is of the form

$$[eqn]$$

Secrecy rate

We aim to analyze the secrecy rate of the communication link between the GU and the GEO satellite in the presence of both an untrusted LEO relay and a friendly LEO jammer. The secrecy rate is defined as the difference between the achievable rates at the GEO satellite and the LEO relay. This rate represents the secure transmission capacity of the information sent from the GU to the GEO satellite.

Mathematically, the secrecy rate Cs is expressed as

$$[eqn]$$

where $[eqn]$ is the SNR at the GEO satellite, and $[eqn]$ is the SNR at the untrusted LEO relay. The term $[eqn]$ ensures that the secrecy rate is non-negative. The friendly jammer reduces $[eqn]$ , thereby improving the secrecy rate by degrading the LEO relay’s ability to intercept the message.

Performance analysis

In this section, we will derive the closed-form expression for the secrecy rate. Observing (11), (13), and (14), we know that we should derive the closed-form expressions for $[eqn]$ and $[eqn]$ .

Theorem 1. In the presence of an untrusted LEO relay and an LEO jammer, the secrecy rate of the considered hybrid cognitive terrestrial-satellite networks takes the form in (14) with the expressions for $[eqn]$ and $[eqn]$ respectively given by

$$[eqn]$$ $$[eqn]$$

where the mean of $[eqn]$ is given by

$$[eqn]$$

The closed-form expressions for $[eqn]$ , $[eqn]$ and $[eqn]$ are respectively

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

Proof of Theorem 1: For $[eqn]$ , we have

$$[eqn]$$

For the second term in the above, we have

$$[eqn]$$

where (a) follows from [22, Eq. (3.381.3)].

For the mean of PU, we have

$$[eqn]$$

where

$$[eqn]$$

where (a) follows from [22, Eq. (3.381.3)]. Combining (23) and (24) yields (18).

For $[eqn]$ , by using the PDF of $[eqn]$ , we have

$$[eqn]$$

where (a) follows from [22, Eq. (3.381.4)]. Referring to [23, Prop. 5.1], we know that

$$[eqn]$$

We conclude the proof. □

Remark 1. Theorem 1 introduces a novel analytical framework for characterizing the secrecy rate in the considered system. Notably, the proposed framework can potentially be extended to address other physical-layer security concerns, such as covert communications [24], where the untrusted relay is unaware of the legitimate user’s transmission and simply amplifies and forwards the received signal.

Numerical results and discussion

In this section, we present the numerical results for the secrecy rate in the considered cognitive satellite-terrestrial network with an untrusted LEO relay and a friendly LEO jammer. The results are validated through Monte Carlo simulations, and we analyze the impact of various system parameters on the secrecy rate. For simplicity, we set $[eqn]$ . In Figs 2–4 , the GU-to-LEO relay link is subject to infrequent light shadowing, with parameters m0 = 20, b = 0.158, and $[eqn]$ . In Fig 5, we examine three different shadowing conditions for the GU-to-LEO relay link: frequent heavy shadowing (m0 = 1, b = 0.063, and $[eqn]$ $[eqn]$ 10^−4^), average shadowing (m0 = 10, b = 0.126, and $[eqn]$ ), and infrequent light shadowing (m0 = 20, b = 0.158, and $[eqn]$ ). The parameter settings for the three shadowing scenarios are summarized in Table 1.

Secrecy rate versus Ith with PM = 10 dB, GAF = 1, m = 2, m0 = 20, b = 0.158, and Ω=1.29.The impact of the interference threshold Ith on the secrecy rate under different artificial noise power levels.

Secrecy rate versus Ith with PM→∞, GAF=1, σZ2=5, m0 = 20, b = 0.158, and Ω=1.29.

Secrecy rate versus Ith with PM = 10 dB, m = 2, m0 = 20, b = 0.158, and Ω=1.29.

Secrecy rate versus Ith with PM = 10 dB, GAF=1, m = 2 and σZ2=5.

Table 1: Parameters in three typical shadowing scenarios [20, Table III].

Fig 2 illustrates the impact of the interference threshold $[eqn]$ on the secrecy rate under different artificial noise power levels. As $[eqn]$ increases, the secrecy rate improves, as a higher threshold allows the GU to transmit with greater power, enhancing the SNR at the GEO satellite. However, beyond a certain point, the secrecy rate stabilizes and becomes nearly unchanged. This is due to the maximum transmit power constraint at the GU, which limits the GU’s ability to further increase its power, regardless of the increase in $[eqn]$ . Therefore, once the GU reaches its maximum transmit power, further increases in $[eqn]$ provide no additional benefit. This indicates that the system is bound by the maximum power limit of the GU, making the adjustment of $[eqn]$ effective only up to a certain level. Beyond that, the secrecy rate plateaus, and the system performance depends more on other factors, such as the artificial noise power $[eqn]$ .

In Fig 3, we remove the maximum transmit power constraint, allowing the GU to transmit with power always close to $[eqn]$ . Under this setting, the secrecy rate initially increases with $[eqn]$ , reaching a peak, and then decreases. The initial increase occurs because a higher $[eqn]$ enhances the link quality between the GU and the GEO satellite, improving the achievable rate at the GEO satellite. However, as $[eqn]$ continues to increase, the untrusted LEO relay’s ability to intercept the message improves as well, because the influence of the artificial noise $[eqn]$ becomes less significant. When $[eqn]$ is sufficiently large, the artificial noise becomes almost ineffective at hindering the untrusted relay, leading to a decrease in the secrecy rate. This suggests that there is an optimal value of $[eqn]$ that balances the improvement in the legitimate link and the negative effect on the eavesdropping link. The value of $[eqn]$ can be adjusted by varying the distance between the GU and the PU, which is a crucial factor in determining the overall secrecy performance.

Fig 4 examines the effect of the AF gain $[eqn]$ on the secrecy rate. As $[eqn]$ increases, the secrecy rate also improves because a higher AF gain amplifies the received signal at the GEO satellite, leading to a better end-to-end SNR. However, as $[eqn]$ becomes sufficiently large, further increases no longer result in noticeable improvements in the secrecy rate. This is due to the fact that a higher AF gain also amplifies the noise in the first hop (GU-to-LEO link). As a result, while the legitimate signal is amplified, the accompanying noise is also magnified, which limits the benefit of increasing $[eqn]$ indefinitely. The results indicate that although a moderate AF gain can significantly improve the secrecy rate, there is a saturation point beyond which the noise amplification "equals" the signal amplification.

Fig 5 shows the secrecy rate under three different shadowing conditions for the GU-to-LEO link: frequent heavy shadowing, average shadowing, and infrequent light shadowing. As expected, lighter shadowing leads to higher SNRs at both the untrusted LEO relay and the GEO satellite. While lighter shadowing improves the decoding capability of the untrusted relay, it also benefits the legitimate link to the GEO satellite by improving the end-to-end SNR. Interestingly, the results imply that lighter shadowing is generally favorable for secure transmission, as the overall improvement in the end-to-end link dominates the potential advantage gained by the untrusted relay. Therefore, despite the improvement in the eavesdropper’s channel quality, the GEO satellite’s ability to decode the confidential message is enhanced more substantially under lighter shadowing conditions, leading to better secrecy performance.

In conclusion, the numerical results reveal that key parameters such as the interference threshold Ith, AF gain $[eqn]$ , and the shadowing conditions play significant roles in determining the secrecy rate. Adjusting these parameters can help balance the improvement in the legitimate link and the impact of eavesdropping, providing valuable insights into how to optimize the system’s secrecy performance in the presence of an untrusted LEO relay and a friendly LEO jammer. Moreover, the excellent agreement between the simulated results and the theoretical analysis in Figs 2–5 fully validates the accuracy and reliability of the developed analytical model, confirming its robustness in predicting system behavior under various conditions.

Conclusions

In this paper, we analyzed the physical layer security of a cognitive hybrid satellite-terrestrial communication system where a ground user communicates with a GEO satellite through an untrusted LEO relay, with the presence of a friendly LEO jammer. We derived the closed-form expression for the secrecy rate and investigated the impact of key system parameters, such as the interference threshold, AF gain, and shadowing effects, on the secrecy performance. Our results show that increasing the interference threshold improves the secrecy rate up to a certain limit, after which the impact of the untrusted relay diminishes the overall security. The AF gain was also found to improve secrecy but with diminishing returns as the noise amplification limits its benefits. Lastly, our analysis revealed that lighter shadowing conditions, despite improving the untrusted relay’s channel, generally favor secure communication by significantly enhancing the end-to-end SNR at the GEO satellite.

The findings highlight the importance of carefully balancing system parameters to optimize security in cognitive satellite-terrestrial networks. Future work can explore further optimizations, such as power allocation strategies and advanced jamming techniques, to enhance the physical layer security in these systems.