Intelligent Spectrum Sensing for NOMA Systems: A Cost-Sensitive LightGBM Approach with Objective-Driven Learning

TL;DR

This paper introduces a new machine learning method for improving spectrum sensing in NOMA systems by aligning training with sensing objectives.

Contribution

The novel contribution is an objective-driven learning approach using cost-sensitive LightGBM for spectrum sensing in NOMA systems.

Findings

The proposed OCL method achieves an average overall sensing ability score of 0.638, outperforming existing methods.

OCL improves sensing ability by 6.26% under challenging signal conditions compared to EBSS-RF.

Eigenvalue-based features help capture discriminative signal patterns in dynamic NOMA transmission.

Abstract

In NOMA-enabled CR systems, superposed PU signals with unequal power levels and independent activity significantly complicate spectrum sensing and channel state discrimination. To address this issue, ML-based sensing exploits spectrum-domain features to perform channel state classification. However, the ML-based methods remain limited under independent PU activity and suffer from the performance tradeoff issue since the spectrum sensing constraints are not explicitly incorporated into the learning process. In this paper, we propose an OCL method that aligns LightGBM multiclass training with spectrum sensing objectives and leverages eigenvalue-based features to capture discriminative signal patterns under dynamic NOMA transmission. The cost-sensitive learning strategy is used to guide the classifier while the objective-driven tuning is used to optimize hyperparameters toward spectrum sensing objectives. To evaluate the overall performance toward Pd and Pfa, we propose an overall sensing ability score by adopting the SPOTIS method. As a result, the proposed OCL method achieves the highest overall sensing ability scores with an average score of 0.638, outperforming EBSS-RF at 0.610 and FBSS-LR at 0.221. Under challenging signal pattern discrimination conditions, the OCL method improves the overall sensing ability score by 6.26% and 0.9 under different power coefficients compared to EBSS-RF, highlighting its effectiveness in addressing the performance tradeoff issue.

Figures

Full text

1. Introduction

Along with the advancement of fifth-generation (5G) and sixth-generation (6G) networks, the accelerated expansion of smart wireless devices has resulted in severe spectrum congestion and scarcity [1]. Meanwhile, conventional static spectrum allocation [2,3,4] is limited in efficiency and scalability for wireless communication systems since it assigns fixed frequency bands to users. Therefore, licensed bands remain severely underutilized due to the intermittent usage behavior of licensed/primary users (PUs), while unlicensed/secondary users (SUs) are prohibited from accessing them, even when they experience excessive congestion. Currently, several promising technologies have been proposed; however, no single solution satisfies future wireless system requirements, thereby encouraging integrated approaches.

The integration of cognitive radio (CR) [5] and nonorthogonal multiple access (NOMA) [6] presents a promising solution to improve spectrum efficiency and support massive user connectivity in emerging smart wireless networks over limited bandwidth. NOMA allows multiple users to share the same time–frequency resources via power-domain multiplexing with different [7] power levels assigned according to channel conditions, while CR enables the SUs to utilize the licensed bands while avoiding PU reactivation and harmful interference. The integration of NOMA into CR networks is one of the promising solutions [8] that allows the SU to access the licensed band together with PUs when the number of active PUs does not fully occupy the NOMA channel capacity [9]. To satisfy the interference constraint, spectrum sensing and channel estimation are employed to acquire information about the interference channels toward PUs [10,11], thereby enabling SUs to adaptively adjust their transmission parameters to meet the imposed interference constraints. Thus, SUs must not only detect the idle or busy state of a PU channel but also identify the number of active PUs.

In downlink NOMA [12], the base station (BS) transmits superimposed signals of multiple users over the same subcarrier with different power levels while the successive interference cancellation (SIC) method is exploited at the receiver to decode the signal. However, conventional spectrum sensing techniques in CR, such as energy detection [13] and maximum eigenvalue detection, cannot provide reliable discrimination of PU activity in NOMA-enabled CR systems due to signal superposition. In contrast, feature-based spectrum sensing (FBSS) [14,15] has emerged as an alternative by exploiting discriminative signal features; however, its performance is limited under low signal-to-noise ratios (SNRs) of NOMA environments. Moreover, the detection task becomes more challenging under varying PU activity patterns, such as scenarios where only a single PU occupies the licensed channel, as well as under low SNR conditions resulting from path loss.

To overcome the limitations of conventional CR spectrum sensing, machine learning (ML) has been employed to enhance spectrum sensing performance without relying on predefined detection thresholds. In [16], the energy detection is combined with logistic regression, k-nearest neighbor (KNN), and neural networks to investigate the effectiveness of classification models. The simulation results indicate that the effectiveness of classification models enhances the spectrum sensing performance as compared to conventional energy detection.

In [17], the NOMA spectrum sensing problem is formulated as a multi-hypothesis problem where the particle swarm optimization is adopted with a Gaussian mixture model (GMM) to estimate channel coefficients. In [18], deep learning models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), are adopted in spectrum sensing. To extract features from the sequential data of signal, a one-dimensional convolutional neural network (1DCNN) was employed to determine the signal pattern of the PU and determine the channel state using a time series classification model such as a long short-term memory network [19]. Deep learning models have a strong capability to extract features, allowing them to achieve superior detection accuracy without relying on predefined detection thresholds. Nevertheless, the non-cooperative spectrum sensing of NOMA-based spectrum sensing suffers from the performance tradeoff issue [20] in which the spectrum sensing only maximizes the probability of detection ( $[eqn]$ ) while failing to minimize the probability of false alarm ( $[eqn]$ ). Thus, the cooperative NOMA-based spectrum sensing is proposed to address the performance tradeoff issue. In [21], the ML-driven method is also exploited in cooperative NOMA-based spectrum sensing to address the performance tradeoff issue by applying both unsupervised methods—K-means clustering and Gaussian mixture models—and supervised models—support vector machine (SVM), KNN, and neural networks—to detect multi-PU activity. Even if the cooperative spectrum sensing can address the performance tradeoff issue, it comes at the cost of computational complexity. Therefore, we focus on the non-cooperative spectrum sensing method.

According to multiuser discrimination ability, the spectrum sensing ability of FBSS is enhanced by employing the classification ability of ML [22], where the channel states can be determined by 0.5 of the threshold. By incorporating logistic regression (LR) with FBSS features, the NOMA-based spectrum sensing performance is improved via varying SNR conditions and allocated power coefficients. Eigenvalue-based channel state classification (EBSS) with ML is introduced in [23], where dominant eigenvalues extracted via eigen-decomposition are used as features. Evaluations with LR, Naive Bayes (NB), and Random Forest (RF) under NOMA-based spectrum sensing show that RF performs best, and EBSS outperforms energy-based and maximum-eigenvalue methods in accuracy and F1-score.

Motivation and Contribution

While integrating NOMA into cognitive radio networks, conventional CR-based spectrum sensing techniques become inadequate for reliable channel state detection, as they are typically designed under binary hypothesis assumptions. Under the superposition of unequal-power PU signals and time-varying PU activity in low-SNR environments, overlapping signal patterns severely hinder channel state discrimination, posing a critical challenge for NOMA-based spectrum sensing. In this paper, we proposed an objective-driven cost-sensitive learning for Light Gradient Boosting Machine-based spectrum sensing (OCL) method under dynamic PU activity in a NOMA downlink scenario where the contribution is three-fold.

First, we introduce a cost-sensitive learning strategy integrated with the Light Gradient Boosting Machine (LightGBM) [24,25] to construct a probabilistic classifier under NOMA-based spectrum sensing with dynamic PU activity. To reflect this practical scenario, we reformulate the binary classification problem of conventional spectrum hypothesis into multiclass hypotheses to infer idle channel, weak PU, strong PU, and superposed PU activity. Using the cost-sensitive learning strategy [26], the multiclass loss regulates the relative importance of PU-active classes and leads the training behavior toward spectrum sensing objectives. Therefore, the proposed OCL method improves the baseline LightGBM training behavior and can address the performance tradeoff issue of non-cooperative NOMA-based spectrum sensing.

Second, we introduce the objective-driven tuning framework that optimizes the LightGBM hyperparameters and class weights under spectrum sensing constraints. Once the baseline selects hyperparameters based on a single global metric, such as overall accuracy, class weights are either predefined based on dataset imbalance ratios or tuned independently from the structural hyperparameters. This disjointed strategy is ill-suited for cognitive radio networks, where spectrum sensing inherently involves the dual and often conflicting objectives of maximizing $[eqn]$ while rigorously constraining the probability of false alarm $[eqn]$ . Consequently, conventional tuning often results in a suboptimal sensing tradeoff, where high detection is achieved at the cost of excessive false alarms. The proposed objective-driven tuning avoids this inconsistency by jointly optimizing the LightGBM hyperparameters and cost-sensitive weights under explicit spectrum sensing constraints. This joint optimization ensures that the decision boundaries are explicitly calibrated to align with the desired $[eqn]$ and $[eqn]$ targets. As a result, the proposed method maintains strong overall sensing performance across transmission distances.

Third, we introduce the alternative evaluation metric—overall sensing ability— by adopting the multicriteria ranking method—stable preference ordering toward ideal solution (SPOTIS) [27]—to determine an integrated score of the probability of detection ( $[eqn]$ ) while controlling the probability of false alarm ( $[eqn]$ ). Once the ideal sensing target corresponds to the desired maximized $[eqn]$ and minimized $[eqn]$ , the overall sensing score enables a fair and unified assessment of sensing methods. The results demonstrate that the proposed OCL method provides the most reliable overall performance compared with the baselines considered.

The remainder of this paper is organized as follows: Section 2 describes the concept of NOMA-based spectrum sensing and the problem statement. The proposed OCL method, including its mathematical formulations, is explained in detail in Section 3. In Section 4, the effectiveness of the proposed OCL is compared against baseline LightGBM under classification metrics, and then it is compared to the existing NOMA-based spectrum sensing methods. Finally, Section 5 presents the conclusion.

2. NOMA-Based Spectrum Sensing

In this section, we describe the system model and the issue of PU activity on NOMA-based spectrum sensing. In the NOMA downlink system, the BS of the licensed network, or primary network (PRN), performs power-domain superposition according to the channel quality of each PU, resulting in a composite received waveform at the SU receiver. To enable NOMA spectrum sensing with active PU number identification, the sensing problem must be formulated as a multiple-hypothesis problem accounting for the absence or the presence of one or more PUs.

2.1. System Model

Consider that $[eqn]$ PUs are simultaneously served on the same time–frequency resource, with different power levels assigned by the NOMA-PRN base station. The transmitted NOMA signal is constructed by the power-domain superposition, where the baseband signals of multiple PUs are combined with a different power allocation coefficient ( $[eqn]$ ) according to their channel conditions. Then, the composite NOMA signal ( $[eqn]$ ) can be formulated as

$$[eqn]$$

where PUs experiencing poorer channel conditions are assigned lower $[eqn]$ with $[eqn]$ , $[eqn]$ is the signal of ith PU, and $[eqn]$ denotes the average symbol power of ith PU.

From the perspective of NOMA-based spectrum sensing, the SU focuses on identifying channel occupancy and the level of PU activity rather than decoding individual PU signals. Depending on the number of active PUs sharing the NOMA channel, the received signal can be modeled under a set of composite hypotheses. Therefore, the hypothesis ( $[eqn]$ ) represents the channel state determined by the active PUs, where c ∈ {1, 2, …, k}, while $[eqn]$ denotes an empty channel where all PUs are inactive. Then, the signal received ( $[eqn]$ ) at the SU can be written as

$$[eqn]$$

and

$$[eqn]$$

where $[eqn]$ denotes the large-scale path loss in the linear scale at distance $[eqn]$ between the NOMA-PRN base station and the SU, $[eqn]$ is reference distance, $[eqn]$ is the path-loss exponent, and $[eqn]$ is circularly symmetric complex additive white Gaussian noise (AWGN).

2.2. Spectrum Sensing Techniques for NOMA System

To perform channel state classification, ML models were employed using signal features traditionally considered in conventional spectrum sensing techniques. Among these, eigenvalue-based features and cyclostationary features are two of the most widely used representations for training ML-based classifiers. The cyclostationary features can be determined by the correlation between the received signal and its delay, while the eigenvalues can be obtained using the eigen-decomposition theorem. The cyclostationary features can be expressed by

$$[eqn]$$

where $[eqn]$ is the length of the received signal sample and $[eqn]$ is the sample shift. Once we employ the eigenvalues as the input features of the proposed method, the details of the eigenvalues calculation will be explained in the next section.

Currently, several machine learning (ML) models have been adopted for ML-based spectrum sensing, where logistic regression and random forest have demonstrated strong performance, reported in [21,23]. In addition, a deep learning model, one-dimensional convolutional neural network (1DCNN) [19], is employed for spectrum sensing due to its capability to effectively extract and classify one-dimensional signal features. Therefore, these models are described in detail in this section.

2.2.1. Logistic Regression

Logistic regression [15] serves as a statistical tool for binary classification, predicting class labels from observed features by transforming a linear combination of inputs into a probability between 0 and 1 using the sigmoid function. By employing the sigmoid function, the final sensing decision of logistic regression ( $[eqn]$ ) can determined by positive class probability and expressed by

$$[eqn]$$

where $[eqn]$ is the ith feature, $[eqn]$ is the intercept, $[eqn]$ is the coefficients, and $[eqn]$ is the number of features.

2.2.2. Random Forest

Random forest is an ensemble technique that builds multiple decision trees and combines their outputs to boost accuracy while curbing overfitting for better generalization. Each tree trains on a bootstrapped sample of the data (drawn with replacement), using a random subset of features for split decisions at every node—this promotes tree diversity and cuts down correlation between them.

For classification, the final prediction for input $[eqn]$ can be obtained by majority vote, which is given by

$$[eqn]$$

where $[eqn]$ is the ith tree’s output and $[eqn]$ is the total number of trees; regression uses averaging instead.

Random forest excels with noisy data, shows low sensitivity to hyperparameters, and handles nonlinear feature relationships effectively. It also offers feature-important scores for insights, despite the ensemble’s black-box nature, making it ideal for high-dimensional, mixed-type datasets across many tasks.

2.2.3. 1DCNN

The 1D Convolutional Neural Network (1D-CNN) is a deep learning architecture designed to extract discriminative features from sequential data such as in-phase (I) and quadrature-phase (Q) radio signals. The objective of the 1DCNN model is to learn a nonlinear decision function, which is given by

$$[eqn]$$

where $[eqn]$ and $[eqn]$ represent the in-phase and quadrature components of the received signal.

In a 1D-CNN, convolution is performed along the time dimension of the input sequence. For an input vector $[eqn]$ , the output of the $[eqn]$ -th convolutional layer is expressed as

$$[eqn]$$

where $[eqn]$ is the kernel size, $[eqn]$ is the bias, $[eqn]$ are convolution kernel weights, and $[eqn]$ is the activation function.

The convolutional layers act as automatic feature extractors, capturing local temporal correlations and energy patterns in the received signal. After convolution and flattening, the learned feature vector is forwarded to fully connected (FC) layers, which perform the final classification.

2.3. Problem Statement

Due to the power-domain superposition of multiple PUs in the same channel, the statistical characteristics of the received signal vector ( $[eqn]$ ) significantly distort based on unequal power levels of multiple PU signals. Therefore, it highlights the fundamental challenge of spectrum sensing in NOMA-enabled cognitive radio networks, where conventional methods such as energy detection and correlation-based sensing experience severe performance degradation, particularly under low-SNR conditions and strong PU power imbalance.

To overcome these limitations, ML-based spectrum sensing approaches have been adopted to model complex signal characteristics. However, existing ML-based solutions still suffer from two fundamental limitations. First, commonly used features fail to adequately represent channel occupancy and the PU activity in NOMA systems, which further degrades sensing reliability. Second, the objective functions are often misaligned with spectrum sensing objectives that aim to maximize detection probability under constrained false alarm rates.

3. Objective-Driven Cost-Sensitive Learning for LightGBM-Based Spectrum Sensing

In this section, we propose an objective-driven cost-sensitive learning strategy, referred to as objective-driven cost-sensitive learning for LightGBM-based spectrum sensing (OCL), to align LightGBM multiclass classification with the composite hypotheses of NOMA spectrum sensing. In our proposed method, eigenvalue-based features are used to capture the statistical characteristics of $[eqn]$ . The cost-sensitive learning strategy [26] is adopted to regulate the training behavior toward spectrum sensing objectives by formulating the heterogeneous PU activity states into multiclass a classification problem. Moreover, the objective-driven tuning is used to optimize both the LightGBM [24,25] hyperparameters and class weights under spectrum sensing constraints.

As shown in Figure 1, the workflow of the OCL method is illustrated. After the signal is received, it is transformed into an eigenvalue-based representation that captures the underlying second-order statistical structure of the signal. These extracted features are then fed into a cost-sensitive LightGBM classifier, which generates class-wise raw scores through an additive ensemble of leaf-wise regression trees, and converts them into posterior class probabilities via a softmax mapping. To obtain the best model, the objective-driven tuning loop is employed to jointly optimize the class weights and LightGBM hyperparameters under spectrum sensing constraints, aiming to maximize $[eqn]$ performance while controlling $[eqn]$ . Finally, the optimized probabilistic outputs are used to infer the multiclass spectrum occupancy state, enabling reliable discrimination among idle channels, weak PU, strong PU, and superposed PU activity.

3.1. Eigen-Based Feature Extraction

Because $[eqn]$ consists of a superposition of multiple PU signals with unequal power levels, a particular signal feature, such as energy, does not uniquely characterize the underlying spectrum state. In our proposed method, the eigenvalue-based features are adopted to capture the statistical structure of the received signal under different channel occupancy conditions.

Let $[eqn]$ represent the complex baseband received signal vector within the $[eqn]$ th window. Then, the sample covariance ( $[eqn]$ ) can be estimated as

$$[eqn]$$

where $[eqn]$ is the Hermitian transpose.

Then, the eigenvalues corresponding to the eigenvectors are calculated by eigendecomposition, which is expressed by

$$[eqn]$$

where $[eqn]$ is the unitary matrix of eigenvectors, $[eqn]$ is the diagonal matrix of eigenvalues, and $[eqn]$ is number of samples of observation vector.

In our proposed method, we obtain the eigenvalues as the input features of the LightGBM classifier, which determines the channel occupancy condition since it represents the second-order statistical structure of the received signal. Therefore, the eigen-feature vector is constructed as $[eqn]$ . Although eigenvalue-based features come at a high computational cost, we mitigate this issue by employing a fixed snapshot length $[eqn]$ . Therefore, the eigenvalue extraction is performed on a small Hermitian covariance matrix, leading to a linear-time complexity with respect to the signal length per sensing window.

3.2. LightGBM-Based Spectrum Sensing Framework

To determine the channel occupancy condition, LightGBM is adopted with a cost-sensitive learning strategy, where the classification model is trained using class-dependent weighting in the multiclass cross-entropy loss. Therefore, the LightGBM classifier reflects asymmetric misclassification costs among noise and PU activity. In this work, we formulate the multiclass NOMA spectrum sensing problem as a probabilistic classification problem with four channel states, including noise-only, weak PU, strong PU, and superposition of weak and strong PUs. Then, the corresponding spectrum class label can be denoted as $[eqn]$ where $[eqn]$ is number of channel states.

By employing LightGBM as the channel condition classifier, the multiclass sensing output is obtained as a class-wise raw score vector formed by aggregating the contributions of multiple regression trees in an additive ensemble manner. It should be mentioned that the class-wise raw score vector is used instead of hard decisions to retain confidence information across multiple PU activity hypotheses. Then, the class-wise raw score ( $[eqn]$ ) for class $[eqn]$ is constructed as

$$[eqn]$$

where $[eqn]$ is the $[eqn]$ th regression tree associated with class $[eqn]$ and $[eqn]$ is the number of boosting trees.

It can be mentioned that the class-wise raw score is formed by summing the outputs of leaf-wise regression trees, where each tree partitions the eigen-feature space into disjoint regions $[eqn]$ and assigns a constant value per region. A piecewise-constant leaf can be expressed as

$$[eqn]$$

where $[eqn]$ is the number of leaves at iteration, $[eqn]$ is the leaf value for class $[eqn]$ , and $[eqn]$ is the indicator function, which equals one if the condition is satisfied and otherwise is zero.

Since $[eqn]$ represents a class-wise raw score rather than a probability, the posterior probabilities ( $[eqn]$ ) are computed by applying the softmax function to the class-wise raw scores as

$$[eqn]$$

Then, the final sensing decision ( $[eqn]$ ) is formulated by

$$[eqn]$$

3.3. Cost-Sensitive Learning Toward Spectrum Sensing

Once the baseline LightGBM assigns equal cost to all misclassifications, its optimization strategy is not explicitly aligned with spectrum sensing requirements, which emphasize maximizing the probability of detection ( $[eqn]$ ) while controlling the probability of false alarm ( $[eqn]$ ). In cognitive radio spectrum sensing, missed detection is more harmful from the spectrum sensing perspective than a false alarm, and this is exacerbated as high-power PUs dominate the received signal and mask low-power PU activity under NOMA transmission. Then, we adopt a cost-sensitive learning strategy based on weighted multiclass cross-entropy to bias learning toward spectrum sensing performance.

By assigning class-dependent cost-sensitive weight to the loss function of baseline LightGBM, the weighted cross-entropy loss ( $[eqn]$ ) is formulated by

$$[eqn]$$

where $[eqn]$ is the class-dependent weight associated with the true spectrum state $[eqn]$ and

$$[eqn]$$

Then, the weight $[eqn]$ is used to elicit control over the tradeoff between missed detection and false alarm behavior in spectrum sensing. The overall objective function ( $[eqn]$ ) for LightGBM training is then given by

$$[eqn]$$

where $[eqn]$ is the regularization term controlling tree complexity.

To enable second-order boosting in LightGBM for the considered multiclass spectrum sensing problem, the first-order (gradient) and second-order derivatives (Hessian matrix) of the cost-sensitive loss with respect to the class-wise raw score $[eqn]$ for each class can be determined by

$$[eqn]$$

and

$$[eqn]$$

where $[eqn]$ is the gradient term and $[eqn]$ is the diagonal Hessian term. Then, the cost-sensitive learning directly rescales the gradient and second-order statistics in the LightGBM optimization process by regulating the model bias toward PU-active classes to better align with spectrum sensing objectives.

3.4. Objective-Driven Tuning Under Spectrum Sensing Constraints

While cost-sensitive weighting regulates the learning behavior, the optimal balance between $[eqn]$ and $[eqn]$ can be achieved by tuning the class weights and model hyperparameters. Therefore, an objective-driven tuning framework is introduced to optimize both hyperparameters and class weights under spectrum sensing criteria.

By comparing classification to spectrum sensing metrics, $[eqn]$ can be determined by the correct classification of PU active states $[eqn]$ , and $[eqn]$ is misclassification of class $[eqn]$ . Then, $[eqn]$ and $[eqn]$ can be determined by classification metrics as

$$[eqn]$$

and

$$[eqn]$$

where $[eqn]$ is the recall of class $[eqn]$ .

Then, the tuning objective ( $[eqn]$ ) is formulated as

$$[eqn]$$

where $[eqn]$ is the tradeoff weight controlling influence of maximizing detection probability over suppressing false alarms and $[eqn]$ .

4. Results and Discussion

In this section, we evaluate the NOMA-based spectrum sensing performance of the proposed OCL method in comparison with two existing ML-based spectrum sensing approaches—feature-based spectrum sensing (FBSS) [14,15] and eigen-based spectrum sensing (EBSS) [23]. Before conducting the spectrum sensing comparison with FBSS and EBSS, we first assess the effectiveness of OCL against an eigen-based feature with baseline LightGBM under classification metrics. Subsequently, the spectrum sensing performance of OCL is compared with FBSS and EBSS by varying path loss effects corresponding to PU–SU distances ranging from 10 to 150 m.

4.1. Classification-Based Evaluation and NOMA-Based Spectrum Sensing Evaluation Dataset Generation

In our simulations, we generate two independent datasets to separately support model training with classification-based evaluation and spectrum sensing evaluation under realistic channel variations. A synthetic PU signal is generated using QPSK modulation, and the received signal is formed under four channel states: (i) idle channel (noise-only), (ii) weak PU, (iii) strong PU, and (iv) superposition of weak and strong PU. The sample signal length ( $[eqn]$ ) is 4096 and tradeoff weight controlling ( $[eqn]$ ) is set to 0.5. Under an urban scenario, the path-loss exponent ( $[eqn]$ ) is set to 3.0 while the reference distance ( $[eqn]$ ) is set to 100 m.

To train and validate the multiclass spectrum classifiers, the dataset is constructed to represent four spectrum occupancy states under varying distances. The number of samples per class is explicitly controlled to match the class support used in the classification-based evaluation, i.e., 12,320 samples for Class 0, and 13,200 samples for each PU-active class (Classes 1–3). It should be noted that classes 0, 1, 2, and 3 indicate the idle channel, weak PU, strong PU, and superposition of weak and strong PU, respectively. A fivefold cross-validation protocol is then applied to provide an unbiased assessment under standard classification metrics, while mitigating dependence on a single train–test split and reducing the risk of overfitting or underfitting.

To evaluate NOMA-based sensing performance, we generate a separate spectrum sensing evaluation dataset using distance-oriented Monte Carlo simulations. Specifically, the PU–SU distance is varied across the considered range to emulate different path loss conditions. For each distance, we conduct 5000 Monte Carlo iterations where each iteration generates an independent channel and noise realization together with the corresponding received signal under the four sensing states. All methods are applied to infer the spectrum class, and the spectrum performance metrics—such as the probability of detection and probability of false alarm—are computed and averaged over the 5000 trials. Therefore, the classification performance can be assessed under spectrum sensing constraints.

4.2. Classification-Based Performance of OCL Against Baseline LightGBM

First, we compare the classification performance of the proposed OCL method against the baseline LightGBM classifier under five standard metrics. Precision, recall, and F1-score are used to evaluate the classification performance of each class, while the overall accuracy and F1-score are reported in terms of macro-averaged and weighted-average scores. The macro average reflects the mean performance across all classes, whereas the weighted average determines the score by weighting each class according to its number of ground-truth samples.

As shown in Table 1, the classification performance of the baseline LightGBM and OCL are compared when $[eqn]$ and $[eqn]$ and when $[eqn]$ and $[eqn]$ . When $[eqn]$ and $[eqn]$ , by using eigenvalue-based features, a baseline LightGBM classifier presents high detection performance of the idle channel (Class 0), with a high recall of 0.9167 and moderate precision of 0.6691. When only the weak PU (Class 1) is active, the baseline LightGBM classifier shows moderate and relatively balanced performance, with a precision of 0.7437 and a recall of 0.6248. The most severe performance degradation is observed when only the strong PU (Class 2) is active, where the recall drops to 0.3577 and the F1-score reaches only 0.4372. It can be observed that the performance of the baseline LightGBM classifier suffers from similar eigenvalue-based features among noise, the strong PU, and long-distance of weak PU. In contrast, the superposition of PUs (Class 3) exhibits comparatively stable performance, with a recall of 0.7632 and an F1-score of 0.7008. Overall, the baseline LightGBM achieves a macro-averaged F1-score of 0.6477 and a weighted F1-score of 0.6455.

When the power coefficients are $[eqn]$ and $[eqn]$ , noticeable changes in precision, recall, and F1-score can be observed for both the baseline LightGBM and the proposed OCL method. Under the idle channel, both models exhibit noticeable improvement on the classification performance, where baseline LightGBM increases its recall 0.9331, while the OCL method further enhances recall to 0.9917. When the PU power coefficients become closer, the eigenvalue-based features of PU-active states deviate more clearly from noise, resulting in a higher F1-score of 0.8090 for baseline LightGBM and 0.8156 for the OCL. In contrast, the recognition of single-user states becomes more challenging as the power coefficients approach similar values. Under weak PU activation, the baseline LightGBM exhibits a clear reduction in recall together with a noticeable decline in F1-score. Although the OCL method preserves relatively high precision, its recall also decreases, indicating increased difficulty in correctly identifying weak-PU activity. When the power coefficients are closer, the eigen-structures of Class 1 and Class 2 become more similar, thereby increasing mutual confusion between these two single-user hypotheses. Under strong PU activation, the baseline model achieves moderate improvements in recall while the OCL method maintains high precision indicating more discriminative decision regions. For the superposed state, the OCL method benefits from clearer eigenvalue separation under balanced power allocation, achieving a recall of 0.8574 and an F1-score of 0.6972, outperforming the baseline model. While the model can reasonably separate the four sensing states overall, it remains unreliable in recognizing strong-PU activity. It can be noticed that the baseline LightGBM training objective treats all misclassification errors equally and thus prioritizes overall accuracy rather than detection-critical PU states. Consequently, strong-PU samples are often misclassified into other PU-active hypotheses.

On the other hand, the OCL method outperforms the overall performance of the baseline LightGBM classifier, achieving an accuracy of 0.6872 with a macro F1-score of 0.6700 and a weighted F1-score of 0.6681 when $[eqn]$ and $[eqn]$ . Under the idle channel state, the OCL method shows a recall of 0.9739, indicating an improvement achieved by penalizing missed PU activity and reducing false idle declarations relative to the baseline LightGBM. When only weak PU is active, both precision and recall improve, reflecting enhanced separability of weak-PU activity. The superposition state also benefits from a higher recall and F1-score, suggesting more robust discrimination of PU-active conditions under NOMA transmission. Under the strong-PU-only state, the OCL method slightly improves the recall of baseline LightGBM, where the recall is 0.3604. When $[eqn]$ and $[eqn]$ , the baseline LightGBM achieves an overall accuracy of 0.6528 with a macro F1-score of 0.6416 and a weighted F1-score of 0.6387, while the OCL method improves the overall sensing performance, reaching an accuracy of 0.6984 with a macro F1-score of 0.6842 and a weighted F1-score of 0.6820. Overall, the OCL method improves the calibration of PU-active decision regions and prioritizes detection-critical classes, whereas the baseline LightGBM optimizes generic classification accuracy.

4.3. Comparison of NOMA-Based Spectrum Sensing Performance

Second, we evaluate the NOMA-based spectrum sensing performance of the proposed OCL method in comparison with feature-based spectrum sensing (FBSS) and eigen-based spectrum sensing (EBSS). As described in [20], conventional ML-based FBSS employs cyclostationary features with logistic regression (FBSS-LR) to perform spectrum sensing under NOMA-based PU signal superposition. To determine the effectiveness of the OCL method employing eigenvalue-based features, we extend the concept of conventional FBSS by incorporating the proposed cost-sensitive and objective-driven learning strategy with cyclostationary features (FBSS-OCL) to demonstrate the suitability of learning-based feature designs for NOMA spectrum sensing. On the other hand, EBSS is realized using eigen-based features with a random forest classifier (EBSS-RF).

In Table 2, the $[eqn]$ of NOMA-based sensing methods under two conditions of power coefficients are evaluated over PU–SU distances from 10 to 150 m. When $[eqn]$ and $[eqn]$ , OCL, EBSS-RF, and 1DCNN achieve $[eqn]$ = 1 at short distances of 10–20 m, whereas FBSS-LR drops sharply at 20 m, indicating weak robustness under moderate attenuation. FBSS-OCL improves over FBSS-LR at 20 m but deteriorates rapidly beyond 30 m and becomes extremely low after 60 m since it is sensitive to the path loss effect, leading to unstable sensing performance in the NOMA-based transmission. Although the 1DCNN shows slightly higher $[eqn]$ than OCL between 30 and 80 m, the OCL preserves relatively higher and more stable detection behavior compared to the others at longer distances beyond 90 m. When $[eqn]$ and $[eqn]$ , the degradation becomes more significant in the mid- to long-distance region. Although OCL, 1DCNN, and EBSS-RF achieve perfect detection at 10–20 m, EBSS-RF performance deteriorates more rapidly beyond 50–60 m because the eigenvalue separation between weak and strong PU states decreases, thereby weakening purely statistical eigenvalue-based discrimination. In the distance range of 30–80 m, OCL and 1DCNN maintain superior and more stable performance relative to FBSS baselines, while EBSS-RF exhibits noticeable instability at larger distances. Beyond 100 m, detection probabilities of all methods decrease significantly. It can be mentioned that the OCL method offers strong nonlinear discrimination for eigen-feature patterns based on LightGBM classifier. By using cost-sensitive weighting, the priority of PU-active states is recognized, leading to the avoidance of missed detection.

As shown in Table 3, the $[eqn]$ of all methods are compared. When $[eqn]$ and $[eqn]$ , 1DCNN and FBSS-LR exhibit a consistently high false-alarm level across all distances, indicating the LR-based FBSS decision rule tends to over-declare PU activity and lacks effective false-alarm control under NOMA transmission. Notably, 1DCNN suffers from the temporal representations of the received signal since it may shift the noise–PU region toward more aggressive PU declarations. On the other hand, OCL consistently maintains a low and stable $[eqn]$ in the range of 0.050–0.059, reflecting conservative and well-regularized decision behavior under noise-dominant conditions. When $[eqn]$ and $[eqn]$ , all methods exhibit improved idle-channel discrimination compared to the configuration of $[eqn]$ and $[eqn]$ since the feature distribution of idle channels is more distinguishable when PU activity occurs. EBSS-RF slightly improves and OCL maintains consistently low and stable $[eqn]$ . However, 1DCNN still shows the worst classification performance in terms of false-alarm behavior. Overall, the results indicate that increasing the power coefficient enhances the contrast between noise and PU-active signal structures, thereby facilitating idle-channel detection. Through LightGBM’s additive boosting mechanism, the OCL method progressively stabilizes the noise–PU boundary in alignment with spectrum sensing objectives. FBSS-OCL achieves the lowest $[eqn]$ among the compared methods, indicating the cost-sensitive objective regulation further suppresses false alarms within the FBSS feature space. EBSS-RF yields moderate $[eqn]$ around 0.10 to 0.12, implying that eigen-statistical features combined with random forest provide reasonable robustness but at a higher false-alarm floor than OCL.

As depicted in Figure 2, when $[eqn]$ and $[eqn]$ , all methods show strong separability between idle and PU-active states at 50 m with an AUC of 1.00. As the distance increases to 90 m and 140 m, the classification performance degrades due to path loss, where OCL consistently maintains superior ROC characteristics and achieves the highest AUC values across all evaluated distances, demonstrating more stable and robust discrimination under increasing attenuation. EBSS-RF maintains relatively stable ROC characteristics at a moderate distance with gradual AUC reduction while FBSS-LR and 1DCNN show significant degradation with low values. As depicted in Figure 3, when $[eqn]$ and $[eqn]$ , all methods still demonstrate strong separability with high AUC values at 50 m. As the distance increases to 90 m and 140 m, OCL consistently maintains the most favorable ROC characteristics and achieves the highest AUC values, while 1DCNN also experiences substantial AUC reduction as distance increases limited capability in learning signal characteristics. EBSS-RF exhibits a more substantial decline in AUC, indicating reduced robustness at extended distances. FBSS-LR shows the most severe AUC degradation while FBSS-OCL achieves moderate AUC improvement over FBSS-LR. Therefore, OCL consistently achieves the highest AUC across all distances, demonstrating the most stable and robust detection performance.

To provide an overall and fair comparison among sensing methods, we adopt the SPOTIS-based ability score as a unified indicator that jointly reflects $[eqn]$ and $[eqn]$ rather than interpreting them separately. Since the SPOTIS method evaluates sensing quality by measuring the normalized distance from an ideal sensing point where lower scores indicate better overall performance, we define an overall sensing ability score as 1- SPOTIS score to obtain a more intuitive metric in which higher values imply stronger performance. The ideal reference is defined by the desired spectrum sensing targets, i.e., the desired maximum $[eqn]$ is 1, desired minimum $[eqn]$ is 0, desired maximum $[eqn]$ is 0.4, and desired minimum $[eqn]$ is 0.

As shown in Figure 4, the ability scores at four representative PU–SU distances of 10, 50, 90, and 140 m are illustrated to capture short-, medium-, and long-range sensing conditions under distance-dependent path loss. As the distance increases, all methods exhibit a decreasing trend due to SNR degradation. As shown in Figure 4a, OCL achieves the highest ability score at 10 m, indicating the most favorable sensing tradeoff under strong received signals. As the distance increases, FBSS-LR suffers the most severe performance degradation while FBSS-OCL offers only a slight robustness improvement. EBSS-RF remains relatively competitive at moderate ranges, reflecting the stability of eigen-statistical features with nonlinear learning. On the contrary, the 1DCNN demonstrates moderate performance at 40 m but experiences noticeable degradation as the distance increases, indicating sensitivity to attenuation. By considering all distances, OCL achieves the highest overall ability score of approximately 0.638, while EBSS-RF, FBSS-OCL, and FBSS-LR provide 0.610, 0.385, and 0.221, respectively. As shown in Figure 4b, OCL maintains the strongest overall ability score under $[eqn]$ and $[eqn]$ , demonstrating consistent robustness across propagation variations. EBSS-RF remains the second-best performer at moderate ranges but shows performance reduction at extended distances. Even if the 1DCNN achieves competitive results at 50 m, its performance significantly degrades at 90 m and 140 m. FBSS-OCL slightly outperforms FBSS-LR, yet both methods remain significantly lower than OCL and EBSS-RF.

Based on the results, when the signal pattern becomes more difficult to discriminate since the distance is over 90 m, the OCL method improves the SPOTIS score by 6.26% compared to EBSS-RF when $[eqn]$ and $[eqn]$ . Under $[eqn]$ and $[eqn]$ , EBSS-RF demonstrates improved idle-state discrimination; however, OCL still achieves higher overall ability scores with a 0.9% improvement over EBSS-RF, highlighting its effectiveness in addressing the performance tradeoff issue through the objective-driven cost-sensitive design of OCL. It can be mentioned that insufficient decision boundary of LR leads to the weakest performance among others. Once the OCL maintains the strongest ability score across the considered distances, the proposed method provides more reliable NOMA-based spectrum sensing under propagation variations.

By employing the leaf-wise boosting structure of the LightGBM classifier, the proposed method can effectively distinguish overlapping eigen-feature patterns caused by path loss and PU signal superposition, while cost-sensitive weighting prioritizes PU-active recognition. Moreover, the objective-driven tuning selects the best hyperparameters and weights according to the spectrum sensing constraints. Therefore, the proposed OCL achieves the highest ability scores by offering stronger nonlinear discrimination, whereas logistic regression is constrained by linear separability and random forest focuses on generic impurity reduction without directly controlling detection-oriented behavior. It can be mentioned that the difficulty in discriminating strong-PU states mainly stems from feature overlap under dynamic NOMA transmission. Strong-PU conditions may exhibit eigenvalue and energy patterns similar to superposed or moderate-power states, particularly in the presence of fading and noise variations. Consequently, the decision boundaries become less separable in the feature space, increasing the risk of misclassification.

4.4. Computational Complexity Analysis

Finally, we compare the inference-time computational complexity of OCL against FBSS-LR, EBSS-RF, and 1DCNN using Big O notation since the performance of spectrum sensing is primarily constrained by online decision latency. Then, the comparison focuses exclusively on inference rather than offline training.

During inference, FBSS extracts cyclostationary features by processing the received signal in the frequency domain across multiple cyclic frequencies while the LR classifier is used to determine channel state. Once the FBSS feature dimension is small, the overall inference complexity of FBSS-LR can be expressed by

$$[eqn]$$

where $[eqn]$ is the number of cyclic frequencies.

For EBSS-RF, inference consists of two stages including eigenvalue-based feature extraction and RF classification. The eigen-based feature extraction employs a fixed snapshot length and block-wise covariance estimation; consequently, the inference complexity per sensing window scales linearly with the signal length. Meanwhile, the RF classifier performs inference by traversing each decision tree. Therefore, the overall inference complexity of EBSS-RF is dominated by

$$[eqn]$$

where $[eqn]$ is the number of trees and $[eqn]$ is the maximum tree depth.

Similar to EBSS-RF, the proposed OCL extracts the received signal features by employing eigenvalue decomposition, while the LightGBM model is used as the classification method. During inference, each tree is traversed from the root to a leaf node requiring a depth-limited sequence of comparisons. Therefore, the overall inference complexity can be expressed as

$$[eqn]$$

where $[eqn]$ is the number of decision trees.

For the 1DCNN, inference is composed of two main stages, including convolutional feature extraction over the IQ sequence and classification using the neural network. The inference consists of convolutional feature extraction followed by a global pooling and a linear classification layer. Thus, the inference-time computational complexity per sensing window can be approximated as

$$[eqn]$$

where $[eqn]$ is the base channel width and $[eqn]$ is the number of convolutional blocks.

It can be mentioned that the FBSS-LR approach relies on FFT-based cyclostationary feature extraction, which incurs a higher computational cost during inference. In contrast, both EBSS-RF and the proposed OCL exhibit linear-time feature extraction complexity due to the use of eigenvalue-based features with a fixed snapshot length. While EBSS-RF achieves classification performance through ensemble diversity using a large number of independently trained trees, OCL exploits boosting-based representation efficiency, enabling improved detection performance with fewer model components. For the 1DCNN, inference is entirely driven by convolutional feature extraction over the raw IQ sequence, followed by global pooling and a linear classifier. Unlike the tree-based methods whose complexity grows primarily with signal length and tree depth, the computational cost of 1DCNN is dominated by convolutional operations. Therefore, compared to EBSS-RF and OCL, the 1DCNN generally requires higher inference-time computational resources.

5. Conclusions

In this paper, we propose an OCL method for NOMA-enabled cognitive radio systems which is explicitly designed to address spectrum sensing under superposed unequal-power PU signals and independent PU activity. To address the performance tradeoff of existing machine-learning–based sensing approaches, OCL formulates spectrum sensing as a constraint-aware multiclass learning problem, while others rely on conventional binary assumptions without considering spectrum sensing constraints. By jointly optimizing LightGBM hyperparameters and cost-sensitive loss weights, the proposed method directly aligns classifier training with detection objectives while leveraging the strong discriminative capability of eigenvalue-based features. Comprehensive evaluations using an overall sensing ability score based on the SPOTIS method demonstrate that OCL consistently achieves superior performance across all path loss conditions, confirming its effectiveness and robustness for dynamic NOMA spectrum sensing.