BeamNet: Unsupervised Beamforming for ISAC Systems Under Imperfect CSI

TL;DR

This paper introduces BeamNet, an unsupervised deep learning framework for beamforming in ISAC systems with imperfect channel information.

Contribution

BeamNet is a novel unsupervised learning framework for ISAC beamforming that learns optimal trade-offs without labels or optimization solvers.

Findings

BeamNet reproduces analytical Pareto-optimal beamforming solutions under Rayleigh fading with perfect CSI.

BeamNet outperforms closed-form beamformers under imperfect CSI and recovers part of the performance loss due to channel estimation errors.

The framework is robust to distribution mismatch between training and test channels in Nakagami-m and Rician fading.

Abstract

Integrated sensing and communication (ISAC) is expected to be a key enabler for future wireless networks, improving spectral and hardware efficiency by jointly performing radar sensing and wireless communication within a unified framework. This paper proposes BeamNet, an unsupervised deep learning framework for transmit beamforming in dual-function radar-communication systems operating over general fading with imperfect channel state information (CSI). BeamNet maps noisy estimates of the communication and sensing channels to a transmit beamforming vector and is trained end-to-end by maximizing a weighted sum of the communication rate (CR) and sensing rate (SR), thereby learning the CR–SR Pareto frontier without beamforming labels or embedded optimization solvers. Using Rayleigh fading with perfect CSI, we first show that BeamNet reproduces the analytical Pareto-optimal beamforming solutions. We then use BeamNet to characterize, for Nakagami-m and Rician fading, the CR–SR trade-off across a range of fading parameters, and to assess robustness under distribution mismatch between training and test channels. Finally, under imperfect CSI, we demonstrate that BeamNet yields CR–SR trade-offs that are consistently sandwiched between the perfect-CSI and mismatched analytical baselines, outperforming the closed-form beamformer applied to imperfect CSI and recovering part of the performance loss caused by channel estimation errors. These results indicate that unsupervised learning offers a flexible and robust approach to ISAC beamforming in fading environments with imperfect channel knowledge.

Figures

Full text

1. Introduction

Next-generation wireless networks are increasingly expected not only to communicate but also to sense their surroundings in real time. Examples include vehicles detecting obstacles while exchanging safety messages and base stations tracking users and objects while serving data traffic. Integrated sensing and communication (ISAC) has emerged as a key paradigm to meet this demand by enabling joint radar sensing and wireless communication over shared spectral and hardware resources [1]. By tightly integrating these functions, ISAC can improve spectral efficiency, reduce latency, and simplify hardware, making it attractive for applications such as autonomous driving, smart transportation, urban infrastructure monitoring, and industrial automation. Unlike conventional frequency-division schemes that allocate separate resources to communication and sensing, ISAC leverages advanced signal processing and joint transceiver design to unify these tasks within a common framework [2].

Beamforming lies at the heart of ISAC, directly shaping both the communication rate (CR) and sensing rate (SR) (In this work, we adopt the SR, as it not only provides an information-theoretic metric directly comparable to the CR for unified CR–SR trade-off analysis, but also captures the fundamental information acquisition capability of the sensing process in ISAC systems). Striking the right balance between these inherently competing objectives is a fundamental yet challenging task, particularly as channel models and hardware constraints become more sophisticated. Existing research has largely focused on deriving analytical beamforming solutions under specific fading distributions. Notably, closed-form Pareto-optimal solutions are currently available only under perfect channel state (CSI) [3], while the practically relevant case of imperfect CSI remains analytically intractable. This gap is further compounded when considering more general fading environments, where exact distributional assumptions may be unknown or mismatched.

These limitations motivate a data-driven alternative for ISAC beamforming in non-ideal conditions. Rather than relying on closed-form designs that assume perfect CSI and specific fading laws, a deep learning (DL)-based approach can learn the mapping from observed (imperfect) channel realizations to beamforming vectors directly from data and then apply it in real time with low online complexity. Unsupervised learning is particularly appealing in this context: by optimizing task-oriented objectives (here, CR and SR) directly, it bypasses the need for explicit beamforming labels or embedded optimization solvers and can naturally accommodate diverse fading environments [4].

In this paper, we introduce BeamNet, an unsupervised learning framework for transmit beamforming in dual-function radar-communication ISAC systems operating over general fading channels with imperfect CSI. BeamNet takes noisy estimates of the communication and sensing channels as input and outputs a beamforming vector that balances CR and SR. The model is trained end-to-end using an unsupervised loss that maximizes a weighted sum of CR and SR, thereby tracing the CR-SR Pareto frontier without requiring hand-crafted labels or closed-form beamforming rules.

The key contributions of this work are as follows:

• We propose an unsupervised learning-based beamforming framework for ISAC systems operating under imperfect CSI, formulating beamforming as a multi-objective optimization problem in terms of CR and SR.
• The proposed approach is inherently distribution-agnostic at the loss level and can be trained on data from general fading channels (BeamNet is not tied to any specific distribution and only requires sample channels.) (e.g., Nakagami-m, Rician), enabling robust performance even under distribution mismatch between training and test channel statistics.
• Using a single trained model and by sweeping the loss weights, BeamNet efficiently characterizes the Pareto-optimal CR-SR trade-off across fading environments, providing a flexible tool to study ISAC performance frontiers beyond analytically tractable regimes.

Using numerical simulations, we first benchmark BeamNet against the analytical Pareto-optimal beamforming solution in [3] under Rayleigh fading with perfect CSI, and show that the learned CR-SR boundary closely reproduces the theoretical one. We then employ BeamNet to characterize the CR-SR Pareto trade-off under Nakagami-m fading and to assess robustness under distribution mismatch between training and testing channel statistics. Finally, under imperfect CSI, we demonstrate that BeamNet yields CR-SR trade-offs that lie between the analytical perfect-CSI upper bound and its imperfect-CSI counterpart, while consistently outperforming the closed-form beamformer obtained by directly applying the perfect-CSI solution to imperfect CSI estimates. These results indicate that unsupervised learning offers a practical and robust approach to ISAC beamforming in realistic fading environments with imperfect channel knowledge.

2. Related Work

Deep learning (DL) has emerged as a powerful tool for tackling key ISAC challenges, including mutual interference, high optimization complexity, and the need for real-time adaptability [5,6,7]. In [5], the authors present DL-based optimization techniques to maximize the weighted sum of sensing and communication rates in uplink 6G networks, achieving substantial real-time gains over conventional optimization methods. In [6], an end-to-end autoencoder-based framework jointly optimizes sensing accuracy, angle estimation, and communication quality, demonstrating resilience to hardware impairments, while [7] extends this to a semi-supervised design that reduces the reliance on labeled data. Predictive beamforming has also been explored; for example, ref. [8] employs a convolutional LSTM-based strategy for vehicular ISAC, predicting beamformers from historical CSI to reduce signaling overhead with performance close to ideal CSI. Beyond conventional communication-centric settings, Li et al. [9] propose an ISAC-based UWB system for fall detection (FallDR), showcasing ISAC’s potential for practical sensing applications.

Beyond supervised and end-to-end autoencoder-based designs, several recent works have explored unsupervised learning for ISAC beamforming. Ye et al. propose a lightweight IBF-Net architecture for RIS-aided ISAC, where image-shaped channel samples are processed, and an unsupervised loss jointly accounts for communication and sensing performance without requiring labeled beamformers [10]. Elrashidy et al. develop an unsupervised teacher-student framework for distributed beamforming in cell-free ISAC, dynamically balancing sensing signal-to-noise ratio ( $[eqn]$ ) and communication signal-to-noise-plus-interference ratio ( $[eqn]$ ) while reducing fronthaul overhead and computational complexity [11]. Temiz and Masouros present an unsupervised deep learning-based ISAC precoder that optimizes joint sensing and communication performance without requiring labeled precoders [4]. Together, these studies demonstrate the potential of label-free optimization for joint sensing and communication. However, they are tailored to specific architectures and do not explicitly characterize the CR-SR Pareto frontier under imperfect CSI.

Learning-based ISAC beamforming has also been investigated for large-scale and cell-free architectures. Demirhan and Alkhateeb propose a graph neural network (GNN)-based framework for cell-free massive MIMO ISAC, showing that heterogeneous GNNs can scale with the number of access points while approaching optimization-based performance without retraining for each topology [12]. This highlights the scalability of ML-driven beamforming, though their approach primarily relies on (near-)supervised targets derived from conventional optimization and pays limited attention to robustness under channel estimation errors.

Robust ISAC transceiver and beamforming designs under imperfect CSI have also attracted significant interest, predominantly using optimization-based (non-learning) methods. Zhang et al. investigate robust transceiver design for covert ISAC with imperfect CSI using bounded and probabilistic error models, and derive worst-case and outage-constrained solutions via S-procedure and Bernstein-type inequalities within an alternating optimization framework [13]. Chen et al. study robust beamforming for secure near-field ISAC systems, maximizing worst-case sensing beam pattern gain while satisfying SINR constraints for legitimate users and limiting information leakage under CSI uncertainty [14]. These methods provide strong robustness guarantees but depend on accurate error bounds and require repeatedly solving complex optimization problems, which can hinder real-time deployment, especially under distribution mismatch.

On the fundamental limits side, Guo et al. characterize the Pareto boundary of the CRB-rate tradeoff in ISAC with arbitrary input distributions using an information-theoretic (non-learning) framework [15], offering a complementary perspective and emphasizing the importance of explicitly tracing performance frontiers (CRB-rate or CR-SR) rather than optimizing a single objective. In a related direction, Zhang et al. introduce the Semi-Integrated-Sensing-and-Communication (Semi-ISAC) framework, evolving from OMA to NOMA and analyzing ergodic communication rate and radar estimation information rate over Nakagami-m fading channels [16]. This further motivates studying ISAC strategies over generalized fading environments beyond the standard Rayleigh model.

Compared with the above literature, the proposed BeamNet differs in three key aspects. First, unlike supervised, semi-supervised, or optimization-aided learning approaches that rely on target beamformers or near-optimal solutions, BeamNet is trained with an unsupervised objective directly defined in terms of the communication rate and sensing rate. This eliminates the need for explicit beamforming labels or embedded optimization solvers. Second, in contrast to robust but purely optimization-based ISAC designs and fundamental-limit analyses that require explicit error models or tractable fading assumptions, BeamNet is inherently distribution-agnostic: the same framework operates over general fading channels, including Nakagami-m, without relying on closed-form beamforming solutions or restrictive channel statistics. Third, while existing unsupervised ISAC beamforming methods demonstrate the promise of label-free optimization in specific architectures, they do not explicitly characterize the CR-SR Pareto frontier under imperfect CSI. In contrast, BeamNet incorporates imperfect CSI directly into the learning process and is used as a tool to trace the CR-SR Pareto boundary under channel estimation errors, thereby bridging analytically derived ideal-CSI designs and practical, data-driven beamforming in realistic fading and estimation conditions.

Beyond ISAC, related multi-functional designs arise in wireless communications frequently, involving coupled trade-offs such as rate-covertness in covert communications and rate-energy in simultaneous wireless information and power transfer (SWIPT) [17,18]. For example, Kang et al. in [17] study covert transmission limits in random-access networks and characterize achievable covert rates under covertness constraints in the presence of randomly activated overt users. In SWIPT-enabled MC-NOMA, Tang et al. [18] formulate joint power splitting and allocation to maximize data rate subject to energy-harvesting requirements, and additionally develop a learning-based approximation approach to reduce online complexity. While these works address different system models and objectives than ISAC beamforming, they illustrate the broader applicability of learning/optimization frameworks for rate-centric trade-offs. Extending BeamNet to these scenarios would require redefining the task-specific utility functions and constraints and is left for future work.

3. System Setup

We consider an ISAC system as shown in Figure 1, where a dual-function radar communication (DFRC) base station (BS), equipped with M transmit, and N receive antennas, simultaneously serves a single-antenna communication user (CU) and detects a single target. This setup aligns with the model presented in [3], which captures the use of shared hardware resources for joint communication and sensing. In this work, we adopt and build upon this setup to develop our learning-based framework.

Let $[eqn]$ represents the DFRC signal matrix, where L denotes the length of the ISAC frame (consisting of communication symbols and the sensing pulses). From a communication point of view, $[eqn]$ for $[eqn]$ signifies the vector of the l-th data symbol. For sensing, $[eqn]$ corresponds to the sensing snapshot transmitted in the l-th time slot.

We consider the ISAC signal matrices in the form $[eqn]$ , where p denotes the power budget, $[eqn]$ is the normalized beamforming vector and $[eqn]$ represents the unit power data stream intended for the CU with $[eqn]$ .

3.1. Communication Model

The signal received at the CU can be written as follows:

$$[eqn]$$

where $[eqn]$ is the additive noise with $[eqn]$ , and  $[eqn]$ represents the state of the communication channel, which determines the quality of the communication link as shown in Figure 1. The CR, given $[eqn]$ , satisfies

$$[eqn]$$

We model the communication channel estimate to be imperfect according to

$$[eqn]$$

where $[eqn]$ denotes the true (perfect) communication channel and $[eqn]$ represents the channel estimation error.

3.2. Sensing Model

The DFRC BS observes the reflected echo signal at its receiver to sense the target. In particular, in response to $[eqn]$ , the BS receives the echo

$$[eqn]$$

where $[eqn]$ is the noise matrix with each entry having zero mean and unit variance, and  $[eqn]$ represents the target response matrix. The target response matrix is modeled according to

$$[eqn]$$

where $[eqn]$ represents the radar cross section (RCS) of the target, which follows a complex Gaussian distribution $[eqn]$ with $[eqn]$ representing the average reflected signal power. The vectors $[eqn]$ and $[eqn]$ correspond to the receive and transmit array steering vectors, respectively, as a function of the target angle $[eqn]$ [3,19].

We model the antenna aperture at the BS using a uniform linear array with half-wavelength spacing, which yields

$$[eqn]$$

We adopt the Swerling-I model [20] to compute the sensing rate, which assumes that a target’s RCS remains constant within each pulse transmission but varies stochastically from pulse to pulse (according to the assumed distribution for $[eqn]$ ). Under this model, for a given $[eqn]$ and using matched-filter receiver beamforming matched to $[eqn]$ , the sensing rate $[eqn]$ can be approximated by

$$[eqn]$$

where $[eqn]$ is the average sensing $[eqn]$ and $[eqn]$ is the sensing channel.

Remark (SR vs. radar KPIs): While the sensing rate $[eqn]$ in Equation (7) is an information-theoretic metric (rather than a detector-specific KPI), it is closely aligned with classical sensing performance measures under the adopted Swerling-I and matched-filter processing model. In particular, Equation (7) is a monotone function of the average sensing $[eqn]$ , $[eqn]$ , so that maximizing $[eqn]$ is equivalent to maximizing $[eqn]$ for fixed $[eqn]$ . Since standard radar detection probability at a fixed false-alarm rate (Neyman–Pearson testing) and typical estimation accuracy trends (e.g., CRB/RMSE behavior) improve monotonically with sensing $[eqn]$ , improvements in $[eqn]$ translate into improved detection/estimation capability once a specific detector/estimator and operating point are chosen. Moreover, the mapping is explicit: $[eqn]$ , enabling direct conversion from the CR-SR frontier to $[eqn]$ -based radar KPIs when desired.

The target’s angle $[eqn]$ is estimated by high-resolution direction-of-arrival algorithms, such as MUSIC or Capon, which process the received echo to extract phase differences across antenna elements in the steering vector $[eqn]$ [21]. We assume the target angle estimation is imperfect, modeled according to

$$[eqn]$$

where $[eqn]$ denotes the true target angle and $[eqn]$ represents the estimation error in the target angle.

Given the above ISAC framework, we aim to analyze the joint sensing and communication performance of the BS.

Note that both $[eqn]$ and $[eqn]$ are influenced by the beamforming vector $[eqn]$ . Finding an optimal $[eqn]$ that maximizes both $[eqn]$ and $[eqn]$ is a multi-objective optimization problem. There are commonly three design paradigms to investigate this problem: (i) communication-centric (C-C), (ii) sensing-centric (S-C), and (iii) Pareto-optimal design.

Despite the elegant structure of the above ISAC model, obtaining closed-form beamforming solutions is generally tractable only under idealized assumptions. For instance, when assuming perfect CSI, ideal synchronization, and absence of hardware impairments, the Pareto-optimal beamforming strategies and the resulting CR–SR trade-offs can be analytically derived, as demonstrated in [3,22]. However, when departing from this ideal regime, such as in the presence of imperfect channel estimation or hardware impairments, the resulting optimization problem becomes analytically intractable. In particular, robust beamforming under imperfect CSI often involves computing expectations over uncertain or mismatched error statistics, and simultaneously optimizing the conflicting CR and SR objectives under these uncertainties further complicates the overall design. As a result, direct extensions of perfect-CSI analytical solutions often yield suboptimal or overly conservative beamformers when applied in realistic conditions.

These limitations motivate a data-driven alternative for beamforming under non-ideal conditions. Rather than relying on closed-form analytical solutions that assume perfect CSI, a DL–based approach can learn the mapping from observed imperfect channel realizations to beamforming vectors directly from data. This strategy offers three major advantages. First, it can be trained to remain robust against CSI imperfections by exposing the model to estimation errors during training. Once trained, it enables real-time beamformer computation with low online complexity. Last, it is well-suited for fast-varying ISAC scenarios in diverse fading environments. In this paper, we introduce BeamNet, an unsupervised DL framework that directly optimizes a CR–SR objective, learns optimal beamforming under imperfect CSI, and generalizes across a broad range of fading conditions beyond the capabilities of existing analytical designs.

4. DL-Based Beamforming Framework

4.1. Deep Learning Model Architecture

First, we provide an overview of our BeamNet architecture, as illustrated in Figure 2, that jointly maximizes CR and SR. In our proposed architecture, the input data and the communication channel $[eqn]$ and sensing channel $[eqn]$ go through a flattening operation to divide into real and imaginary parts to process complex-valued inputs. The model predicts $[eqn]$ outputs, which are then mapped to the real and imaginary components of the beamforming vector $[eqn]$ .

The architecture employs fully connected layers with Rectified Linear Unit (ReLU) activations to map the concatenated input features to the desired beamforming vector. The model processes the input through a hierarchy of layers, progressively learning the non-linear relationship between the channels and the beamforming weights.

4.2. Deep Learning Model Training

We use the following loss function to train the BeamNet

$$[eqn]$$

where $[eqn]$ controls the CR–SR trade-off: $[eqn]$ yields a communication-centric model (maximizing $[eqn]$ ), $[eqn]$ yields a sensing-centric model (maximizing $[eqn]$ ), and intermediate values balance the two objectives.

BeamNet takes the imperfect CSI estimates $[eqn]$ as input. During training on synthetic data, the loss in Equation (9) is evaluated using the underlying true channels $[eqn]$ to provide stable gradients and to explicitly learn a mapping from noisy CSI to a beamformer that maximizes the true CR and SR.

To facilitate reproducibility, Table 1 summarizes the training and implementation settings. We generate $[eqn]$ independent channel realizations following the fading model under consideration (Nakagami-m or Rician), and draw $[eqn]$ to construct the sensing channel. The dataset is split into 85% for training and 15% for testing.

For each scalarization weight $[eqn]$ , the network parameters are trained for 500 epochs using Adam with learning rate $[eqn]$ and full-batch updates over the training set. We use PyTorch’s default Kaiming-uniform initialization for linear layers and do not apply explicit $[eqn]$ / $[eqn]$ regularization or weight decay. The CR-SR Pareto boundary is obtained by sweeping $[eqn]$ over 101 uniformly spaced values in $[eqn]$ (step size $[eqn]$ ). Unless otherwise stated, all real-valued tensors and parameters use 32-bit floating point precision (FP32), while complex-valued channel variables are stored as complex64. The resulting model requires approximately $[eqn]$ FLOPs per inference (per channel realization) and achieves an average inference latency of $[eqn]$ ms $[eqn]$ ms on an NVIDIA GeForce RTX 2080 Ti (CUDA 12.7).

In Section 5.4, we evaluate robustness by varying the channel estimation error variance $[eqn]$ and the target angle estimation error variance $[eqn]$ over the range $[eqn]$ and reporting the corresponding CR-SR trade-offs.

Remark-Synthetic dataset generation and reproducibility: All training/test samples are generated using Monte Carlo simulation from the statistical models specified in Section 3 and Section 5. For each realization, we draw a true communication channel $[eqn]$ from the stated fading distribution (e.g., Nakagami-m: $[eqn]$ with $[eqn]$ and $[eqn]$ ), draw $[eqn]$ to construct the sensing channel, and then generate imperfect CSI using the estimation-error models with variances $[eqn]$ and $[eqn]$ . Thus, given the reported parameter settings (Table 1 and Section 5), the full dataset and all results are reproducible without relying on any external dataset.

5. Experimental Results

In this section, we present numerical results evaluating the performance of the proposed ISAC BeamNet beamformer. The proposed framework is inherently independent of the underlying channel fading distribution. To validate its accuracy, we first consider a Rayleigh fading channel and benchmark our results against the analytical solution in [3] under perfect channel conditions. We then extend the evaluation to Nakagami-m fading under perfect channel conditions.

The Nakagami-m fading channel was considered in this paper because it offers a versatile and general framework for modeling a wide variety of wireless propagation conditions. Specifically, it reduces to Rayleigh fading for $[eqn]$ , can approximate Rician fading for $[eqn]$ , and captures more severe fading than Rayleigh when $[eqn]$ [23]. This flexibility allows it to represent diverse practical scenarios such as urban microcells, indoor channels, or obstructed environments, where different fading severities may occur. Owing to these properties, Nakagami-m is widely used in the literature as a fading channel model. In our work, its inclusion enables us to evaluate the robustness of the proposed method across a broad range of fading conditions, rather than restricting the analysis to a single channel model.

To that end, the channel fading vector $[eqn]$ is modeled as a multi-dimensional Nakagami-m random vector, where each entry $[eqn]$ is independently drawn from a Nakagami-m distribution with shape parameter m and scale parameter $[eqn]$ , expressed as $[eqn]$ where $[eqn]$ and $[eqn]$ .

We present the Pareto-optimal CR-SR trade-off boundaries for both Rayleigh fading and general Nakagami-m fading scenarios. Unless otherwise stated, the main simulation parameters are: $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , and  $[eqn]$ .

After benchmarking our results with respect to the theoretical optimum, we present numerical evaluations under imperfect CSI conditions, considering Nakagami-m fading. To this end, we set $[eqn]$ 0.05 and 0.1 and demonstrate that the proposed DL-based framework outperforms the analytical framework under imperfect channel conditions.

5.1. Benchmark BeamNet Under Rayleigh Fading (m=1) with Perfect CSI

In this section, we consider perfect channel conditions, where $[eqn]$ and $[eqn]$ .

5.1.1. Communication-Centric Design

In this scenario, the optimum beamforming vector $[eqn]$ is given by [3],

$$[eqn]$$

Using Equation (10) as a benchmark giving the optimum beamforming vector for communication, we provide the performance of BeamNet in Figure 3a. The results show that BeamNet attains a communication rate almost identical to the analytical optimum reported in [3] under the C-C design.

5.1.2. Sensing-Centric Design

In this scenario, the optimum beamforming vector $[eqn]$ is given by [3],

$$[eqn]$$

Using Equation (11) as a benchmark giving the optimum beamforming vector for sensing, we provide the performance of BeamNet in Figure 3b. The results show that BeamNet attains a sensing rate almost identical to the analytical optimum reported in [3] under the S-C design.

5.1.3. Pareto Optimal Design

In this scenario, the optimum beamforming vector $[eqn]$ is obtained by solving

$$[eqn]$$

where $[eqn]$ is a rate-profile parameter and R is the optimization objective, which represents a trade-off between communication rate and sensing rate [3,22,24].

As presented in [3], the Pareto boundary of the rate region can be attained using the beamforming vector,

$$[eqn]$$

where $[eqn]$ and the weighting factor $[eqn]$ varies within the range $[eqn]$ .

To discover the Pareto boundary achieved by BeamNet, we increment the hyperparameter $[eqn]$ with a step size of $[eqn]$ . Figure 3c shows that the Pareto boundary obtained using the BeamNet closely follows the one obtained with Equation (13).

We further use frequency-division sensing-and-communication (FDSAC) as another benchmark to compare the performance achieved by BeamNet. The FDSAC scheme is adopted as a conventional baseline, serving to illustrate the performance gain achievable through integration in ISAC systems. For the FDSAC case, the CR and the SR are given by $[eqn]$ and $[eqn]$ , where $[eqn]$ and $[eqn]$ are the fractions of power and bandwidth dedicated to the communication task, respectively [3].

In summary, Figure 3 shows that BeamNet consistently achieves performance close to the analytical optimum under Rayleigh fading across all three considered cases, confirming its effectiveness.

5.1.4. Impact of the Number of Layers (Ablation Study)

To isolate the impact of network depth, we conduct a depth ablation study in Figure 3c. The performance of BeamNet improves and approaches the analytical Pareto boundary in [3] as the number of hidden layers (and thus model capacity) increases. Specifically, Figure 3c compares the full BeamNet with two deliberately reduced-capacity variants: BeamNet-1, obtained by removing the $[eqn]$ hidden layer, and BeamNet-2, obtained by removing both the $[eqn]$ and $[eqn]$ hidden layers. These variants are included only as ablation baselines (not as recommended architectures). Hence, their lower performance is expected and serves to quantify the depth-performance trade-off. The results show that increasing depth systematically improves the learned CR-SR boundary (BeamNet-2 → BeamNet-1 → BeamNet). Based on extensive experiments, we adopt the architecture in Figure 2, which delivers near-optimal performance with moderate computational complexity. Increasing width or depth beyond this point yields negligible gains at higher computational cost.

5.2. Performance Under Nakagami-m Fading with Perfect CSI

Next, we examine the impact of Nakagami-m fading on BeamNet’s performance by analyzing the $[eqn]$ – $[eqn]$ trade-off under different fading conditions. Specifically, we evaluate four shape parameters, $[eqn]$ , while fixing the spread parameter at $[eqn]$ .

Remark-Why are both CR and SR reported in Figure 4 and Figure 5 below: Although the C-C and S-C designs optimize a single objective (CR-only or SR-only), the resulting transmit beamformer $[eqn]$ simultaneously shapes both links, and therefore induces a well-defined value for the other metric as well. For completeness, Figure 4 and Figure 5 report both the optimized rate and the corresponding non-optimized rate to quantify the dual-task performance at these two extreme operating points. These extreme points also serve as endpoint references that help interpret the Pareto-optimal CR-SR trade-offs reported in Section 5.2.3.

5.2.1. Communication-Centric Design

Figure 4 illustrates the communication and sensing rates when the model is trained for the C–C design under Nakagami-m fading with $[eqn]$ . In this configuration, BeamNet optimizes the beamforming vector solely to maximize the CR, without explicitly accounting for the SR. For instance, at  $[eqn]$ and $[eqn]$ dB, the CR reaches $[eqn]$ bps/Hz, while the SR is $[eqn]$ bps/Hz. As expected, the CR improves with increasing m; however, the incremental improvement becomes less pronounced as the fading condition transitions from severe to nearly non-fading.

5.2.2. Sensing-Centric Design

Figure 5 shows the communication and sensing rates when the model is trained for the S–C design under Nakagami-m fading with $[eqn]$ . In this setting, BeamNet optimizes the beamforming vector exclusively to maximize the SR, without explicitly considering the CR. For example, at  $[eqn]$ and $[eqn]$ dB, the SR reaches $[eqn]$ bps/Hz, while the CR is $[eqn]$ bps/Hz. Compared to the C–C design results in Figure 4 for the same m and $[eqn]$ , the SR is higher; however, the CR is lower.

5.2.3. Pareto Optimal Design

Figure 6 presents the Pareto boundaries under Nakagami-m fading, where BeamNet jointly optimizes the beamforming vector to maximize both the CR and the SR. For example, at  $[eqn]$ and $[eqn]$ dB, when operating in a communication-centric regime (small $[eqn]$ ), the CR reaches approximately $[eqn]$  bps/Hz with an SR of about $[eqn]$  bps/Hz, which is consistent with the C-C design in Figure 4. In contrast, for a sensing-centric setting ( $[eqn]$ ), the SR attains its maximum value of $[eqn]$  bps/Hz with a corresponding communication rate of $[eqn]$  bps/Hz, in agreement with the S-C design in Figure 5. Moreover, for the same m and $[eqn]$ , intermediate $[eqn]$ operating points along the Pareto boundary can be achieved; for instance, $[eqn]$  bps/Hz with $[eqn]$  bps/Hz (e.g., $[eqn]$ ) and $[eqn]$  bps/Hz with $[eqn]$  bps/Hz (e.g., $[eqn]$ ).

BeamNet works with different transmit and receive antenna configurations. To demonstrate this, we evaluate a system with 16 transmit antennas and 4 receive antennas under Nakagami-m fading with $[eqn]$ and $[eqn]$ , as shown in Figure 7. The results show that BeamNet achieves performance very close to the optimal rates. This confirms that BeamNet generalizes well across different antenna configurations.

5.3. Model Robustness Test and Generalization

Next, we evaluate the BeamNet robustness when the training data distribution does not exactly follow the test data distribution. Recall that we train the model using 85,000 data points, and we use 15,000 data points for model testing, both sampled from the same distribution with parameters $[eqn]$ . To evaluate the model under varying channel conditions, we conduct tests using 15,000 data points per set, but with channel parameters $[eqn]$ . In Figure 8, we illustrate that BeamNet performs very close to the trade-off boundary even when the test data distribution differs from the distribution generating training and testing data.

This result demonstrates the robustness of BeamNet to variations in channel statistics between training and testing, and its ability to generalize beyond the training and testing parameters.

5.4. Performance Under Nakagami-m Fading with Imperfect CSI

Next, we analyze the effect of imperfect channel estimation on the CR-SR trade-off. It is important to note that the CR-SR Pareto boundary under imperfect CSI is not available in closed form. Hence, in this part of the paper, we assess the BeamNet performance using the analytical perfect-CSI Pareto boundary as an ideal upper performance bound and the closed-form beamformer computed from imperfect CSI estimates as a mismatched lower performance bound, as explained below.

During training, BeamNet receives noisy $[eqn]$ as input, while the unsupervised loss in Equation (9) is evaluated using the true $[eqn]$ to stabilize learning and encourage robustness against estimation errors. During testing, the model is fed with $[eqn]$ , and the results are compared against the analytical beamforming vector derived from Equation (13) under perfect CSI conditions.

We obtain an upper bound on the BeamNet performance by substituting the perfect CSI $[eqn]$ into Equation (13) (referred to as the Analytical with Perfect CSI bound), and a lower bound on the BeamNet performance by substituting the imperfect CSI $[eqn]$ into Equation (13) (referred to as the Analytical with Imperfect CSI bound).

The reported performance results use single-precision floating point for all real-valued tensors and network parameters, while complex-valued channel variables are stored as complex64 (FP32 real and imaginary parts), as noted above. This is a standard and numerically stable setting for wireless channel modeling. Unless otherwise specified, we set $[eqn]$  dB, $[eqn]$ , $[eqn]$ , $[eqn]$ , and  $[eqn]$ .

Remark: Although we present the main numerical results for a representative baseline configuration to enable direct comparison with analytical references, we also verified BeamNet under larger antenna arrays (e.g., $[eqn]$ ) and under more severe CSI errors beyond the nominal range. The same qualitative CR-SR trends persist, and BeamNet exhibits graceful degradation until the CSI becomes highly unreliable. These results are omitted for brevity.

Figure 9 shows the CR-SR Pareto boundary under Rayleigh fading with $[eqn]$ , where the BeamNet Pareto boundary lies between the Analytical with Perfect CSI upper-bound and the Analytical with Imperfect CSI lower-bound baselines, while remaining visibly closer to the perfect-CSI curve across most operating points. The shaded region represents the $[eqn]$ confidence interval. The figure demonstrates that, even when operating solely with imperfect CSI inputs, BeamNet learns beamforming strategies that are significantly more robust than directly applying the analytical solution under imperfect CSI conditions. For moderate sensing rates around $[eqn]$ bps/Hz, BeamNet achieves a communication rate that is approximately $[eqn]$ higher than the Analytical with Imperfect CSI lower-bound, while remaining within about $[eqn]$ – $[eqn]$ of the Analytical with Perfect CSI upper-bound. This confirms that, even when only imperfect CSI is available at test time, the learned beamformer closely tracks the ideal Pareto boundary and avoids the degradation caused by directly using imperfect CSI in the closed-form solution provided in Equation (13).

From a deployment perspective, we also evaluate a hardware-constrained, low-precision implementation of BeamNet suitable for resource-limited processors and edge accelerators. To obtain a low-complexity model with a reduced memory footprint and efficient integer arithmetic support, we apply Quantization-Aware Training (QAT) and export an INT8 version of the network. Specifically, activations are quantized using per-tensor affine quantization (uint8, 0–255), and weights are quantized using per-channel symmetric quantization (int8, $[eqn]$ to 127). This reduces parameter storage by approximately $[eqn]$ compared to BeamNet FP32 implementation while preserving the same feed-forward inference structure. As shown in Figure 9, the INT8/QAT Pareto boundary closely tracks the full-precision boundary, indicating that the learned beamformer is robust to 8-bit quantization and remains effective under practical hardware constraints. For reference, the model requires approximately $[eqn]$ FLOPs per inference (per channel realization) in both cases.

Figure 10 further illustrates the impact of estimation error variances on the CR-SR trade-off, along with $[eqn]$ conficence intervals (shaded areas). As the error variances $[eqn]$ and $[eqn]$ increase from 0.05 to 0.1, the Analytical with Imperfect CSI curve deviates significantly from the Analytical with Perfect CSI curve in both CR and SR. In contrast, the BeamNet curve exhibits a more graceful degradation and remains consistently sandwiched between the two bounds, staying closer to the upper bound across all considered operating points and error levels. For example, at $[eqn]$ bps/Hz and $[eqn]$ , BeamNet improves the communication rate by approximately $[eqn]$ compared to the mismatched analytical solution, while the gap to the perfect-CSI bound is about 6– $[eqn]$ . These results indicate that the benefit of BeamNet becomes more pronounced as CSI imperfections increase, effectively narrowing a substantial fraction of the performance loss introduced by estimation errors.

In Figure 11, we repeat the same uncertainty-quantification study under Rician-k fading ( $[eqn]$ ), which introduces a Line-of-Sight (LoS) component in addition to scattered multipath. The shaded regions denote the $[eqn]$ confidence intervals of the CR–SR operating points under imperfect CSI. Consistent with Figure 10, BeamNet continues to produce CR–SR trade-offs that remain between the analytical perfect-CSI upper bound and the mismatched analytical imperfect-CSI baseline, while the confidence bands remain relatively tight, indicating stable behavior across random channel and estimation-error realizations.

We next assess the performance of BeamNet across different fading conditions by considering Nakagami-m fading with $[eqn]$ under imperfect CSI with $[eqn]$ . The corresponding CR-SR curves are depicted in Figure 12. In all three cases, the BeamNet curve lies noticeably above the Analytical with Imperfect CSI baseline and remains close to the Analytical with Perfect CSI benchmark. Around a representative operating point with $[eqn]$ bps/Hz, BeamNet achieves a communication rate of about $[eqn]$ , $[eqn]$ , and $[eqn]$ bps/Hz for $[eqn]$ , $[eqn]$ , and $[eqn]$ , respectively, compared to approximately $[eqn]$ – $[eqn]$ bps/Hz for the Analytical with Imperfect CSI design, corresponding to a relative $[eqn]$ gain on the order of 60– $[eqn]$ at the same sensing rate. This behavior demonstrates that the proposed unsupervised framework generalizes reliably across different channel conditions.

To further support the distribution-agnostic nature of the proposed approach, we repeat the imperfect-CSI evaluation under Rician-k fading for different k values in Figure 13. This figure shows the resulting CR–SR Pareto boundaries for $[eqn]$ at $[eqn]$ dB with $[eqn]$ . Consistent with the Nakagami-m case in Figure 12, BeamNet continues to yield trade-off curves that track the perfect-CSI benchmark substantially more closely than the mismatched analytical imperfect-CSI baseline, indicating that the same robustness trends persist in LoS-dominant propagation environments.

6. Conclusions and Future Work

In this paper, we introduced BeamNet, an unsupervised deep learning framework for transmit beamforming in ISAC systems operating over general fading channels with imperfect CSI. BeamNet maps noisy estimates of the communication and sensing channels to a beamforming vector and is trained end-to-end using a task-oriented loss that directly combines the communication rate (CR) and sensing rate (SR). By sweeping the loss weight, a single trained model yields the CR-SR Pareto frontier without requiring labeled beamformers or embedded optimization solvers.

Our numerical results show that, under Rayleigh fading with perfect CSI, BeamNet accurately reproduces the analytically derived Pareto boundary and matches the communication- and sensing-centric benchmarks, thereby validating the proposed learning framework against known optimal solutions. Extending to Nakagami-m fading, BeamNet efficiently traces CR-SR trade-off curves across a range of fading parameters and remains robust when the test-channel statistics differ from those seen during training, demonstrating its ability to generalize beyond a single channel law. Under imperfect CSI, BeamNet is trained and evaluated with noisy channel estimates and yields CR-SR trade-offs that are consistently sandwiched between the analytical perfect-CSI upper bound and the analytical beamformer applied to imperfect CSI, providing systematic gains over the latter across estimation error levels and fading conditions. Taken together, these results highlight BeamNet not only as a practical beamforming scheme under imperfect CSI, but also as a flexible numerical tool for exploring ISAC performance frontiers in regimes where analytical solutions are unavailable or intractable.

Future research directions include (i) extending BeamNet to joint channel estimation and beamforming, so that channel uncertainty is incorporated natively into the learning process; (ii) investigating beamforming under practical hardware impairments such as low-resolution analog-to-digital converters (ADCs) and digital-to-analog converters (DACs), nonlinear RF front-ends, and quantization constraints; and (iii) integrating synchronization-related imperfections (e.g., timing offsets and carrier-frequency offsets) into the end-to-end design. Collectively, these extensions will further tighten the achievable CR–SR Pareto frontier under realistic hardware and synchronization impairments and provide new design insights for deploying unsupervised-learning-based ISAC schemes in next-generation wireless systems.