Constrained Soft Actor–Critic for Joint Computation Offloading and Resource Allocation in UAV-Assisted Edge Computing

TL;DR

This paper introduces a new deep reinforcement learning method to optimize computation offloading and resource allocation in UAV-assisted edge computing systems.

Contribution

The novel contribution is Constrained Soft Actor–Critic (C-SAC), a deep reinforcement learning algorithm that explicitly handles latency constraints using a CMDP formulation.

Findings

C-SAC achieves an 18.9% constraint violation rate, significantly better than other methods.

Learned policies adaptively adjust computation based on channel quality with a strong correlation coefficient of −0.894.

C-SAC remains robust with minimal performance variation even when channel variability triples.

Abstract

Unmanned Aerial Vehicle (UAV)-assisted edge computing supports latency-sensitive applications by offloading computational tasks to ground-based servers. However, determining optimal resource allocation under strict latency constraints and stochastic channel conditions remains challenging. This paper addresses the joint computation partitioning and power allocation problem for UAV-assisted edge computing systems. We formulate the problem as a Constrained Markov Decision Process (CMDP) that explicitly models latency constraints, rather than relying on implicit reward shaping. To solve this CMDP, we propose Constrained Soft Actor–Critic (C-SAC), a deep reinforcement learning algorithm that combines maximum-entropy policy optimization with Lagrangian dual methods. C-SAC employs a dedicated constraint critic network to estimate long-term constraint violations and an adaptive Lagrange multiplier that automatically balances energy efficiency against latency satisfaction without manual tuning. Extensive experiments demonstrate that C-SAC achieves an 18.9% constraint violation rate. This represents a 60.6-percentage-point improvement compared to unconstrained Soft Actor–Critic, with 79.5%, and a 22.4-percentage-point improvement over deterministic TD3-Lagrangian, achieving 41.3%. The learned policies exhibit strong channel-adaptive behavior with a correlation coefficient of −0.894 between the local computation ratio and channel quality, despite the absence of explicit channel modeling in the reward function. Ablation studies confirm that both adaptive mechanisms are essential, while sensitivity analyses show that C-SAC maintains robust performance with violation rates varying by less than 2 percentage points even as channel variability triples. These results establish constrained reinforcement learning as an effective approach for reliable UAV edge computing under stringent quality-of-service requirements.

Figures

Full text

1. Introduction

The advancement of sixth-generation (6G) wireless networks is driving unique demands for intelligent, adaptive systems capable of supporting latency-sensitive applications such as autonomous vehicles, industrial automation, and extended reality. Unmanned Aerial Vehicles (UAVs) have emerged as a key enabler for such applications by offering deployment flexibility, favorable line-of-sight channels, and the capability to provide on-demand coverage where terrestrial infrastructure is lacking [1]. When UAVs are equipped with sensing and computing capabilities, they can serve as mobile edge nodes by acquiring environmental data and performing real-time processing for critical decision-making [2]. One of the major challenges associated with UAV-assisted edge computing is determining how to split computational tasks between processing them locally on the UAV’s resources and transmitting them to ground-based edge servers for processing [3]. Local processing eliminates transmission delays, but it limits the processing capability due to limited availability of onboard resources. On the other hand, offloading allows the use of superior processing capabilities at the servers but it depends on the quality of the wireless channel and consumes significant transmission energy [4]. Determining how to split the tasks is particularly difficult when there are strict latency requirements, since missing deadlines can lead to serious consequences, ranging from degraded user experience to safety-critical failures [5]. Furthermore, the stochastic nature of wireless channels causes offloading performance to vary randomly [6]. In general, optimizing this process using traditional optimization techniques relies on accurate mathematical models. These techniques typically assume that perfect channel state information (CSI) is available. This is impractical for many real-world deployments, where the environment changes randomly and over time [7]. Recently, deep reinforcement learning (DRL) has been identified as a promising alternative method for such problems. DRL enables model-free policy learning that adapts to complex and stochastic environments without requiring explicit knowledge about the system [8]. There has also been significant interest in recent years in applying DRL to optimize the computation offloading process in edge computing systems [9,10]. However, many DRL methods rely on either implicit or explicit reward shaping to solve these problems. DRL techniques also rely on penalty terms to enforce the latency constraints. This may require large amounts of hyperparameter tuning and provides little guarantee of satisfying the constraints. This is particularly problematic in applications where meeting deadlines is absolutely necessary. Constrained reinforcement learning is a framework that uses Lagrangian relaxation as a means of solving optimization problems subject to constraints. Although this approach has been extensively adopted in prior studies for wireless resource allocation [11,12], existing works have not considered UAV-assisted edge computing scenarios with continuous computation partitioning. As a result, a clear disconnect remains between recent advances in constrained learning methods and their practical deployment in UAV-enabled edge computing systems. In this study, we bridge this gap by developing a Constrained Soft Actor–Critic (C-SAC) algorithm for UAV-assisted edge computing with latency constraints. We first formulate the joint computation partitioning and power allocation problem as a Constrained Markov Decision Process (CMDP), and then employ Lagrangian dual optimization to handle the constraints in a principled manner. Our approach builds upon the maximum-entropy framework [13] and utilizes stochastic policies, which provide inherent robustness to uncertainty in the wireless channel. This property does not hold for deterministic methods, which are the most common approaches currently employed in the literature [14]. Our research contributes to the literature in the following ways:

• 1.Unlike prior works that rely on binary or partial offloading with heuristic optimization for split transmission and joint resource allocation (e.g., game-theoretic methods [15] or convex approximations [16]), we formulate the joint computation partitioning and power allocation problem for UAV-assisted edge computing as a CMDP with continuous action spaces [17]. This formulation explicitly captures the coupling between offloading decisions and wireless resource allocation under hard latency constraints in a single stochastic model, without relying on implicit penalties or deterministic simplifications. Our approach extends traditional optimization methods [18] and unconstrained RL techniques [19,20] by incorporating stochastic policies to better handle stochastic channel dynamics and strict QoS requirements in high-variance UAV-MEC environments.
• 2.We propose a constrained reinforcement learning algorithm called C-SAC, which combines maximum-entropy policy optimization with Lagrangian dual methods. C-SAC is particularly well-suited for UAV-assisted edge computing because (i) its stochastic policies provide inherent robustness to the high-variance wireless channels characteristic of air-to-ground links, where deterministic policies fail under rapid channel fluctuations [21]; (ii) the maximum-entropy framework encourages exploration across the continuous action space of partitioning ratios and power levels, critical for discovering energy-efficient solutions in the complex coupling between computation and communication decisions; (iii) it proactively avoids long-term latency issues rather than just fixing immediate ones, which is essential for applications where latency is critical for safety [22]; and (iv) the adaptive Lagrange multiplier automatically balances energy efficiency against latency satisfaction without manual hyperparameter tuning, addressing the impracticality of manual tuning in dynamic UAV deployments. This makes C-SAC superior to deterministic methods like TD3-Lagrangian, which exhibit instability under channel uncertainty, and unconstrained methods like vanilla SAC, which cannot guarantee constraint satisfaction.
• 3.Extensive experiments show that C-SAC achieves an $[eqn]$ constraint violation rate, representing a $[eqn]$ relative reduction compared to unconstrained SAC and a $[eqn]$ relative improvement over deterministic TD3-Lagrangian. The learned policies also exhibit strong channel-adaptive behavior (Pearson correlation of −0.894 between the local computation ratio and channel quality) without explicit channel modeling in the reward function.
• 4.Finally, we conducted a series of ablation studies and sensitivity analyses demonstrating that C-SAC maintains its performance across a wide variety of channel conditions, with the variation rate being less than 2 percentage points even as the channel variability increases threefold.

The remainder of this paper is organized as follows: Section 2 reviews prior work on UAV-assisted edge computing and constrained reinforcement learning. Section 3 presents the system model, including the task generation model, wireless channel model, computation model, and energy consumption model. In Section 4, we formulate the resource allocation problem as a CMDP. In Section 5, we describe the proposed C-SAC algorithm and baselines. In Section 6, we report the experimental results, ablation studies, and sensitivity analyses. Finally, in Section 7, we conclude the paper and outline several possible avenues for future work.

2. Related Work

This section provides a summary of prior related work, focusing on UAV-MEC architectures, task offloading strategies, and constrained reinforcement learning methods to ensure quality of service (QoS) for latency-sensitive tasks in real-time applications.

UAVs are becoming increasingly significant in modern wireless networks as they provide on-demand wireless access and typically a line-of-sight (LoS) connection to end devices [21,23]. The capabilities of UAVs in wireless networks were described in detail by Wu et al. [1], who emphasized their potential as mobile base stations and edge servers. Recently, there have been several studies focused on more dynamic environments; for example, Wang et al. [21] studied both trajectory planning and resource allocation for multiple UAV systems, highlighting the difficulty of coordinating interference and energy consumption. Energy-efficient frameworks for joint trajectory and power optimization were presented by Li et al. [18]. Their approach focused on minimizing the long-term energy usage of the UAVs, rather than meeting strict per-task latency deadlines.

Binary offloading is no longer sufficient to meet the demands of modern MEC. Saeedi et al. [3,24] provided a comprehensive overview of recent advances in offloading techniques, pointing out that continuous task partitioning provides a much finer level of granularity when it comes to resource allocation. Vehicular networks present additional challenges to offloading techniques. The adaptive scheduling method proposed by Ning et al. [25] was shown to increase throughput by balancing workloads between local devices and edge servers. Furthermore, the integration of sensing and communication has created new dimensions to consider when it comes to offloading. Truong et al. [26] examined the use of Reconfigurable Intelligent Surfaces (RISs) to enhance UAV-MEC performance, demonstrating how environmental control can decrease transmission delays in challenging propagation conditions.

More recently, split transmission and partial offloading strategies have been explored extensively using classical optimization techniques for joint resource allocation. Sun et al. [27] addressed multi-UAV-assisted MEC systems by formulating a multi-objective optimization problem that jointly considers task offloading, computation resource allocation, and UAV trajectory control. Their approach employs block alternate descent combined with distributed splitting, threshold rounding, Karush–Kuhn–Tucker (KKT) conditions, and successive convex approximation (SCA) to minimize task completion delay and UAV energy consumption while maximizing the number of offloaded tasks. Similarly, Qin et al. [28] investigated RIS-assisted UAV-enabled MEC systems with non-orthogonal multiple access (NOMA), proposing an iterative algorithm based on Dinkelbach’s method and block coordinate descent (BCD) to jointly optimize bit allocation, transmit power, phase shift, and UAV trajectory for energy efficiency maximization. Their work demonstrates that Lagrangian dual methods and DC programming can effectively handle the coupled optimization variables in UAV-MEC environments. Additionally, Yong et al. [29] proposed an alternating optimization algorithm (EPSO-GA) for partial subtask offloading in D2D-enabled MEC networks, achieving low latency and energy costs through joint optimization of transmit power allocation and subtask offloading strategies. Building on earlier non-ML approaches such as game-theoretic methods [15] and convex approximations [16,18], these optimization-based works provide fine-grained control over offloading decisions. However, they generally assume known and stationary system models with perfect or near-perfect channel state information (CSI), making them less suitable for the stochastic and time-varying channel conditions characteristic of UAV air-to-ground links. This limitation motivates the use of model-free reinforcement learning approaches that can learn adaptive policies directly from interaction with the environment.

DRL has proven to be a valuable tool for addressing the randomness inherent in wireless channels. Luong et al. [30] pointed out that while many previous methods used value-based approaches, actor–critic methods are generally preferred for continuous action space problems, such as power and resource allocation. Another study by Kong et al. [20] employed a TD3 algorithm to investigate task offloading in maritime edge computing, achieving excellent results. However, as previously mentioned, most DRL-based methods, including those by Wang et al. [19,31], rely on reward shaping to satisfy constraints. Reward shaping is effective in terms of average performance; however, it does not guarantee the satisfaction of strict latency requirements, as the agent may still prefer rewards over satisfying the constraints in edge cases.

To address the limitations of standard DRL, constrained reinforcement learning (CRL) has attracted considerable attention [32]. By modeling safety and latency requirements as Constrained Markov Decision Processes (CMDPs), researchers can incorporate them directly into the problem formulation. Paternain et al. [32] developed the theoretical foundation of the primal–dual methods in CMDPs, demonstrating their convergence in complex environments. In wireless applications, Liu et al. [33] applied CRL to network slicing, demonstrating that Lagrangian-based methods can maintain the reliability of latency requirements far better than unconstrained baselines. In recent years, Fasihi et al. [34] proposed state-augmented algorithms, which further improved the reliability of satisfying the constraints in highly dynamic 6G environments.

Although the literature contains many improvements in the field, there exists a significant research gap concerning the combination of continuous computation partitioning with explicit, hard latency constraints in UAV-assisted systems [23,24,34]. As shown in Table 1, many previous works—including recent optimization-based approaches such as [27,28,29]—treated constraints implicitly through penalty terms or weighted objectives, or used deterministic policies that may not be robust under extreme channel variability. The purpose of this paper is to fill this gap by proposing C-SAC, a stochastic-policy-based method using maximum-entropy optimization combined with an adaptive Lagrange multiplier. Unlike prior optimization-based methods that assume perfect CSI and stationary channels, our approach learns adaptive policies that explicitly satisfy hard latency constraints even under high-variance UAV air-to-ground channel conditions.

3. System Model

We consider a mobile edge computing system using a UAV for latency-sensitive tasks in 5G and beyond-5G wireless networks. We illustrate this system architecture in Figure 1, where a rotary-wing UAV hovers at a fixed altitude h over a roadside intersection and receives continuous data from various sensors for traffic monitoring, incident detection, or autonomous vehicle navigation support. The UAV is connected to a roadside unit (RSU) on the ground equipped with an edge computing server allowing the UAV to offload computation for each task depending on the channel quality and the computational requirements [35]. A critical issue arises when deciding whether to process each UAV task locally or remotely on the server. If processing occurs locally, then there is no transmission delay; however, it consumes more of the UAV’s computing capability than if the UAV were able to offload some of the processing to the server. However, if the UAV decides to offload some or all of its processing to the server, the performance depends on the quality of the communication link between the UAV and the server.

To make this decision in real time while meeting the application and UAV constraints, the decision-maker needs to know the communication link quality and the processing required to complete each UAV task within the maximum allowable delay $[eqn]$ , a common constraint in intelligent transportation systems. Table 2 summarizes the symbols used in this study.

3.1. Task Generation

The amount of data generated for each sensing cycle is modeled as a truncated Gaussian random variable with parameters ( $[eqn]$ , $[eqn]$ ):

$$[eqn]$$

The processing required for each sensing cycle is represented as a function of the generated data, i.e., $[eqn]$ CPU cycles, where $[eqn]$ represents the processing intensity (number of CPU cycles/number of data bits) [36].

3.2. Wireless Channel

The UAV communicates with the RSU via a radio frequency link between the ground and the UAV, which uses bandwidth B. In this problem, the UAV’s communication channel is modeled using a block fading process, where the UAV experiences a static channel during every time slot, but each time slot has its own independent channel characteristics [21]. Therefore, let $[eqn]$ represent the UAV’s signal-to-noise ratio (SNR) in decibels at time slot t, while transmitting at maximum power $[eqn]$ ; then the distribution of $[eqn]$ can be expressed as a normal distribution [26]:

$$[eqn]$$

where $[eqn]$ is the mean SNR experienced by the UAV when transmitting at the fixed altitude, and $[eqn]$ represents the variability of the SNR due to other environmental conditions. The achievable data rate $[eqn]$ of the UAV at time t is given by the expression

$$[eqn]$$

Equation (3) shows how both the channel quality $[eqn]$ and the transmit power $[eqn]$ impact the effective transmission rate through the use of adaptive resource allocation techniques [37]. Specifically, increasing the transmit power $[eqn]$ directly increases the data rate $[eqn]$ , which in turn reduces the transmission time $[eqn]$ for offloaded data (5). However, transmit power does not affect the local computation time $[eqn]$ or the edge server computation time $[eqn]$ , as these depend solely on the computation partitioning ratio and processor frequencies. Therefore, higher transmit power is particularly beneficial when the UAV offloads a large portion of the task (i.e., when $[eqn]$ is low), as it accelerates the offloading path without impacting the local processing path.

3.3. Computation Partitioning

For each task with size $[eqn]$ , the UAV chooses a split ratio $[eqn]$ , which defines how many of the $[eqn]$ bits should be computed locally. By using a continuous split ratio instead of a discrete (binary) offloading scheme, we can make fine-grained adaptations depending on the current conditions. The time required for computing $[eqn]$ bits locally on the UAV’s onboard processor operating at frequency $[eqn]$ is given by

$$[eqn]$$

The remaining $[eqn]$ bits are sent to the edge server for remote processing. The transmission time depends on the current data rate:

$$[eqn]$$

The edge server operating at frequency $[eqn]$ computes the offloaded part in

$$[eqn]$$

Therefore, $[eqn]$ corresponds to local processing, $[eqn]$ corresponds to full offloading, and all other values correspond to hybrid execution where the UAV uses both resources to compute the task.

3.4. End-to-End Latency

The total time $[eqn]$ needed to complete task t is defined as the sum of a fixed sensing duration $[eqn]$ and the processing delay. As local computation and offloading use different resources (CPU vs. radio), they can run concurrently. Therefore, the end-to-end delay is determined by the maximum of the two paths:

$$[eqn]$$

Using the previous equations, we obtain the expanded version of the total delay:

$$[eqn]$$

Thus, there is a basic trade-off here: Increasing $[eqn]$ results in lower communication costs, but increases the time required to process the task locally. More specifically, increasing $[eqn]$ has the following effects on the end-to-end latency $[eqn]$ : (i) the local computation time $[eqn]$ increases proportionally with $[eqn]$ as shown in (4); (ii) the transmission time $[eqn]$ decreases because fewer bits $[eqn]$ need to be offloaded (5); and (iii) the edge computation time $[eqn]$ also decreases for the same reason (6). Since the end-to-end latency is determined by the maximum of the local path and the offloading path (7), the optimal $[eqn]$ balances these two competing paths. When channel conditions are poor (low $[eqn]$ ), the transmission time dominates, favoring higher $[eqn]$ (more local processing). When channel conditions are good (high $[eqn]$ ), offloading becomes faster, favoring lower $[eqn]$ . The best value of $[eqn]$ strongly depends on the current channel state $[eqn]$ and the size of the task $[eqn]$ , which motivates an adaptive, learning-based method to find the best partitioning of the task. In addition, the system must meet the constraint on the maximum allowed delay:

$$[eqn]$$

with $[eqn]$  ms selected as a stringent latency constraint for our experimental simulations in this UAV-assisted MEC environment [22].

3.5. Energy Consumption

Due to limited battery life, energy efficiency is crucial for UAV-based computations. As shown below, the total controllable energy consumed by the UAV during one time slot consists of three parts:

Sensing Energy: The sensor consumes a constant power $[eqn]$ for the whole duration of the sensing operation:

$$[eqn]$$

Computation Energy: The computation energy is proportional to the third power of the clock frequency according to the cubic relationship between the switching power and the clock frequency for CMOS circuits [38]:

$$[eqn]$$

where $[eqn]$ represents the effective switched-capacitance coefficient. This parameter is a hardware-specific constant that reflects the energy consumed per CPU cycle, with the value $[eqn]$ J/cycle used as a standard representation for high-performance mobile processors in the MEC literature [39].

Communication Energy: The energy consumed by transmitting the offloaded data is

$$[eqn]$$

The total controllable energy is

$$[eqn]$$

The computation partitioning ratio $[eqn]$ affects the total UAV energy consumption $[eqn]$ through two opposing mechanisms: (i) increasing $[eqn]$ raises the local computation energy $[eqn]$ proportionally (11), as more bits are processed by the UAV’s onboard processor; and (ii) increasing $[eqn]$ reduces the communication energy $[eqn]$ (12), since fewer bits need to be transmitted to the edge server. The transmit power $[eqn]$ directly increases $[eqn]$ ; however, higher $[eqn]$ also increases the data rate $[eqn]$ (3), which reduces the transmission time $[eqn]$ , partially offsetting the energy increase. It is important to note that all energy quantities in $[eqn]$ refer to the UAV-side controllable energy expenditure; the edge server’s computation energy is not included in the optimization objective, as it is powered by the grid infrastructure and does not affect UAV mission endurance.

To focus on the impact of computation partitioning and transmit power allocation, we model sensing duration $[eqn]$ and sensing power $[eqn]$ as fixed constants (common in UAV-MEC offloading studies where sensing is periodic and independent of offloading decisions) [18]. Similarly, under the fixed-altitude hovering assumption, propulsion energy is treated as constant and thus omitted from the optimization objective, as widely adopted in hovering UAV-MEC scenarios to isolate controllable communication and computation energy [40]. While propulsion energy is often the dominant energy component [41], for a rotary-wing UAV in a fixed-altitude hovering state, the propulsion power is a constant offset ( $[eqn]$ ) and does not affect the optimal resource allocation policy.

3.6. Optimization Problem

The main goal of this work is to minimize the total energy consumption while fulfilling the delay constraint. At every time slot, the UAV has to choose two decision variables: the computation partitioning ratio $[eqn]$ and the transmit power $[eqn]$ . Both decisions are based on the current channel state $[eqn]$ , the size of the task $[eqn]$ , and past experience of the UAV, leading to a sequential decision-making problem under uncertainty. The interplay between $[eqn]$ and $[eqn]$ is complex. If $[eqn]$ is increased, less data needs to be communicated. However, the processing time on the UAV is increased. If $[eqn]$ is increased, the data rate is improved and therefore the energy consumption is increased. To achieve the optimal trade-off of these factors, the UAV has to adapt continuously to the stochastic nature of both the channel state and the task sizes in real time. This is a problem suited for reinforcement learning as outlined in the subsequent sections.

4. Problem Formulation

We have developed a CMDP formulation to model the sequential nature of the resource allocation problem, while accounting for the hard latency constraint imposed by edge computing applications. Our CMDP incorporates an auxiliary cost function to represent the constraint violation of the latency requirement. We also established that the problem’s inherent characteristics—continuous state and action spaces, unknown and potentially nonstationary transition dynamics, coupled decision variables, and real-time operation requirements—necessitate a model-free learning approach. Table 2 gives the specific description of each term used in problem formulation.

The CMDP formulation is formally defined using the tuple $[eqn]$ , where $[eqn]$ denotes the state space, $[eqn]$ represents the action space, $[eqn]$ is the transition probability distribution, r is the reward function, $[eqn]$ is the constraint cost function, and $[eqn]$ is the discount factor. A critical difference from unconstrained MDPs is that feasible policies must not only optimize the rewards but also ensure that the expected cumulative cost satisfies the specified constraints. In the context of wireless resource allocation, such constraints reflect the quality-of-service (QoS) requirements of edge computing applications that impose hard boundaries on acceptable latencies and thus cannot be traded off with other objectives [42]. Our CMDP consists of the following elements.

4.1. State Space

The five-dimensional state vector given below captures all information relevant to making the current decision. The state vector includes four continuous-valued elements and one categorical element.

$$[eqn]$$

$[eqn]$ is the current task size and normalized by min-max scaling. $[eqn]$ is the current channel SNR in the normalized form. $[eqn]$ is the previous computation partitioning ratio. $[eqn]$ is the feedback about previous performance obtained by checking the previous latency value. $[eqn]$ is the magnitude of the prior constraint violation. The previous latency and cost information is quite useful in the agent knowing about the constraint satisfaction. It actually makes the agent aware of the near- or beyond-feasibility boundary.

4.2. Action Space

The action vector controls both local processing and remote processing. The first element of the action vector computation partitioning ratio $[eqn]$ determines the portion of the task executed locally, as we have discussed above in Section 3.3. Normalized transmit power $[eqn]$ is the second action in our action space and it controls UAV transmission power. The second element determines the portion of the task executed remotely.

$$[eqn]$$ $$[eqn]$$

Both of these continuous actions empower fine-grained control over the energy–latency trade-off. It differentiates our action spaces from discrete action spaces that can miss optimal operating points, especially where the channel condition is variable.

4.3. Reward Function

The objective is to decrease and stabilize the amount of energy used to increase the length of time that the UAV will stay aloft and fulfill the requirements of the latency. The immediate reward is defined as the negative of the normalized energy use at any particular moment:

$$[eqn]$$

Here, $[eqn]$ is a reference energy (a constant number representing the average energy per action taken using a random policy), which helps stabilize the numbers during the training process. Using the components of the energy model as outlined in Section 3.5, the reward is defined as

$$[eqn]$$

By maximizing cumulative rewards over time, the agent develops an ability to minimize the total amount of energy spent for the entire duration of the mission.

Constraint Cost Function:

To meet this latency constraint, we create a cost function to measure when deadlines are missed:

$$[eqn]$$

We can see that there are at least three advantages of this approach: (1) the cost is either zero or greater than zero as it cannot be negative; (2) costs for all actions that are considered possible are zero; (3) the larger the violation in time from meeting a deadline, the larger the cost of such a violation will be. This provides a continuous increase in the direction of feasibility. This is achieved by expanding the cost function to incorporate the values from the latency model [43].

$$[eqn]$$

4.4. Optimization Objective

The optimization problem aims to find a policy $[eqn]$ that maximizes the cumulative reward. It is subject to the constraint on the expected cumulative cost.

$$[eqn]$$

where $[eqn]$ represents the expected cumulative reward, $[eqn]$ represents the expected cumulative cost, and d represents the constraint threshold.

$$[eqn]$$ $$[eqn]$$

For $[eqn]$ , where d is a constraint threshold, when d ≈ 0, there will be nearly no tolerance for violating any of the latency constraints because of how critical some types of applications are with regard to their latency. Also, $[eqn]$ is used as a discount factor to promote long-term thinking but to ensure that these infinite summations are convergent. The constraint on the cumulative cost in (21) bounds the expected cumulative cost over all tasks, not just individual violations. It is possible for the number of violations of each task to be greater than zero for many tasks (this may simply be a matter of physical feasibility given the channel conditions); however, the long-run average violation rate for each task will be minimized.

4.5. Problem Characteristics

This CMDP model exhibits several characteristics that make it difficult to solve using classical methods. The primary issues are: (i) continuous state and action space (hence, table-based methods cannot be used); (ii) uncertainty and variability of the environment (both due to the randomness of the wireless channel and because of the arrival of new tasks); (iii) coupled decisions (decisions regarding splitting of tasks and transmission power affect each other, and are critical in achieving the delay bounds); and (iv) time constraints (decisions must be made rapidly, on the order of milliseconds, thus making slow computational processes unfeasible). Therefore, we require a method that can learn from direct experience and does not require prior knowledge of the environment’s behavior. Thus, we apply a model-free deep reinforcement learning technique. However, since we aim to minimize both delay and energy consumption, we will develop novel model-free policy learning based on Lagrangian dual optimization. The algorithm iteratively performs two main types of steps: initially, the agent learns a policy through trial and error in the environment; secondly, the agent modifies an additional parameter (the Lagrange multiplier $[eqn]$ ) such that the delay constraint is satisfied. Through repeated iterations of the two-step process, the agent develops a suitable policy that minimizes energy usage and satisfies the delay constraints required by edge computing applications.

5. Research Methodology

This section presents our solution approach for the CMDP formulated in Section 4. We first introduce the Lagrangian relaxation technique that transforms the constrained problem into a tractable form, and then develop the C-SAC algorithm that combines maximum-entropy reinforcement learning with adaptive constraint handling [44]. Two baseline methods have also been described for comparative evaluation.

The proposed approach addresses the key challenges identified in Section 4.5 through a principled combination of stochastic policy optimization and dual constraint handling. By leveraging the maximum-entropy framework, our method maintains exploration throughout training while the Lagrangian formulation ensures systematic constraint satisfaction. This combination is particularly effective for the UAV-MEC environment, where channel stochasticity demands robust policies and latency constraints require explicit enforcement mechanisms.

Table 3 summarizes the hyperparameters used across all algorithms.

5.1. Lagrangian Relaxation

Optimizing the objective of the Constrained MDP in (21) is difficult since traditional RL is based on unconstrained reward maximization. The best way to formulate the constrained problem is by using Lagrangian relaxation. This method uses a dual variable that represents a penalty that is used to discourage constraint violations. The Lagrangian function is the combination of the reward objective and a weighted penalty on the constraint cost, and is expressed as follows:

$$[eqn]$$

where $[eqn]$ denotes the Lagrange multiplier (dual variable). Expanding the above objective yields

$$[eqn]$$

The original constrained problem is equivalent to finding the saddle point:

$$[eqn]$$

The min-max formulation has an intuitive interpretation: the policy $[eqn]$ wants to maximize the rewards it receives while minimizing the constraint costs; meanwhile, $[eqn]$ is acting like an adaptive penalty that gets larger if the constraints are broken and smaller if they are met. We can interpret the Lagrange multiplier $[eqn]$ as being the price of violating a constraint. If $[eqn]$ is large, the policy is going to focus on reducing latency; if $[eqn]$ is small, then the policy will be more concerned with energy efficiency. Rather than having to manually tune this trade-off, our algorithm will learn $[eqn]$ automatically through the process of optimizing the dual objective through gradients.

5.2. Maximum-Entropy Framework

The primary goal of standard reinforcement learning is to identify policies which achieve the highest expected accumulated reward. Maximum-entropy reinforcement learning (RL) introduces an additional “entropy bonus” to the primary goal of maximizing the expected accumulated reward to encourage exploration and increase robustness.

$$[eqn]$$

where $[eqn]$ is the policy entropy and $[eqn]$ is the temperature parameter that controls the trade-off between exploration and exploitation. Two key benefits exist from this approach to our particular problem. First, the probabilistic nature of the policy provides a means of handling the uncertainty present in wireless communications—rather than selecting one specific action, the policy selects a distribution that can provide protection against potential variations in the communication channel. Second, by introducing an entropy penalty, we are able to prevent the agent’s policy from prematurely converging to a less optimal deterministic strategy, thereby allowing the agent to discover more optimal, and therefore more robust, solutions.

5.3. Constrained Soft Actor–Critic (C-SAC)

The Constrained Soft Actor–Critic algorithm is proposed in this paper, and it extends the maximum-entropy framework so that it is able to treat explicit constraints as part of its decision-making process using a Lagrangian dual optimization technique. The Constrained SAC incorporates three main features: (i) entropy-regularized policy optimization for robust exploration, (ii) a constraint critic to anticipate long-term constraint violations, and (iii) an adaptive Lagrange multiplier to allow for automatic tuning of constraints.

5.3.1. Network Architecture

C-SAC utilizes trained neural networks—an actor, two reward critics, and a constraint critic. All networks are implemented as a multi-layer perceptron with [256, 256] hidden layers and ReLU activations, with each critic additionally having a target network updated via soft updates for training stability. Actor network $[eqn]$ outputs a Gaussian distribution over actions. Given state $[eqn]$ , it produces mean $[eqn]$ and log-standard-deviation $[eqn]$ , from which actions are sampled using the reparameterization trick. Twin reward critics $[eqn]$ , $[eqn]$ estimate the expected cumulative reward. Using two critics and taking their minimum mitigates overestimation bias common in actor–critic methods [45]. Constraint critic $[eqn]$ estimates the expected cumulative constraint cost:

$$[eqn]$$

This dedicated network enables the agent to anticipate long-term constraint violations rather than reacting only to immediate costs. Figure 2 shows the working principle of the C-SAC algorithm. Each critic has a corresponding target network (updated via soft updates) that provides stable learning targets.

5.3.2. Constrained Policy Objective

Combining the maximum-entropy objective with Lagrangian constraint handling, C-SAC optimizes:

$$[eqn]$$

This integrates three components: reward maximization for energy efficiency, entropy regularization for robust exploration, and constraint cost penalization for latency compliance. The balance among these objectives is determined by $[eqn]$ and $[eqn]$ , both learned automatically during training.

5.3.3. Learning Updates

At each training step, the algorithm samples a mini-batch from the replay buffer and performs the updates outlined below. These updates are performed sequentially, with each component contributing to the overall learning objective. The mini-batch sampling from the replay buffer enables off-policy learning, allowing the algorithm to reuse past experiences for improved sample efficiency.

Reward Critic Update. The critics are trained by minimizing the Bellman residual:

$$[eqn]$$

The use of twin critics, where the minimum Q-value is taken, helps mitigate the overestimation bias that commonly affects actor–critic algorithms, where the target value incorporates the entropy bonus:

$$[eqn]$$

Constraint Critic Update. The constraint critic minimizes a similar loss:

$$[eqn]$$

with target

$$[eqn]$$

It is also important to note that the constraint target does not include the entropy bonus since the entropy regularizer can be applied only to the reward function. This separation ensures that the constraint critic provides an unbiased estimate of future constraint violations, which is essential for accurate Lagrangian penalty computation.

Actor Update. The policy is updated to maximize the Lagrangian objective:

$$[eqn]$$

The loss is meant to encourage actions that achieve high rewards, low constraint costs, and enough entropy to allow for exploration.

Temperature Update. The entropy coefficient $[eqn]$ is adjusted to maintain a target entropy level $[eqn]$ :

$$[eqn]$$

Lagrange Multiplier Update. The dual variable $[eqn]$ is updated via gradient ascent:

$$[eqn]$$

$[eqn]$ will increase when the estimated constraint cost exceeds a given threshold of d. This in turn will amplify the penalty for taking an action that violates the constraint. Conversely, when $[eqn]$ is decreased because the estimated constraint cost is below the threshold of d, this means the cost of the last action taken was within the constraint. In either case, the clipping helps ensure that $[eqn]$ is always non-negative and has a finite value so as to promote numerical stability.

Target Network Update. All target networks are updated via an exponential moving average with coefficient $[eqn]$ :

$$[eqn]$$

This soft update mechanism provides stable learning targets, preventing the oscillations that can occur when target networks are updated abruptly. The small value of $[eqn]$ ensures gradual target network changes while still tracking the learning progress of the main networks.

The reinforcement learning hyperparameters follow established practices for stable, sample-efficient learning. We use a near-unity discount factor ( $[eqn]$ ) for long-horizon planning and conservative learning rates ( $[eqn]$ ) for stable updates, as per SAC conventions [44]. The Lagrange multiplier learning rate ( $[eqn]$ ) is set higher for responsive constraint handling, a choice validated by our ablation study (Section 6.6) [46]. All neural networks (actor, reward critics, and constraint critic) employ a multi-layer perceptron (MLP) architecture consisting of two hidden layers, each containing 256 neurons (denoted as $[eqn]$ in Table 3). The input layer receives the 5-dimensional state vector, while the output layer produces either action distribution parameters (for the actor, outputting mean and log-standard deviation for each of the 2 action dimensions) or scalar value estimates (for the critics). ReLU activation functions are applied after each hidden layer to introduce non-linearity. This architecture provides sufficient capacity without excessive computation [45], and the target entropy is automatically set to $[eqn]$ , a standard heuristic for this action space [44].

The complete algorithm is presented in Algorithm 1. Algorithm 1: Constrained Soft Actor–Critic (C-SAC)

5.4. Baseline Algorithms

To evaluate the effectiveness of our approach, we compare it against two baselines representing alternative design choices.

Soft Actor–Critic (SAC). The unconstrained variant of our algorithm, which optimizes only the maximum-entropy objective without explicit constraint handling [44]. SAC uses identical network architecture and hyperparameters but lacks the constraint critic $[eqn]$ and Lagrange multiplier $[eqn]$ . Any constraint awareness must emerge implicitly through reward shaping, which we do not employ. This baseline tests whether explicit constraint handling is necessary.

TD3-Lagrangian. A constrained variant of Twin Delayed DDPG that uses deterministic policies with Lagrangian constraint handling [45]. Unlike the stochastic policies in SAC and C-SAC, TD3 outputs deterministic actions with added exploration noise during training. This baseline tests whether stochastic policies offer advantages in our high-variance channel environment. The implementation follows the original TD3 with the addition of a constraint critic and Lagrange multiplier update identical to C-SAC.

Their main differences are summarized as follows: C-SAC and SAC both employ stochastic Gaussian policies with entropy regularization, while TD3-Lagrangian uses deterministic policies with additive noise for exploration. On the other hand, C-SAC and TD3-Lagrangian both incorporate Lagrangian constraint handling, whereas SAC relies only on reward maximization. These design choices enable us to distinguish the contributions of (i) explicit constraint handling and (ii) stochastic versus deterministic policy architectures.

6. Simulation Results

In this section, we will evaluate our proposed C-SAC method against the following baselines: (1) the unconstrained SAC (Soft Actor–Critic); (2) deterministic TD3-Lagrangian. Firstly, we explain the experimental setting and provide an analysis of training dynamics, policy behavior, and overall performance. In addition, ablation studies and sensitivity studies are provided to test our design decisions under different settings.

6.1. Experimental Setup

All experiments were conducted for 500 training epochs with each epoch having 200 time steps. The simulation parameters in Table 3 reflect the UAV-assisted edge computing environment as characterized in the literature. The simulation environment was implemented in Python (3.11.9) using PyTorch (2.9.1) for neural network training. To reflect real intelligent transportation workloads, the mean task size ( $[eqn]$ Mbits) and computational intensity ( $[eqn]$ cycles/bit) are set based on typical edge tasks like object detection [36]. The 10:1 ratio between edge server ( $[eqn]$ GHz) and UAV processor ( $[eqn]$ GHz) frequencies justifies offloading and aligns with commercial hardware capabilities [18]. The stringent latency deadline ( $[eqn]$ ms) is derived from URLLC and autonomous vehicle requirements [22].

The channel is configured with standard 5G bandwidth ( $[eqn]$ MHz) [37]. A mean SNR of $[eqn]$ dB with $[eqn]$ dB deviation represents realistic, variable air-to-ground links [21,26]. The transmit power range ( $[eqn]$ W, $[eqn]$ W) models practical UAV power constraints [18].

6.2. Training Convergence and Constraint Satisfaction

The training curves are shown in Figure 3, which demonstrates fundamental differences between these methods in their trade-offs between energy usage and constraint violation rates.

6.2.1. Reward and Violation Trade-Off

SAC has the highest return (approximately −20), which it obtains by using no constraint knowledge and very aggressive reduction in energy use; however, SAC’s large energy savings are offset by violations of 79%. This is why SAC is considered unsuitable for latency-critical applications. C-SAC achieves a return of roughly $[eqn]$ , representing significantly more energy usage than SAC and constraint violations of 18.9%. TD3-Lagrangian’s performance is somewhere in between C-SAC and SAC, with constraint violations of about 41%; however, there are some oscillations in TD3-Lagrangian’s training path. There is clear stability in the return and latency violation ratio of C-SAC. The most important finding is that the extra energy that C-SAC uses (about 57% more than SAC) represents the cost of having satisfied the constraint reliably. If the application has consequences as severe as those resulting from violations of deadlines, then this trade-off is considered favorable [38]. C-SAC has a significant advantage over the baselines:

$$[eqn]$$

6.2.2. Lagrange Multiplier Dynamics

As we can see in Figure 3c, both constrained methods increase $[eqn]$ rapidly in the initial training phases and they reach a maximum $[eqn]$ value of $[eqn]$ by epoch 100. The maximum $[eqn]$ value shows the high activity of the constraint; the algorithms will always use the maximum penalty pressure. There is a big difference in how the algorithm responds to this penalty. C-SAC transforms this penalty into constant constraint satisfaction. In contrast, TD3-Lagrangian’s deterministic policy does not adequately respond to this penalty and therefore continues to have constraint violations, even though the same penalty magnitude is applied. This fundamental difference in stability stems from the mismatch between TD3-Lagrangian’s deterministic policy and the highly stochastic UAV-MEC environment. A deterministic policy outputs a single fixed action for each state, making it inherently brittle when the wireless channel exhibits significant randomness. During training, TD3-Lagrangian occasionally encounters channel conditions where its deterministic action leads to severe constraint violations. However, because the policy provides no distributional coverage over alternative actions, it cannot hedge against uncertainty. When the policy updates in response to these violations, it can overcorrect—shifting dramatically to avoid the problematic state–action pairs. This creates oscillations where the policy alternates between overly conservative (high $[eqn]$ , minimal offloading) and overly aggressive (low $[eqn]$ , maximum offloading) strategies. C-SAC’s stochastic policy mitigates this instability through two mechanisms: (1) the policy outputs a distribution over actions, naturally hedging against channel uncertainty by maintaining diverse action candidates, and (2) the entropy regularization explicitly discourages premature commitment to deterministic strategies, preserving exploration throughout training. This distributional coverage allows C-SAC to smoothly adapt to constraint violations without the abrupt policy shifts that characterize deterministic methods.

The constraint critics shown in Figure 4 give additional insight into learning constraints from data.

The C-SAC constraint critic $[eqn]$ is constrained to a limit below 0.05 by epoch 100 from its starting point of 1.6 and then remains relatively steady. The smoothness of this convergence indicates that it has learned the constraint cost function well. In stark contrast, the constraint critic for TD3-Lagrangian shows large spikes up to 4.0 at epochs 250 and 400, which aligns with the periodic violations shown in Figure 3b. These fluctuations are due to difficulties in estimating constraint costs based on the deterministic nature of the policy over the stochastic state space—something that stochastic policies can inherently overcome through their distributional coverage.

6.3. Channel-Adaptive Policy Behavior

A central finding of this work is that the learned policies exhibit strong channel-adaptive behavior without explicit channel modeling in the reward function. Figure 5 shows the Pearson correlation coefficient between the local computation ratio $[eqn]$ and instantaneous SNR over training.

The optimal strategy is a negative correlation: when SNR is high, the agent should reduce $[eqn]$ to offload more computation, exploiting the favorable channel for efficient transmission. Conversely, poor channel conditions favor local processing to avoid transmission delays. All three algorithms discover this strategy, but with varying degrees of consistency:

• C-SAC ( $[eqn]$ ): Achieves strong, stable correlation by epoch 80 with minimal variance thereafter.
• TD3-Lagrangian ( $[eqn]$ ): Achieves the strongest final correlation but exhibits instability, with spikes toward $[eqn]$ at epochs 220 and 400.
• SAC ( $[eqn]$ ): Shows weaker adaptation, as the unconstrained objective provides less incentive for latency-aware decisions.

A deceptive inconsistency emerges: TD3-Lagrangian attains stronger $[eqn]$ -SNR correlation yet has low constraint satisfaction. This arises because correlation measures average behavior, while constraint satisfaction depends on worst-case performance. TD3-Lagrangian’s deterministic policy sometimes overreacts to channel variations, causing the spikes visible in Figure 5. During these instability periods, the policy temporarily loses its adaptive behavior, leading to violation bursts. The stochastic policy of C-SAC provides more consistent adaptation but, slightly weaker in correlation scale, it results in better overall constraint satisfaction [47]. Figure 6 further characterizes the action distributions.

C-SAC has $[eqn]$ , which means that it is in favor of local computing as much as possible to have some buffer for channel uncertainty. The SAC has a very aggressive offloading strategy ( $[eqn]$ ) and accepts high-latency risk so that it can minimize its local computing power consumption. TD3-Lagrangian is somewhere in the middle ( $[eqn]$ ), but there are significant spikes up to 0.85 during unstable time periods. The standard deviation plot shows another important aspect of how the algorithms behave differently. C-SAC and SAC have a consistent level of variability ranging from 0.35 to 0.40; this suggests that they have a good and consistent balance between exploration and exploitation. The standard deviation of TD3-Lagrangian drops to 0.10 during times of instability (epochs 220–280, 380–420); this indicates that the algorithm temporarily collapsed into nearly deterministic behavior. This collapse into nearly deterministic behavior would explain why the violation spikes occur: when the action space is limited by reduced action variety, the algorithm cannot effectively respond to changes in the channels.

6.4. Comparative Performance Analysis

Figure 7 represents a summary of the final performance results from all of the algorithms. Table 4 provides the numerical values.

Here are some points for discussion:

Constraint Satisfaction. C-SAC experiences 18.9% violations, which means that it satisfies over 81% of its tasks within the 35 ms deadline. Compared to SAC and TD3-Lagrangian, C-SAC has achieved a 4.2× better performance as compared to SAC in terms of constraint satisfaction and a 2.2× better performance compared to TD3-Lagrangian. The comparison highlights the significant advantage of using explicit constraint handling via Lagrangian methods over implicit constraint handling methods.

Latency Performance. C-SAC produces a mean latency of 29.97 ms, which is 14% under the 35 ms deadline. In contrast, TD3-Lagrangian produces a mean latency of 33.69 ms with a 4% margin to the deadline. Finally, SAC has an extremely high mean latency of 329 ms (approximately 10× the deadline), demonstrating how completely SAC ignores time constraints.

Trade-Off Between Energy and Constraints. C-SAC uses about 57% more energy than SAC. As such, whether or not the additional energy use is acceptable will depend on the specific needs of each application scenario. For instance, if there are mission-critical applications that cannot afford to violate deadlines, then the increased energy use by C-SAC can be justified. The increase in energy usage is primarily due to C-SAC being more conservative. It uses higher average $[eqn]$ (which means it uses more local processing), and it transmits more power when it needs to offload reliably.

6.5. Energy–Constraint Trade-Off Analysis

Figure 8 visualizes the training trajectories in the energy–violation space, revealing how each algorithm navigates the fundamental trade-off.

SAC has been unable to escape from the high-violation, low-energy corner for most of the training process, due to the fact that SAC’s objective does not provide an incentive to decrease violation levels. Both of the constrained methods moved to lower violation levels, but they have very different properties. C-SAC moved along a generally straight-line path to a small cluster centered at (31, 19%), while the path taken by TD3-Lagrangian was more irregular, and explored a much larger area of space, eventually coming to rest near (27, 41%) with large variability in both dimensions. The better trade-off offered by C-SAC’s final position compared to TD3-Lagrangian’s final position is particularly noteworthy: for about 15% more energy, C-SAC achieved over 50% more violations reduced. This may indicate that the stochastic policy used by C-SAC has found a more efficient operating point on the constraint boundary defined by the energy constraint. In addition, the more tightly clustered final positions of C-SAC demonstrate more stable behavior upon convergence.

6.6. Ablation Study

We performed the ablation study using the same testbed as for the evaluation of C-SAC’s performance in order to assess how individual components contribute to the overall performance. We disabled or modified important parts of the mechanism, as illustrated in Table 5.

6.6.1. Adaptive Lagrange Multiplier

A reduction in the learning rate $[eqn]$ of 10×, which affects how quickly the constraints can be updated, increases the number of constraint violations from 18.2% to 44.4%. It also allows the policy to reach a constraint-violating equilibrium because the slow update speed does not allow $[eqn]$ to be sufficiently large in the early stages of training. An increase in $[eqn]$ of 10× produces little change, showing that the chosen value of the learning rate was sufficient. Thus, the adaptive nature of the mechanism has an asymmetric effect on the learning process, and should therefore be tuned to have a faster adaptation of the penalty. This phenomenon occurs because the Lagrange multiplier $[eqn]$ acts as a dynamic penalty weight on constraint violations. When $[eqn]$ is too small, $[eqn]$ increases too slowly during early training when the policy is still learning. Consequently, the constraint cost penalty remains weak relative to the energy reward, allowing the policy to settle into a local optimum that prioritizes energy minimization over latency satisfaction. Once stabilized at this equilibrium, the small learning rate prevents $[eqn]$ from increasing rapidly enough to escape the constraint-violating region. In practical terms, this is analogous to a control system with insufficient gain—the feedback signal (constraint violations) is detected but the corrective action (increasing $[eqn]$ ) is too weak to drive the system toward the desired operating point (constraint satisfaction). The adaptive mechanism with appropriately tuned $[eqn]$ avoids this by enabling rapid penalty escalation during initial training, preventing the policy from converging prematurely to infeasible solutions. Even a manually selected value of $[eqn]$ , which could produce good results, produced only 20.4% constraint violation compared to 18.2% when using the adaptive mechanism. Therefore, the adaptive mechanism provides a significant practical advantage over using fixed values of $[eqn]$ [46], as there is no need for manual searches for suitable values of $[eqn]$ .

6.6.2. Adaptive Entropy Temperature

In removing the ability of the entropy temperature to automatically adjust, its parameters degrade both the constraint violation and channel adaptation. Using a fixed value of $[eqn]$ (too small), the policy converged too quickly and achieved 24.1% constraint violations with a significantly reduced $[eqn]$ -SNR correlation ( $[eqn]$ vs. $[eqn]$ ). Using a fixed value of $[eqn]$ (too big) resulted in too much randomness in the policy and prevented reliable convergence, achieving 39.5% constraint violations with a severely degraded $[eqn]$ -SNR correlation ( $[eqn]$ ). As demonstrated in Figure 3, the adaptive mechanism of the entropy temperature converged to $[eqn]$ . This represents the optimal point at which to balance exploration and exploitation. This automatic tuning eliminates a sensitive hyperparameter that would otherwise need problem-specific modification.

6.7. Sensitivity Analysis

In this section, we examine how well our algorithms can perform in different environments. A three-dimensional analysis is performed: deadline tolerance, the sensitivity of the algorithms to changes in channel characteristics, and sensitivity to the value of Lagrange multipliers.

6.7.1. Deadline Sensitivity

The results of deadline sensitivity change are shown in Table 6. The table presents constraint violation percentages as deadlines vary from a tight deadline of 25 ms to a relaxed deadline of 55 ms.

C-SAC demonstrates gradual degradation of performance as tighter constraints are imposed on the deadline. Although C-SAC experiences a high number of constraint violations $[eqn]$ when the deadline is set to 25 ms, this is still better than the baselines tested. As deadlines are relaxed, the number of constraint violations experienced by C-SAC decreases monotonically to $[eqn]$ at 55 ms. This predictable relationship between the deadline and performance makes C-SAC suitable for applications with varying latency requirements. Regardless of the deadline, SAC continues to experience constraint violations at rates greater than $[eqn]$ , confirming that an unconstrained version of SAC is fundamentally unsuited for use in latency-sensitive applications. TD3-Lagrangian appears to scale reasonably well; however, at 55 ms, TD3-Lagrangian displays an anomaly where it achieves a low constraint violation rate of $[eqn]$ , but at the same time, its $[eqn]$ -SNR correlation collapses to −0.400. This indicates that the policy learned by TD3-Lagrangian has adopted a near-constant strategy rather than one which adapts to the changing environment. Thus, we have demonstrated another common failure mode of deterministic strategies under easy-to-satisfy constraints [48].

6.7.2. Channel Variability Sensitivity

The results of channel sensitivity are presented in Table 7. In these tests, we examined how our algorithms respond to different levels of channel fading by varying $[eqn]$ over the values of 4 dB (a mild level of fading), 8 dB (an average level of fading), and 12 dB (a severe level of fading).

C-SAC demonstrates remarkable robustness: as channel variability triples, the violation rate increases by only 1.9 percentage points (17.1% to 19.0%). This stability arises from the stochastic policy’s inherent hedging against uncertainty—by maintaining action diversity, C-SAC performs well across the distribution of possible channel realizations [47].

TD3-Lagrangian exhibits non-monotonic sensitivity with a 20.1-percentage-point range. Interestingly, it performs best under moderate fading (27.8% at $[eqn]$ dB) and worst under severe fading (47.9% at 12 dB). Moderate variability may provide beneficial curriculum learning during training, while severe variability overwhelms the deterministic policy’s limited exploration capacity.

6.7.3. Lagrange Multiplier Bound

The study in Table 8 demonstrates how changes in $[eqn]$ affect the balance between energy and violation constraints.

An increase in $[eqn]$ allows for greater constraint enforcement but comes at the price of higher energy consumption. The curve shows that one increasing their lambda values (doubling them) from 50 to 100 will result in a 1.8 percent reduction in violations while increasing their energy usage by 7.3 percent. We conclude based on our study that an optimal value for lambda is $[eqn]$ .

6.8. Summary of Findings

The experimental evaluation supports the following conclusions:

• 1.Explicit constraint handling is essential. C-SAC reduces violations by 76% compared to unconstrained SAC, demonstrating that implicit reward shaping cannot achieve reliable constraint satisfaction.
• 2.Stochastic policies outperform deterministic alternatives. C-SAC achieves 54% lower violations than TD3-Lagrangian despite using identical Lagrangian constraint handling, with the gap widening under severe channel conditions.
• 3.Adaptive mechanisms are critical. Both adaptive $[eqn]$ and adaptive $[eqn]$ contribute significantly; removing either degrades performance by 30–140%.
• 4.Channel-adaptive behavior emerges naturally. All algorithms discover the optimal negative $[eqn]$ -SNR correlation without explicit channel modeling, but C-SAC maintains this behavior most consistently.
• 5.C-SAC exhibits robust generalization. Violation rates vary by less than 2 percentage points as channel variability triples, enabling reliable deployment across diverse operating conditions.

7. Conclusions

This paper addressed the joint computation partitioning and power allocation problem for UAV-assisted edge computing systems operating under stringent latency constraints. We formulated the resource allocation problem as a Constrained Markov Decision Process and proposed the Constrained Soft Actor–Critic (C-SAC) algorithm, which combines maximum-entropy reinforcement learning with Lagrangian dual optimization for principled constraint handling. The algorithm employs a dedicated constraint critic network to anticipate long-term violations and an adaptive Lagrange multiplier that automatically balances energy efficiency against constraint satisfaction without manual hyperparameter tuning.

Comprehensive experiments demonstrated that C-SAC achieves 76% lower constraint violations compared to unconstrained SAC and 54% improvement over deterministic TD3-Lagrangian, while maintaining a mean latency of 29.97 ms against the 35 ms deadline. The learned policies exhibit strong channel-adaptive behavior with a correlation coefficient of $[eqn]$ between the local computation ratio and channel SNR, despite no explicit channel modeling in the reward function. This emergent behavior demonstrates that deep reinforcement learning can discover sophisticated resource allocation strategies directly from environment interaction. Notably, while TD3-Lagrangian achieved stronger instantaneous channel adaptation, its deterministic nature led to instability under channel uncertainty, confirming that stochastic policies provide inherent robustness valuable for wireless systems with rapidly fluctuating conditions. Ablation studies verified that both the adaptive Lagrange multiplier and automatic entropy tuning are essential, with their removal degrading performance by 30–140%. Sensitivity analyses further showed that C-SAC maintains robust performance across varying channel conditions, with violation rates changing by less than 2 percentage points as channel variability triples.

The current work has several limitations that suggest directions for future research. The formulation assumes a single UAV with fixed hovering position; extending it to multi-UAV scenarios with trajectory optimization would enable coordinated resource allocation and improved coverage. The channel model, while capturing essential fading characteristics, does not account for interference from competing transmissions. Future work could incorporate multi-agent reinforcement learning for cooperative UAV networks, joint trajectory and resource optimization, and transfer learning approaches to reduce training time when deploying to new environments. Hardware testbed validation would provide insights into real-world deployment challenges and computational requirements for onboard policy execution. As sixth-generation wireless networks evolve toward tighter integration of sensing, communication, and computation, the C-SAC framework offers a principled approach to intelligent resource allocation under strict quality-of-service constraints.