Solving robotics tasks with prior demonstration via exploration-efficient deep reinforcement learning

TL;DR

This paper introduces a new deep reinforcement learning framework that efficiently learns robotics tasks using demonstrations and reduces inefficient exploration.

Contribution

The novel action selection module in DRLR reduces bootstrapping errors and prevents sub-optimal policy convergence using SAC instead of TD3.

Findings

The DRLR framework successfully learns complex robotics tasks like bucket loading and open drawer.

The framework demonstrates robustness across tasks with varying state-action dimensions and demonstration qualities.

The DRLR framework was successfully deployed on a real-world wheel loader for bucket loading.

Abstract

This paper proposes an exploration-efficient deep reinforcement learning with reference (DRLR) policy framework for learning robotics tasks incorporating demonstrations. The DRLR framework is developed based on an imitation bootstrapped reinforcement learning (IBRL) algorithm. Here, we propose to improve IBRL by modifying the action selection module. The proposed action selection module provides a calibrated Q-value, which mitigates the bootstrapping error that otherwise leads to inefficient exploration. Furthermore, to prevent the reinforcement learning (RL) policy from converging to a sub-optimal policy, soft actor–critic (SAC) is used as the RL policy instead of twin delayed DDPG (TD3). The effectiveness of our method in mitigating the bootstrapping error and preventing overfitting is empirically validated by learning two robotics tasks: bucket loading and open drawer, which require extensive interactions with the environment. Simulation results also demonstrate the robustness of the DRLR framework across tasks with both low and high state–action dimensions and varying demonstration qualities. To evaluate the developed framework on a real-world industrial robotics task, the bucket loading task is deployed on a real wheel loader. The sim-to-real results validate the successful deployment of the DRLR framework.

Figures

Overview of the proposed exploration-efficient DRLR framework. The proposed framework extends a sample-efficient DRL-Ref method with a simple action selection module to mitigate inefficient explorations caused by (1) bootstrapping errors leading to the RL policy selecting out-of-distribution actions; (2) Ref policy failing to provide good actions under state distribution shifts.

Selected tasks for testing the proposed framework. (a) Bucket loading. (b) Open drawer.

Exp2: Validation of the effectiveness of the proposed new action selection method using the open drawer task. (a) Q-value estimation with original IBRL. (b) Q-value estimation with our proposed action selection module. (c) BC loss. (d) Mean reward.

Exp3: Validation of the effectiveness of the proposed new action selection method using the bucket loading task. (a) Q-value estimation with original IBRL. (b) Q-value estimation with proposed new IBRL. (c) BC loss. (d) Mean reward.

Exp4: Validation of the effectiveness of SAC using the open drawer task. (a) Dense reward setting. (b) Sparse reward setting.

Exp5: Validation of the effectiveness of SAC using the bucket loading task.

Exp6: Validation of the robustness of our framework toward varying demonstration qualities using the open drawer task. (a) Comparison between the IL policy and offline RL policy. (b) Reward convergence with the varying demonstration qualities.

Illustration of the implementation of applying the proposed framework to the automatic wheel loader loading task.

Comparison of the estimated external torque during penetration between simulation and real-world experiments. In the real-world experiment (orange), the external torque is measured while loading dry sand. In the simulation experiment, the external torque (green and blue) is generated by loading sand and stone piles, using the same penetration motion as in the real-world experiment.

States–action pairs for training the Ref policy. Each curve represents the data recorded in one bucket loading demonstration.

Three phases of the wheel loader loading process.

Experiment site showing MUSTANG 2040 operating in wet sand and stone pile fields.

Sim-to-real results of 15 trials for loading stone with different pile geometries. Each curve represents the data recorded in one bucket-loading sim-to-real experiment.

Tables

TABLE 1. Averaged rewards of evaluating each RL policy at the last time step over five episodes.

Task	IBRLTD3	OursTD3	IBRLSAC	OursSAC
Open drawer (dense)	1,055	2,735	2,747	3,455
Open drawer (sparse)	682.6	2,475	2,150	3,053
Bucket loading (sparse)	71.7	76.5	69.9	81.8

TABLE 2. Domain randomization parameters and their sampling ranges.

Domain randomization	Range
Density	[1700±100,2600±100]kg/m3
Pile geometry	( 25° , 45° , 55° )
Particle friction	[0.3,0.4]
White noises on observations	[-1e−4, 1e−4]

TABLE 3. Average bucket-fill rates for simulation and real-world experiments.

Material	Simulation	Real world
Sand	93.71%	85.81%
Stone	90.01%	78.77%

Full text

Introduction

1

Model-free deep reinforcement learning (DRL) has shown great potential in learning continuous control tasks in robotics (Allshire et al., 2022; Qi et al., 2023; Rudin et al., 2022; Haarnoja et al., 2018a; Nguyen and La, 2019; Ibarz et al., 2021). However, there remain challenges that limit the widespread applicability of these methods in real-world robotic applications. One major challenge is the poor sample efficiency of learning with model-free DRL; even relatively simple tasks can require millions of interaction steps, while learning policies from high-dimensional observations or complex environments may require significantly more interactions (Haarnoja et al., 2018b; Osinski et al., 2020; Raffin et al., 2021). A primary cause for the poor sample efficiency is on-policy learning (Haarnoja et al., 2018b) since some of the most widely used DRL algorithms, such as A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017), require new interactions with the environments for each gradient step. Consequently, on-policy DRL is often impractical for real-world systems as allowing untrained policies to interact with real systems can be both costly and dangerous. Even when learning occurs solely in simulation, it is still preferred to utilize previously collected data instead of starting from scratch (Levine et al., 2020). On the other hand, off-policy DRL methods improve sample efficiency by reusing past experience and have demonstrated strong performance on continuous control tasks (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b; Fujimoto et al. 2019; Fujimoto and Gu, 2021; Kumar et al., 2020). However, for complex robotics tasks where data collection itself is expensive, e.g., in construction machines, educational agents, or medical devices, even off-policy approaches become costly when the DRL policy requires extensive explorations. Under these scenarios, improving exploration efficiency is as crucial as sample efficiency to reduce the exploration needed for achieving a good policy.

Therefore, effectively leveraging prior demonstrations to facilitate efficient exploration is considered a promising strategy for the broad application of off-policy DRL in real-world industrial robotics. Two main research directions have emerged to achieve this goal:

Offline-to-online DRL

1.1

Pretraining DRL with prior expert demonstrations, along with continuous training with online data, has shown its impressive performance in exploration efficiency (Vecerik et al., 2017; Nair et al., 2020; Uchendu et al., 2023; Zhou et al., 2024; Goecks et al., 2019; Lee et al., 2022). Early studies initialize training by mixing offline demonstrations and online interaction in the replay buffer and use a prioritized replay mechanism to enable the reinforcement learning (RL) policy for efficient exploration (Vecerik et al., 2017; Song et al., 2022). More recent approaches separate offline pretraining from online fine-tuning and report superior exploration efficiency (Gao et al., 2018; Goecks et al., 2019; Nair et al., 2020; Lee et al., 2022). In offline training, a behavior cloning (BC) loss or Kullback–Leibler (KL) divergence is typically used to encourage the RL policy to closely follow the behavior policy, which is used to generate the demonstrations, thereby facilitating efficient exploration in online interactions. However, when transferring to the online interacting phase, some methods are required to “recalibrate” the offline Q-estimates to the new online distribution to maintain learning stability and mitigate forgetting of pre-trained initializations (Nair et al., 2020; Uchendu et al., 2023; Ball et al., 2023).

DRL-Ref policy

1.2

Some novel studies have proposed to explicitly integrate a reference policy, trained from the prior demonstration to guide DRL training (Zhang et al., 2023; Hu et al., 2024). In these works, a stand-alone reference policy is trained using offline demonstration and then used to provide additional guidance in the DRL online learning phase. In this work, we consider the imitation bootstrapped reinforcement learning (IBRL) framework as an ideal approach for learning robotics tasks with prior demonstrations as it prevents catastrophic forgetting of pre-trained initializations and automatically balances offline and online training (Hu et al., 2024).

However, the IBRL framework is built on off-policy RL and imitation learning (IL). It risks the same challenges posed by bootstrapping errors in off-policy RL (Kumar et al., 2019, Kumar et al., 2020, Fujimoto et al., 2019, Fujimoto and Gu, 2021), where the target critic and actor networks are updated using out-of-distribution (OOD) actions with overestimated Q-values (Kumar et al., 2019, Kumar et al., 2020). Meanwhile, the IL policy in IBRL could also face the state distribution shift (Hussein et al., 2017), when OOD actions keep getting selected. To address these challenges, in this work, we propose an exploration-efficient DRL with reference (DRLR) policy framework, as shown in Figure 1, and summarize our contributions as follows:

1. Identify and analyze the main cause of the failure cases trained with the IBRL framework: distribution shift due to the bootstrapping error.
2. Propose a simple action selection module and use a maximum entropy RL to mitigate inefficient explorations caused by bootstrapping errors and convergence on a sub-optimal policy due to overfitting.
3. Demonstrate the effectiveness and robustness of the proposed framework on tasks with both low and high state–action dimensions and demonstrations of different quality.
4. Showcase an implementation and deployment of the proposed framework on a real industrial task.

Overview of the proposed exploration-efficient DRLR framework. The proposed framework extends a sample-efficient DRL-Ref method with a simple action selection module to mitigate inefficient explorations caused by (1) bootstrapping errors leading to the RL policy selecting out-of-distribution actions; (2) Ref policy failing to provide good actions under state distribution shifts.

Problem statement

2

The proposed framework is generalized toward learning robotics tasks with the following problems: 1) collecting a large amount of data is costly. 2) Learning requires extensive interactions. 3) A small number of expert demonstrations are available. Based on the characteristics, bucket loading (Shen and Sloth, 2024a) and open drawer (Makoviychuk et al., 2021a) tasks are selected to evaluate the effectiveness of the proposed framework. The task environments are shown in Figure 2.

Selected tasks for testing the proposed framework. (a) Bucket loading. (b) Open drawer.

Compared to the selected DRL-Ref framework, IBRL, the proposed framework attempts to mitigate distribution shift caused by bootstrapping errors and prevent convergence to a sub-optimal policy from overfitting to the demonstrations.

Bootstrapping error can arise in off-policy RL when the value function is updated using Bellman backups. It occurs because the target value function and policy are updated using OOD actions with overestimated Q-values (Kumar et al., 2010). Studies have shown that bootstrapping error can lead to unstable training and even divergence from the optimal policy (Kumar et al., 2019, Kumar et al., 2020), particularly when the current policy output is far from the behavior policy, which is used to generate the transitions in the replay buffer (Fujimoto et al., 2019; Fujimoto and Gu, 2021; Kumar et al., 2020; Kumar et al., 2019).

In the IBRL, the critic (value) function’s parameters $[eqn]$ are updated with the following Bellman backup (Hu et al., 2024):

$$[eqn]$$

where

$$[eqn]$$

Here, $[eqn]$ is the estimated Q-value with states and actions sampled from the replay buffer $[eqn]$ , while the target value $[eqn]$ in Equation 2 is estimated using the current RL policy $[eqn]$ or IL policy $[eqn]$ . IBRL training starts with a replay buffer mixed with expert demonstrations and transitions collected during interactions, which introduces a mismatch between the current RL policy and behavior policy. Although IBRL allows for selecting actions from the IL policy, whose output is closer to the behavior policy in the demonstration, it relies on an accurate value estimation between $[eqn]$ and $[eqn]$ . However, because of the exploration noises during the online interaction, the future rollout states $[eqn]$ sampled from $[eqn]$ are likely OOD relative to the offline demo buffer, $[eqn]$ (Lee et al., 2022; Nakamoto et al., 2023; Zhou et al., 2024). When the IL policy proposes actions in these OOD states, the critic networks have no prior data for these state–action pairs and could assign a lower Q-value than the OOD actions proposed by the RL agent. As a result, the lower bounds brought by the IL policy fail if the RL policy is updated with bad OOD actions with an overestimated Q-value. Such errors could be corrected by attempting the OOD action during online interaction and observing its actual Q-value, but this, in turn, leads to insufficient policy exploration. Thus, finding a reliable and calibrated Q-value estimation is crucial for mitigating the bootstrapping error (Nakamoto et al., 2023).

Another disadvantage of bootstrapping error is that OOD actions selected by the RL policy during online interaction can lead to state distribution shift. When the IL agent fails to provide high-quality actions for the unseen interaction states, the exploration efficiency of IBRL will be degraded. Furthermore, although the IBRL has stated that both twin delayed DDPG (TD3) and soft actor–critic (SAC) can be used as RL policies for continuous control tasks (Hu et al., 2024), the authors exclusively used TD3 in their experiments due to its strong performance and high sample efficiency in challenging image-based RL settings. However, we argue that the deterministic RL algorithm, TD3, is less suitable for high-dimensional, continuous state-based tasks as it is more prone to overfitting offline data, converging to sub-optimal policies, and suffering from inefficient exploration (Haarnoja et al., 2018b). To prevent the RL policy from convergence to a sub-optimal policy because of overfitting, a maximum entropy stochastic RL method, SAC, is adopted.

Preliminaries

3

This section presents an overview of maximum entropy DRL and IBRL.

Maximum entropy deep reinforcement learning

3.1

For sample efficiency, off-policy DRL methods have been widely studied due to their ability to learn from past experiences. However, studies have also found that the off-policy DRL method struggles to maintain stability and convergence in high-dimensional continuous state–action spaces (Haarnoja et al., 2018b). To address this challenge, maximum entropy DRL has been proposed.

As the state–action spaces are continuous in the selected robotics tasks, we consider a Markov decision process (MDP) with continuous state–action spaces: an agent explores and interacts with an environment, at each time step $[eqn]$ ; the agent observes the state $[eqn]$ , takes action $[eqn]$ based on the RL policy $[eqn]$ with parameters $[eqn]$ , and receives rewards $[eqn]$ . Different from standard RL, which aims to find a policy that maximizes the expected return in Equation 3

$$[eqn]$$

maximum entropy DRL aims to maximize the discounted reward and expected policy entropy $[eqn]$ at each time step in Equation 4:

$$[eqn]$$

where $[eqn]$ is the terminal time step, $[eqn]$ is the discount factor, and $[eqn]$ is the temperature parameter, which determines the relative importance of the entropy term against the reward and thus controls the stochasticity of the optimal policy (Haarnoja et al., 2018b). With this objective, maximum entropy DRL methods have shown great potential in DRL-efficient online exploration with sparse reward settings (Hiraoka et al., 2021; Ball et al., 2023), consistent with the goal of this paper.

To apply maximum entropy RL in continuous spaces, one of the widely used methods, SAC (Haarnoja et al., 2018b), is applied.

Imitation bootstrapped reinforcement learning

3.2

IBRL is a sample-efficient DRL framework that combines a stand-alone IL policy with an off-policy DRL policy (Hu et al., 2024). First, IBRL requires an IL policy $[eqn]$ trained using expert demonstrations $[eqn]$ . The goal of $[eqn]$ is to mimic an expert behavior and can be trained by minimizing a BC loss $[eqn]$ in Equation 5:

$$[eqn]$$

Then IBRL leverages trained $[eqn]$ to help the DRL policy $[eqn]$ with online exploration and its target value estimation, referred to as the actor proposal phase and the bootstrap proposal phase, respectively. In the actor proposal phase, IBRL selects between an IL action, $[eqn]$ , and an RL action, $[eqn]$ . The action with a higher Q-value computed by the target critic networks, $[eqn]$ , is selected for the online interaction. Thus, the action selection module in IBRL is defined in Equation 6:

$$[eqn]$$

Furthermore, to prevent local optimum Q-value update, the soft version of IBRL selects actions according to a Boltzmann distribution over Q-values instead of considering $[eqn]$ .

Similarly, in the bootstrap proposal phase, the future rollout will be carried out by selecting the action by $[eqn]$ or $[eqn]$ between $[eqn]$ . The critic networks $[eqn]$ are updated as presented in Equation 1. The RL policy network, $[eqn]$ , is updated as in the selected off-policy DRL.

Methods

4

To reduce the exploration time wasted in correcting unreliable overestimated Q-values and, in turn, improve exploration efficiency, it is crucial for the policy to favor distributions with more stable Q-values. This motivates selecting batches with reliable Q-value evaluations when updating both the critic and policy networks. Previous studies have shown that the Q-value estimates of $[eqn]$ are only reliable when $[eqn]$ is sampled from the same distributions as the dataset used to train $[eqn]$ (Kumar et al., 2019; Nakamoto et al., 2023). In our critic network update process, instead of selecting between $[eqn]$ and $[eqn]$ , where both $[eqn]$ and $[eqn]$ could be OOD state–action pairs, we propose to select between $[eqn]$ and $[eqn]$ , where $[eqn]$ is only sampled from $[eqn]$ . This modification ensures that $[eqn]$ is always from the same distribution as $[eqn]$ , providing a reliable and calibrated Q-value estimates of the reference policy, whose values are on the similar scale as the true return value of $[eqn]$ (Nakamoto et al., 2023). With $[eqn]$ fixed, we compare the mean estimated return of $[eqn]$ sampled from $[eqn]$ against the bootstrapping-error-free ground-truth mean return of $[eqn]$ , thereby reducing the accumulated bootstrapping error in the action selection process. Thus, Equation 2 when updating the critic network becomes (Equation 7)

$$[eqn]$$

Compared with IBRL, the key modification is a simple action selection module, denoted as $[eqn]$

$$[eqn]$$

where $[eqn]$ denotes the mean of estimated Q-values, $[eqn]$ is the state from $[eqn]$ , and $[eqn]$ is the state only sampled in $[eqn]$ .

In the bootstrap proposal phase, the future rollouts $[eqn]$ are sampled randomly from $[eqn]$ . One can select $[eqn]$ by finding the states closest to $[eqn]$ within $[eqn]$ , to enable more precise comparisons between nearby state–action pairs. However, for implementation simplicity, current $[eqn]$ is uniformly random-sampled from $[eqn]$ . By simple random sampling, the expected sample mean Q-value, $[eqn]$ , from each batch converges to the population mean Q-value of the expert buffer (Rice, 2007). Therefore, even though the comparison is made across different states, it remains valid because we have compared the mean Q-values of the distributions induced by the IL policy and the RL policy.

Similarly, to align the policy strategy in the online interaction phase with the policy selected to propose future rollouts, the same action selection module (Equation 8) is used. With fewer OOD actions getting selected, the state distribution shift is also mitigated. However, if $[eqn]$ fails to provide good or recovery actions toward any state distribution shift, the considered action selection module might fail as $[eqn]$ is not updated with fixed $[eqn]$ , and the same bad behavior from the reference policy is repeatedly selected. Therefore, to leverage this action selection module for enhanced exploration efficiency, the initial online exploration states should lie within or near those in $[eqn]$ , and the reference policy should remain robust under small shifts in the state distribution.

Furthermore, to prevent the RL policy from overfitting the demonstration dataset and converging on a sub-optimal policy, we propose to replace TD3 with SAC. In SAC, the critic parameter $[eqn]$ is updated by minimizing the soft Bellman residual:

$$[eqn]$$

where $[eqn]$ is estimated using Equation 10:

$$[eqn]$$

The stochastic actor parameter $[eqn]$ is updated by minimizing the expected KL-divergence:

$$[eqn]$$

where the stochastic action is $[eqn]$ and $[eqn]$ is an input noise distribution, sampled from some fixed distribution (Haarnoja et al., 2018b). We propose that the distribution can be the demonstration $[eqn]$ , but in this study, we only consider a simple Gaussian distribution $[eqn]$ . $[eqn]$ is the log-probability of the stochastic action $[eqn]$ under the current policy $[eqn]$ .

Finally, to leverage the robustness of the proposed framework toward the quality of the demonstration, we propose to choose offline DRL as the reference policy (IL policy in the IBRL framework) when the quality of the demonstration is unknown or imperfect. With strong sequential decision-making ability, offline DRL can be more robust to the demonstration quality than IL methods (Kumar et al., 2020; Fujimoto and Gu, 2021).

Combining all the modifications, DRLR is introduced in Algorithm 1; our new modifications are marked in red.

Algorithm 1DRLR.

• 1: Input: Critic networks $[eqn]$ and target critic networks $[eqn]$ with random initial parameter values; policy network $[eqn]$ and target policy network $[eqn]$ ;
• 2: Initialize replay buffer $[eqn]$ and expert buffer $[eqn]$ ;
• 3: Train an reference policy $[eqn]$ with expert buffer $[eqn]$ by IL or offline RL.
• 4: for each episode $[eqn]$ do
• 5:  Reset environment to initial state $[eqn]$ .
• 6:  for each time step $[eqn]$ do
• 7:   Observe $[eqn]$ from the environment and compute IL action $[eqn]$ and RL stochastic action $[eqn]$
• 8:   Compute Q-values from the target critic networks $[eqn]$ .
• 9:   Execute $[eqn]$ based on (equation 8).
• 10:   Store transition $[eqn]$ in replay buffer $[eqn]$ .
• 11:   Randomly sample a minibatch of $[eqn]$ transitions from the replay buffer $[eqn]$ and $[eqn]$ , respectively.
• 12:   Update critic network parameters using Equation 9.
• 13:   Update actor network parameters using Equation 11.
• 14:   Update target networks.
• 15:  end for
• 16: end for

Experiment design and evaluation

5

In this section, experiments are designed and conducted in the simulation to evaluate the proposed method. The experimental design and evaluation aim to answer the following core questions.

How generalizable is DRLR across environments with varying reward densities and state–action space complexities?

5.1

To answer the question, the tasks selected in the problem statement are studied under both dense reward and sparse reward settings. For the bucket loading task, the state and action dimensions are 4 and 3, respectively. The details, such as reward design, domain randomization, and prior demonstration collection, are provided in Section 6. For the open drawer task, the state and action dimensions are 23 and 9, respectively. The details of the open drawer task are provided in Makoviychuk et al. (2021a). The original reward design for the open drawer task is dense and contains distance reward, open drawer reward, and some bonus reward for opening the drawer properly. To study the same task with a sparse reward setting, we simply set the distance reward gain to 0. To collect simulated demonstrations for the open drawer task, a TD3 policy was trained with dense, human-designed rewards. A total of 30 prior trajectories are recorded by evaluating the trained TD3 with random noise added to the policy output.

Both tasks are trained with Isaac Gym (Makoviychuk et al., 2021b). All experiments with the open drawer task were run with 10 parallel environments, using two different random seeds (10 and 11) to ensure robustness and reproducibility. All experiments with the bucket loading task were run in a single environment, using two different random seeds (10 and 11). The detailed configurations for training each task are shown in Section 8.1.

Question A is answered through the following evaluation results: Figure 5 demonstrates the performance of DRLR to learn the open drawer task with both sparse reward and dense reward, by achieving the highest reward in both reward settings; the results validate the robustness of DRLR toward varying reward densities. Figures 5, 6 present the performance of DRLR under different state–action space complexities. By outperforming IBRL on the open drawer task and achieving comparable reward in the bucket loading task, the results validate the ability of DRLR to generalize across varying levels of state–action space complexities.

How effective is the proposed action selection module in addressing the bootstrapping error and improving exploration efficiency during learning compared to IBRL?

5.2

To examine the effectiveness of the action selection module in addressing bootstrapping error and improving exploration efficiency, we conducted experiments in which only the action selection module of the original IBRL framework was replaced. The reference policy used is the IL policy, while the RL policy remains TD3 in both setups. Four criteria are recorded during training: 1) the Q-value of the Ref policy during action selection in the online interaction phase; 2) the Q-value of the RL policy during action selection in the online interaction phase; 3) BC loss: $[eqn]$ , for measuring the difference between sampled actions in the replay buffer and the actions output by the RL policy; 4) reward convergence over training steps. Figures 3, 4 present a comparison of the considered criteria between the baseline IBRL and our proposed method across two selected tasks in the sparse reward setting.

Exp2: Validation of the effectiveness of the proposed new action selection method using the open drawer task. (a) Q-value estimation with original IBRL. (b) Q-value estimation with our proposed action selection module. (c) BC loss. (d) Mean reward.

Exp3: Validation of the effectiveness of the proposed new action selection method using the bucket loading task. (a) Q-value estimation with original IBRL. (b) Q-value estimation with proposed new IBRL. (c) BC loss. (d) Mean reward.

The results for the open drawer task are shown in Figure 3. In Figure 3a, we compare the Q-value of the Ref policy and RL policy during action selection in the online interaction phase in the IBRL. The Q-values of the Ref policy appear closely estimated to that of the RL policy, and both Q-values have high variances during training. Combining the results of the BC loss between sampled actions and the agent’s output actions in Figure 3c indicates a mismatch between the updated policy and the behavior policy, suggesting that OOD actions are being selected due to the bootstrapping error discussed in Section 2. As a result, the Ref policy failed to get selected to provide reliable guidance, as reflected in the degraded performance in Figure 3d. Figure 3b presents a stable Q-value estimation of the Ref policy and a clear higher mean value compared with the RL policy in the early training steps, which aligns with the core idea of the IBRL framework. The corresponding BC loss in Figure 3c is significantly reduced by approximately $[eqn]$ compared to the BC loss of IBRL, indicating that the bootstrapping error is effectively mitigated using our action selection method. Consequently, the Ref policy succeeded in efficient guidance throughout the RL training, as demonstrated by the improved reward convergence in Figure 3d. The proposed action selection module achieved a mean reward approximately four times higher than IBRL during the interaction steps.

The results for the bucket loading task are shown in Figure 4. Notably, the experiments of the bucket loading task were run in a single environment since it is computationally expensive to simulate thousands of particles in parallel environments. Thus, the results of the bucket loading tasks have higher variance than those of the open drawer task, where 10 environments are running in parallel. The results suggest that the action selection module has less effect on the low-dimensional state–action task, and the original IBRL can already score a near-optimal reward. This can also be attributed to the performance of the Ref policy. If the RL policy can easily acquire a higher Q-value than the Ref policy, the effect of our action selection module will be limited. Nevertheless, the stable Q-value estimation of the Ref policy in Figure 4b still validates the effectiveness of our action selection module in maintaining reliable Q-value estimations.

How effective is SAC in improving exploration efficiency during learning compared to the initial IBRL?

5.3

To examine the effectiveness of the SAC in improving exploration efficiency, we conducted 1) the original IBRL, denoted as $[eqn]$ ; 2) the IBRL with our action selection module, denoted as $[eqn]$ ; 3) the IBRL with SAC to be the RL policy, denoted as $[eqn]$ ; 4) our DRLR framework, denoted as $[eqn]$ . The Ref policy remains the IL policy in all setups.

The reward convergence over training steps is recorded as the main evaluation criteria. Figures 5, 6 present a comparison of the considered experiments across two selected tasks. The results for the open drawer task with varied reward settings are shown in Figure 5. The reward convergence suggests that with the same training steps, the experiments with $[eqn]$ are able to explore higher rewards than those with $[eqn]$ , which converged on a sub-optimal reward. The results for the bucket loading task in sparse reward settings are shown in Figure 6. The results suggest that our method and IBRL achieve similar performance in low-dimensional state–action spaces.

Exp4: Validation of the effectiveness of SAC using the open drawer task. (a) Dense reward setting. (b) Sparse reward setting.

Exp5: Validation of the effectiveness of SAC using the bucket loading task.

The final evaluation results of each algorithm across two tasks are shown in Table 1. Table 1 shows that DRLR achieves the best evaluation performance in both tasks. In the open drawer task with sparse rewards, DRLR improves the averaged reward by approximately $[eqn]$ , showing a dramatic improvement.

What is the impact of demonstration quality on the performance of our method?

5.4

To evaluate the robustness of the proposed method toward varying demonstration qualities, the following experiments were conducted: we fill the demonstration dataset with 1) 50% data from the random policy, denoted as $[eqn]$ . 2) Sub-optimal demo: noise is added to the expert policy outputs. For simplicity, a BC policy is selected as the IL policy. A minimalist approach to offline RL, known as TD3 + BC (Fujimoto and Gu, 2021), is selected as our Ref policy. Due to the complexity involved in designing such experiments, only the open drawer task with sparse rewards, which is the most difficult to learn, is evaluated in the experiments. The results are shown in Figure 7. Figure 7a demonstrates that TD3 + BC can learn a good policy even from $[eqn]$ , while BC cannot. Furthermore, TD3 + BC also learns a better policy using the sub-optimal demo. Figure 7b validates the robustness of our method toward varying demonstration qualities, by achieving the same level of rewards with both datasets.

Exp6: Validation of the robustness of our framework toward varying demonstration qualities using the open drawer task. (a) Comparison between the IL policy and offline RL policy. (b) Reward convergence with the varying demonstration qualities.

To this end, we have demonstrated the effectiveness of the proposed method. The method is also applied to a real industrial application to showcase the implementation process and sim-to-real performance.

Real industrial applications

6

This section presents an application of the proposed framework to the wheel loader loading task, where only a limited number of expert demonstrations are allowed to demonstrate the data efficiency. The detailed implementation is illustrated in Figure 8.

Illustration of the implementation of applying the proposed framework to the automatic wheel loader loading task.

Bucket–media simulation

6.1

Before learning with the proposed framework, it is important to create an environment similar to the real world to enable policy exploration while applying domain randomization to deal with observation shifts. In the simulation, the wheel loader is configured with the same dynamic parameters obtained from a real machine. Because it is impractical to directly model the hydraulic actuation force or the bucket–media interaction force under different materials and geometries, this paper attempts to regularize the external torque rather than modeling it. We proposed to use admittance controllers to decrease the variances in the external torque by changing the position reference. The implementation of the admittance controller is provided in the Supplementary Appendices.

Table 2 shows the parameters we randomized to simulate bucket–media interactions with different pile geometries and pile materials. A comparison of the estimated external torque during penetration of the pile between simulation and real-world experiments is presented in Figure 9. Different from real-world settings, the external torque is estimated from contact sensors in the simulation, due to the poor performance of the force sensor in Isaac Gym.

Comparison of the estimated external torque during penetration between simulation and real-world experiments. In the real-world experiment (orange), the external torque is measured while loading dry sand. In the simulation experiment, the external torque (green and blue) is generated by loading sand and stone piles, using the same penetration motion as in the real-world experiment.

DRLR implementation

6.2

Both the Ref and RL policies have four inputs, namely, $[eqn]$ , representing boom joint position, bucket joint position, advancing length, and estimated external torque, and three outputs, namely, $[eqn]$ , where $[eqn]$ are desired position references for boom and bucket joint positions, respectively, and $[eqn]$ is the desired torque reference for admittance control that is only used during penetration.

To train the Ref policy, 10 expert demonstrations of loading dry sand piles with changing pile geometries are recorded. During demonstration, $[eqn]$ are directly used as inputs, and $[eqn]$ is scaled at $[eqn]$ . Position references are acquired from the forward dynamics of the actuation signals sent in the demonstrations; they are first normalized and then used as $[eqn]$ , while scaled $[eqn]$ is directly assigned as $[eqn]$ . The state–action pairs that are used for training the reference policy are shown in Figure 10. For simplicity, BC is used to train the reference policy.

States–action pairs for training the Ref policy. Each curve represents the data recorded in one bucket loading demonstration.

The wheel loader loading process can be divided into three phases, as shown in Figure 11: penetrate, shovel, and lift (Sarata et al., 2004).

Three phases of the wheel loader loading process.

To train DRL, the bucket loading task is divided into two sub-tasks as shown in Equation 12:

$$[eqn]$$

In phase 1, $[eqn]$ , the boom and bucket penetrate the pile with an admittance controller tracking $[eqn]$ , and the loader moves forward with a constant velocity. In phases 2 and 3, $[eqn]$ , the controller switches to an inverse dynamics controller with only tracking the position references $[eqn]$ , and the loader stops moving forward. The transition between $[eqn]$ and $[eqn]$ is determined by the point at which the loader stops moving forward. Based on observing the demonstrations, this transition is identified when the desired bucket reference position $[eqn]$ surpasses approximately −0.5.

The goal for the bucket loading task is to achieve a full bucket-fill rate and the boom–bucket joint reaching its designated end position, corresponding to the maximum allowable value within the position reference range. This leads to a natural sparse reward setting, where the reward only occurs at the end of the tasks. However, sparse reward requires a longer training time because it is more difficult for the RL agent to explore than dense reward settings. Although Shen and Sloth (2024a) demonstrated a successful performance with dense rewards, designing such rewards is challenging and may lead to sub-optimal actions. Since our framework has shown robust performance in sparse reward settings, a simpler sparse reward setting is designed in Equation 13:

$$[eqn]$$

where $[eqn]$ represents the final step of an episode. A loading failure $[eqn]$ occurs if the bucket-fill rate reward $[eqn]$ and the end reward $[eqn]$ do not achieve at least half of their maximum designed values by the end step $[eqn]$ . The rewards $[eqn]$ and $[eqn]$ are defined as Equation 14:

$$[eqn]$$

where $[eqn]$ is the bucket capacity, $[eqn]$ is the current bucket load volume, and $[eqn]$ , where $[eqn]$ is the particle density and $[eqn]$ is the length of boom. $[eqn]$ is the Euclidean distance between the current boom–bucket joint position and the end position, while $[eqn]$ is the Euclidean distance between the initial boom–bucket joint position and the end position.

Sim-to-real results

6.3

The reward convergence results learning the bucket loading task are shown in Figure 6. The trained actor is deployed on a real machine MUSTANG 2040 operating in wet sand and stone pile fields. The experiment site is shown in Figure 12.

Experiment site showing MUSTANG 2040 operating in wet sand and stone pile fields.

In the experiments, the inputs $[eqn]$ are measured in radians with inertial measurement units (IMUs) mounted on the boom and bucket. $[eqn]$ denotes the forwarding distance of the loader, determined using GNSS antennas mounted on the machine. $[eqn]$ is computed based on the pressure sensor readings obtained from both sides of the hydraulic pistons in the boom and bucket hydraulic pump. All the sensors operate at an update rate of $[eqn]$ . The outputs $[eqn]$ are from the deployed Neural Networks (NNs), while the loader’s forwarding motion is manually controlled by an operator at a random speed. The operator halts the forward motion upon noticing the boom’s lift.

First, a two-sided admittance controller with both position and torque reference is tested. However, due to the high compaction nature of wet sand and stone pile, the downward curl of the bucket generates extremely large normal forces, causing the admittance controller to fail to track $[eqn]$ and, consequently, leading to boom and bucket vibrations during penetration and unstable outputs from the deployed actor network. These unstable NN outputs could result from a state distribution shift caused by the large normal forces during interaction with compacted material. In the simulation environment, such a compaction effect is not accurately modeled as the material pile is simulated using discrete particles that lack adhesive or cohesive properties. A penalty for causing such unsafe behavior should be considered in the future reward design.

Due to safety and stable performance, only a one-sided admittance controller is tested in the following experiments with position reference $[eqn]$ and $[eqn]$ N.m to prevent the bucket from getting stuck.

To evaluate the policy, 25 experiments were carried out, involving 10 trials for loading wet sand and 15 trials for loading stone. Sim-to-real results for loading stones are presented in Figure 13. Despite changing environments, including pile geometries, material types, and forwarding velocities, all the experiments successfully loaded and lifted the materials. The average bucket-fill rates for loading sand and stone in the simulation and real-world experiments are provided in Table 3. To compare the sim-to-real performance in terms of the bucket-fill rate, the bucket-fill rates in simulation are also recorded and averaged over five episodes. The bucket-fill rate differences between simulation and real-world experiments may stem from environmental uncertainties present under real-world conditions, such as the irregular pile shapes.

Sim-to-real results of 15 trials for loading stone with different pile geometries. Each curve represents the data recorded in one bucket-loading sim-to-real experiment.

Conclusion

7

This paper proposes and implements an exploration-efficient DRLR framework to reduce the need for extensive interaction when applying off-policy DRL to real-world robotic tasks. The designed experiments empirically validate the effectiveness of our framework in mitigating bootstrapping errors and addressing convergence to sub-optimal policies, ultimately reducing the exploration required to attain high-performing policies compared to IBRL. Furthermore, we demonstrated the implementation details for using the DRLR framework on a real industrial robotics task, wheel loader bucket loading. The sim-to-real results validate the successful deployment of the considered framework, demonstrating its potential for application to complex robotic tasks.

In future work, one could improve the action selection module by selecting $[eqn]$ by determining the states closest to $[eqn]$ within $[eqn]$ using Euclidean or Mahalanobis distance, thereby facilitating more precise comparisons between neighboring state–action pairs. To better demonstrate the advantages of DRLR, it is necessary to compare it against established offline-to-online DRL baselines that explicitly addressed bootstrapping errors, such as CAL-QL, RLPD, and WSRL (Nakamoto et al., 2023; Ball et al., 2023; Zhou et al. 2024).

Moreover, one could also consider using deep ensembles to quantify the uncertainties in the demonstrations and utilize these uncertainties as prior data for the SAC entropy. Integrating the concepts of active learning and uncertainty-aware RL into the proposed framework could further improve the exploration efficiency.