Poke and Strike: Learning Task-Informed Exploration Policies

TL;DR

This paper introduces a reinforcement learning approach for robotic exploration that efficiently identifies physical object properties for successful task execution, significantly reducing exploration time and increasing success rates.

Contribution

It presents a novel task-informed exploration policy that uses sensitivity-based rewards and an uncertainty mechanism to optimize property estimation in robotic tasks.

Findings

Achieves 90% success rate in striking task with minimal exploration time

Outperforms baselines with at most 40% success or inefficient retraining

Successfully validates property identification and task adjustment on physical robot

Abstract

In many dynamic robotic tasks, such as striking pucks into a goal outside the reachable workspace, the robot must first identify the relevant physical properties of the object for successful task execution, as it is unable to recover from failure or retry without human intervention. To address this challenge, we propose a task-informed exploration approach, based on reinforcement learning, that trains an exploration policy using rewards automatically generated from the sensitivity of a privileged task policy to errors in estimated properties. We also introduce an uncertainty-based mechanism to determine when to transition from exploration to task execution, ensuring sufficient property estimation accuracy with minimal exploration time. Our method achieves a 90% success rate on the striking task with an average exploration time under 1.2 seconds, significantly outperforming baselines that achieve at most 40% success or require inefficient querying and retraining in a simulator at test time. Additionally, we demonstrate that our task-informed rewards capture the relative importance of physical properties in both the striking task and the classical CartPole example. Finally, we validate our approach by demonstrating its ability to identify object properties and adjust task execution in a physical setup using the KUKA iiwa robot arm.

Figures

Tables

Table 1. Table 1: One-shot task must succeed in a single attempt. Immediate task execution after exploration, without simulator queries or policy retraining. Adaptive exploration time automatically adjusted for sufficient but minimal exploration. Overlapping action space shared between exploration and task execution.

Method	One-shot task	Immediate task execution	Adaptive exploration time	Overlapping action space
BayesSim [12]	✗	✗	✗	✓
AdaptSim [17]	✗	✓	✗	✓
IRP [18]	✗	✓	✗	✓
DISaM [21]	✗	✓	✓	✗
Task-Oriented [9]	✓	✗	✗	✓
ASID [1]	✓	✗	✗	✓
Ours	✓	✓	✓	✓

Table 2. Table 2: Randomization range of physical properties.

Task	Physical Property	Distribution
Striking	Static Friction	$𝒰[0.05,0.3]$
	Dynamic Friction	$𝒰[0.05,0.3]$
	Restitution	$𝒰[0.0,1.0]$
	Mass ($\mathrm{kg}$)	$𝒰[0.02,0.5]$
	Center of Mass Distance ($m$)	$𝒰[0,0.7\times \text{puck radius}]$
	Center of Mass Angle ($\mathrm{rad}$)	$𝒰[0,2\pi ]$
Edge Pushing	Static Friction	$𝒰[0.05,0.3]$
	Dynamic Friction	$𝒰[0.05,0.3]$
	Restitution	$𝒰[0.0,1.0]$
	Mass ($\mathrm{kg}$)	$𝒰[0.02,0.5]$
	Center of Mass Distance ($m$)	$𝒰[0,0.7\times \text{box width}]$
	Center of Mass Angle ($\mathrm{rad}$)	$𝒰[0,2\pi ]$
CartPole	Cart Joint Friction	$𝒰[0.0,1.0]$
	Pole Joint Friction	$𝒰[0.0,1.0]$
	Cart Mass ($\mathrm{kg}$)	$𝒰[0.1,10.0]$
	Pole Mass ($\mathrm{kg}$)	$𝒰[0.1,10.0]$

Table 3. Table 3: Key hyperparameters used for PPO training of the task and exploration policies.

Hyperparameter	Value
Parallel Environments	8192
Initial Learning Rate ($\alpha$)	$3\times {10}^{-4}$
Optimizer	Adam
Batch Size	4096
Rollout Steps	Task: 16, Exploration: 64
Number of Epochs	8
Discount Factor ($\gamma$)	0.99
GAE Lambda ($\lambda$)	0.95
Clip Range ($ϵ$)	0.2
Entropy Coefficient ($\beta$)	0.0
Total Timesteps	24,000

Table 4. Table 4: Summary of step-wise noise, simulating sensor noise at each timestep, and episodic noise, simulating calibration and systematic errors across the entire episode, for the Striking and CartPole tasks.

Task	Input	Noise Distribution
		Step	Episode
Striking/ Edge Pushing	Object position ($m$)	$𝒩(0,{0.0025}^2)$	$𝒩(0,{0.0025}^2)$
	Object orientation ($\mathrm{rad}$)	$𝒩(0,{0.01}^2)$	$𝒩(0,{0.01}^2)$
	Pusher position ($m$)	$𝒩(0,{0.0025}^2)$	$𝒩(0,{0.0025}^2)$
CartPole	Pole joint position ($\mathrm{rad}$)	$𝒩(0,{0.15}^2)$	$𝒩(0,{0.15}^2)$
	Pole joint velocity ($\mathrm{rad}/s$)	$𝒩(0,{0.15}^2)$	$𝒩(0,{0.15}^2)$
	Cart joint position ($m$)	$𝒩(0,{0.05}^2)$	$𝒩(0,{0.05}^2)$
	Cart joint velocity ($m/s$)	$𝒩(0,{0.05}^2)$	$𝒩(0,{0.05}^2)$

Table 5. Table 5: Ablation on task-agnostic vs. task-informed exploration rewards. We provide detailed results in  Section   C.5 .

Method	Success Rate (%)
	Exploration	Task
Task-agnostic Min.	8.0	2.9
Task-agnostic Max.	91.7	47.3
Task-informed	92.3	90.1

Equations

$$r_{\text{exp}}=\{\begin{array}{ll}{r}_{\text{estimation}}& \text{if }\forall j{\epsilon }_{\text{estimation},j}<{\epsilon }_{\text{threshold},j}\\ r_{\text{failure}}& \text{otherwise}\end{array},$$

$${\epsilon }_{\text{estimation},j}=\left|{\varphi }_j^{\ast }-\widehat{{\varphi }_j}\right|.$$

$$g_j({\epsilon }_{\text{threshold},j})\ge p\cdot \underset{\epsilon }{\max }{g}_j(\epsilon )$$

$${\hat{\varphi }}_t=\frac{1}{M}\sum _{i=1}^M{\hat{\varphi }}_{i,t},{\hat{\Sigma }}_t=\underset{\begin{array}{c}\text{mean of the}\\ \text{individual covariances}\end{array}}{\underbrace{\frac{1}{M}\sum _{i=1}^M{\hat{\Sigma }}_{i,t}}}+\underset{\begin{array}{c}\text{covariance of the}\\ \text{mixture means}\end{array}}{\underbrace{\frac{1}{M}\sum _{i=1}^M\left({\hat{\varphi }}_{i,t}{\hat{\varphi }}_{i,t}^T\right)-{\hat{\varphi }}_t{\hat{\varphi }}_t^T}}.\text{vskip}-1.00006pt$$

$${\mathcal{L}}_{est}=\frac{1}{2}\ln \left(\left|{\hat{\Sigma }}_{i,t}\right|\right)+\frac{1}{2}{\left({\varphi }^{\ast }-{\hat{\varphi }}_{i,t}\right)}^T{\hat{\Sigma }}_{i,t}^{-1}\left({\varphi }^{\ast }-{\hat{\varphi }}_{i,t}\right).$$

Full text

Keywords: Interactive Perception, Manipulation, Reinforcement Learning, System Identification

1 Introduction

Exploratory motions are essential for identifying system parameters and adjusting actions accordingly when executing interactive tasks. Unlike visual properties, identifying physical properties—such as friction, mass, weight distribution, and restitution—requires active interaction with objects. For example, a robot might push an object to identify friction for striking [1], poke a sponge to evaluate stiffness for wiping [2], or shake [3] or stir [4] liquids to determine viscosity for pouring. While such exploratory behaviors naturally emerge in humans [5], enabling robots to autonomously explore through physical interactions remains a significant challenge.

Task-informed exploration. A key challenge in achieving exploratory behaviors is coming up with informative exploration strategies that identify task-relevant properties. Most existing approaches rely on pre-defined, human-designed exploratory motions tailored to specific tasks [3, 4, 6, 2, 7, 8], which can be suboptimal and cumbersome to obtain for every new task. Alternatively, Memmel et al. [1] propose learning informative exploratory motions given a set of properties of interest, while Liang et al. [9] introduce exploration policy learning guided by task information using reinforcement learning (RL), implicitly prioritizing the estimation of task-relevant properties. However, both approaches [1, 9] require inefficient simulator queries and re-optimization of task motions for each object with the identified properties at test time. Our approach, in contrast, enables immediate task execution after exploration, in addition to learning exploratory motions to identify task-relevant properties using rewards automatically generated from the task policy.

Uncertainty-based policy switching. Another challenge lies in the transition from exploration to task execution. Prior approaches often rely on a fixed-length exploration phase [1, 9], which may be either too short, leaving the robot with insufficient information, or excessively long, which is inefficient. To autonomously switch from exploration to task execution, the robot needs sufficiently accurate property estimates for a given task. However, the estimation accuracy is inaccessible outside of the simulated environment. Additionally, the robot must determine how accurately it needs to estimate each property to ensure task success. To address this, our approach computes uncertainty estimates and, in simulation, we determine the uncertainty thresholds based on task completion.

In summary, the contributions of this work are:

• •

We propose a task-informed exploration RL approach that trains an exploration policy using rewards automatically generated from the sensitivity of a privileged task policy to errors in estimated properties, leading to the identification of task-relevant properties.

• •

We demonstrate that autonomously discovered exploration rewards lead to improved task performance by more accurately identifying relevant properties.

• •

We introduce the simultaneous learning of an online estimator with the exploration policy, enabling estimation within distribution of the exploratory observed states.

• •

We present a method that autonomously transitions from exploration to task execution using uncertainty in property estimates, with thresholds computed from task success.

• •

We validate our approach on a physical robotic setup, showing that it successfully adjusts task motions for different object properties.

2 Problem Statement

We address the problem of performing one-shot robotic tasks that involve interacting with objects or environments with unknown physical properties, where the robot is unable to recover from failure or retry without human intervention. Examples include striking pucks with varying friction to an unreachable goal, tossing objects with different centers of mass, or shooting a basketball with unknown restitution. These tasks require identifying the physical properties of the manipulated object prior to task execution, as initiating the task with incorrect assumptions about the properties can lead to irreversible failure—making online adaptation or correction unsuitable.

In this work, we seek to develop a method that addresses the following questions: (1) how to automatically discover informative exploratory motions for identifying task-relevant properties, when these motions differ from optimal task motion and vary depending on the task; (2) how to adaptively determine the timing to transition from exploration to task execution, ensuring sufficient property estimation accuracy for a given task while minimizing exploration time; and (3) how to execute the task immediately after exploration, without additional simulation queries or retraining at test time.

3 Related Work

Domain randomization. Domain randomization [10, 11] is a powerful technique in RL for enhancing the robustness of machine learning models and policies by introducing variability in environmental factors, such as the visual and physical properties of objects. While this approach improves the robustness of learnt policies across diverse environmental conditions, it often results in policies that exhibit average behavior [12, 1]. To adapt behavior based on the physical properties of objects, Ramos et al. [12] propose non-uniform domain randomization. However, this method requires obtaining the posterior distribution of object properties and retraining the task policy for those specific properties each time the robot executes the task. On the other hand, Peng et al. [13] use Recurrent Deterministic Policy Gradient (RDPG) in RL, aiming to infer dynamics by capturing temporal dependencies. While inspired by these methods, we find that they alone are insufficient for tasks requiring exploratory motions to identify physical properties prior to task execution.

System Identification. System identification [14] is crucial for modeling system dynamics from observed data. A key challenge is generating exploratory motions that yield informative observations for parameter identification. Many prior approaches rely on predefined motions [15, 7, 6, 8, 3, 4, 16]. Others iteratively update parameters through repeated task executions [17, 18]. However, optimal task motions may differ from those most informative for parameter identification. To address this, prior works optimize exploratory motions by minimizing property estimation errors [19], reducing state transition prediction errors [20], or maximizing Fisher information to generate observations that most effectively distinguish differences in properties [1]. However, these methods assume equal importance of all given parameters and fail to identify task-relevant ones.

Liang et al. [9] introduce task-oriented exploration using RL, where task rewards implicitly guide the exploration policy to focus on properties critical for task success. However, their approach incurs inefficient simulation queries and requires a fixed exploration duration to infer parameters by matching simulated and real-world observations, aiming to mitigate the out-of-distribution issue with learned estimators [19]. In contrast, our method trains the online property estimator simultaneously with the exploration policy to align the training data distribution, while enabling adaptive exploration length based on uncertainty estimates. Additionally, these prior works [9, 1] require re-optimization of the task policy at test time. Our method enables immediate task execution after exploration using an online property estimator and a privileged task policy. Dass et al. [21] also propose a dual-policy framework for task and information-seeking, but assume minimal action space overlap—precluding applicability in scenarios where exploratory and task actions interfere, such as physical property identification. In contrast, as summarized in Table 1, our method enables one-shot task execution immediately after exploration, with automatically determined exploration time, and without constraints on action spaces.

Privileged learning for robot control. Privileged learning [22] leverages information inaccessible during real-world deployment. This privileged information accelerates the learning process by providing the agent with critical knowledge, such as hidden states or ground truth parameters, which would otherwise require extensive exploration to uncover. Chen et al. [23] and subsequent works [24, 25, 26, 27] proposed training a student policy to imitate the behavior of a privileged teacher policy using partial observations. Del Aguila Ferrandis et al. [28] use a privileged policy to generate data for learning a state estimator, and Yu et al. [29] use for a property estimator; others incorporate privileged information into auxiliary losses [30, 31] or the critic network [32, 30]. While most existing work in these settings focuses on identifying hidden states [28, 24, 25, 26, 27, 31] or parameters [25, 26, 29] from passive observation during task execution, our work is the first to address the challenge of uncovering parameters, specifically physical property parameters in our case, that require active exploration motions different from task motions.

4 Method

We propose a task-informed exploration approach. The core idea is to leverage the privileged task policy to automatically generate rewards based on its sensitivity to errors in estimated properties. These rewards guide the training of the exploration policy to identify task-relevant properties, while simultaneously training the online property and uncertainty estimator. At deployment, the robot begins with the exploration policy, uses uncertainty estimates to transition to task execution, and immediately executes the task policy after exploration, adjusting its motion based on the final property estimates from the exploration.  Section A.1 outlines a detailed overview of the proposed method.

4.1 Problem Formulation

We formulate the problem as a family of Markov Decision Processes (MDPs) where the dynamics of the environment depends on the physical properties $\phi\in\Phi$ of the manipulated object. We assume that $\phi$ remains constant throughout the episode. Each instance takes the form of the tuple $M(\phi)=(S,A,P_{\phi},R,\gamma)$, with a finite horizon $H$, where $S$ is the state space, $A$ is the action space, $P_{\phi}$ is the transition model conditioned on $\phi$, $R$ is the reward function and $\gamma\in[0,1)$ is the discount factor. The objective is to learn a policy $\pi$ that maximizes the expected discounted return across different values of the physical properties $\phi$: $\max_{\begin{subarray}{c}\pi\end{subarray}}\mathbb{E}_{\phi\sim p(\phi),\pi,P_{\phi}}\left[\sum_{t=0}^{H-1}\gamma^{t}R(s_{t},a_{t},s_{t+1})\right]$. The physical properties $\phi$ of the object are hidden, and the robot must infer them for successful task execution. Since the one-shot nature of the task precludes online adaptation or retrying the task, the robot must first explore the object to infer these properties and then adjust its task execution accordingly.

4.2 Privileged Task Policy Learning

We first train a privileged task policy $\pi_{\text{task}}$ using RL with access to ground truth physical property parameters $\phi^{*}$ in simulation. We hypothesize that leveraging this privileged information reduces the need for extensive exploration and prevents convergence to suboptimal policies, leading to a higher success rate by adapting to different properties.

Task policy. During training, the task policy receives the ground truth physical property values $\phi^{*}$ as input, along with the current state observation $s_{t}$ and the goal $g$ in goal-conditioned settings, and computes actions $a_{t}$ at each timestep. We define the task policy reward, $r_{\text{task}}$, based on the task’s objectives, with positive rewards for achieving desired behaviors, such as reaching a target, and negative rewards for undesirable ones, such as violating constraints or workspace boundaries. We randomize the initial state to enable safe task execution in scenes perturbed by exploration.

4.3 Learning of Exploration and Property Estimation

While privileged access to ground truth physical property values aids task policy training, these values are unavailable in real-world settings. To address this, we simultaneously train (1) an exploration policy $\pi_{exp}$ to perform motions that are informative for estimating the physical properties $\phi$ prior to task policy execution, and (2) an online property estimator $f_{\phi}$ to infer these properties from state observations during exploration. This estimator allows immediate task execution after exploration—unlike prior works [9, 1] that require querying a simulator for offline inference.

Exploration policy. Next, we train an exploration policy $\pi_{\text{exp}}$ that computes actions $a_{t}$ to obtain observations informative for identifying the physical properties of the manipulated object, given the current states $s_{t}$ of the robot and the object through RL. We define the exploration policy reward as

[TABLE]

where $\varepsilon_{\text{estimation},j}$ represents the estimation error for the $j$-th physical property, and $\varepsilon_{\text{threshold},j}$ is the threshold for the $j$-th property. The robot receives a positive reward $r_{\text{estimation}}$ if the estimation errors of all properties are below their respective thresholds. We obtain the estimation errors $\varepsilon_{\text{estimation}}$ by computing the difference between the ground truth physical property values $\phi^{*}$ and the estimated physical property values $\hat{\phi}$ estimated by the physical property estimator $f_{\phi}$, as

[TABLE]

We adopt on-policy RL for stable training under non-stationary rewards caused by dependence on the simultaneously trained estimator. The robot receives a negative reward $r_{\text{failure}}$ for violating the workspace boundaries, ensuring feasible task execution after exploration, without manual reset.

Online physical property estimator. For online physical property estimation during exploration, we employ a Long Short-Term Memory (LSTM) [33], as temporal information is essential for capturing object dynamics (see Section C.1 for alternative approaches, e.g., a Transformer-based estimator). Between each training update, the exploration policy collects a new dataset $\mathcal{D}$ of rollouts, consisting of states $s_{t}$ and the corresponding ground truth physical property values $\phi^{\ast}$. At each update of the exploration policy, we also update the estimator by minimizing the estimation loss $\mathcal{L}_{est}$ (see Section A.2 for details) using the dataset $\mathcal{D}$. This simultaneous training of the estimator on observations from the most recent exploration policy mitigates out-of-distribution issues, a known challenge in learned estimators [19] and forward models [1, 34].

4.4 Task-Informed Exploration Reward Design

The exploration policy reward function in  Eq. 1 requires specifying the estimation error thresholds $\varepsilon_{\text{threshold},j}$. While humans can leverage intuition and physics knowledge to identify which parameters are relevant to the target task, manually specifying optimal threshold values for multiple properties is non-intuitive. Moreover, tuning these thresholds for each task is cumbersome, as the relevance of each physical property varies across tasks. To address this, we propose a method to automatically generate these estimation thresholds by modeling task performance sensitivity to estimation error in each property. Since we compute these thresholds from task performance sensitivity, these exploration rewards serve as a surrogate for the task reward. This reward design enables the robot to learn exploratory motions that lead to high task performance, prioritizing the identification of task-relevant properties—without executing the task policy during exploration policy training.

Modeling task performance sensitivity. For each physical property, we assume a uni-modal relationship between the task success rate $y$ and the estimation error $\varepsilon$, where task performance is highest when the estimation is accurate and decreases as the error increases. The rate at which task success deteriorates with increasing error reflects the sensitivity of the task to each property.

To model this relationship, we fit a parametric uni-modal function $g_{j}(\varepsilon)$ to empirical data $\mathcal{D}_{\varepsilon,j}$ for each $j$-th physical property, where $g$ represents any uni-modal function. The dataset $\mathcal{D}_{\varepsilon,j}$ consists of pairs of estimation error levels $\varepsilon$ and the corresponding task success rates $y$ achieved by the task policy. During data collection, we roll out the privileged task policy, systematically replacing the ground truth value of the $j$-th physical property with perturbed values at varying levels of estimation error. We model the sensitivity of each property individually by introducing errors into one property at a time while keeping all others unperturbed, assuming that the impact of each property on task performance is independent of the others.

Computing task-informed exploration rewards. From the fitted uni-modal function $g_{j}$, we compute a set of estimation error thresholds $\varepsilon_{\text{threshold},j}$ for each physical property $j$, such that the task success rate remains above a proportion $p$ of the maximum success rate achieved by the privileged task policy. Specifically, we solve:

[TABLE]

for each property. These estimation error thresholds define the success criteria in the exploration reward in Eq. 1. Properties that are more relevant to the task result in tighter thresholds, encouraging the exploration policy to estimate those properties with higher accuracy.

4.5 Uncertainty-Based Policy Switching

In the training phase, we define exploration success using estimation error, as in Eq. 1. However, this error is inaccessible at test time on a physical setup. To transition from exploration to task execution once the property estimates are sufficiently accurate, we 1) estimate the uncertainty of the property estimates, and 2) determine the uncertainty thresholds required for successful task execution by modeling the relationship between each property’s uncertainty and the task outcome.

Uncertainty estimator. We estimate the predictive uncertainty of the physical property estimator $f_{\phi}$ using an ensemble approach that captures both aleatoric uncertainty (from data noise) and epistemic uncertainty (from model limitations) [35]. We define $f_{\phi}$ as an ensemble of $M$ neural networks, indexed by $i$, each comprising two heads: one for the predicted mean $\hat{\phi}_{i}(s_{t})$ (denoted $\hat{\phi}_{i,t}$) and another for the predicted covariance $\hat{\Sigma}_{i}(s_{t})$ (denoted $\hat{\Sigma}_{i,t}$) [36, 37, 38]. Assuming a heteroscedastic setting, i.e. the uncertainty depends on the models’ input, where $p(\phi|s_{t})=\mathcal{N}(\hat{\phi}_{i,t},\hat{\Sigma}_{i,t})$, we train each model $i$ to minimize the negative log-likelihood. The ensemble consists of an equally weighted mixture of Gaussians which, for simplicity, we approximate with a single Gaussian distribution, such that $p(\phi|s_{t})=\mathcal{N}(\hat{\phi}_{t},\hat{\Sigma}_{t})$, where the mean and covariance are those of the mixture [37]:

[TABLE]

We use the mean $\hat{\phi}_{t}$ as the predicted physical property values, and the covariance $\hat{\Sigma}_{t}$ as the measure for predictive uncertainty, where the mean of the individual covariances captures aleatoric uncertainty, while the covariance of the mixture means captures epistemic uncertainty.

Computing uncertainty thresholds. Finally, we compute the uncertainty thresholds required for successful task execution. We roll out the exploration and task policies to collect uncertainty data labeled with task outcomes (success or failure). Then, we calculate the p-th percentile of the uncertainty values for each property from successful task trials (i.e., $q\%$ of successful trials have uncertainty values lower than this threshold). Therefore, the task policy is likely to succeed when the uncertainty values are below these thresholds. These thresholds enable the robot to assess exploration success and switch to the task policy without direct access to estimation error during testing.

5 Experiments

We evaluate our task-informed exploration approach in simulation and on a physical robot. We provide additional implementation details in Appx. B.

Tasks. We evaluate our method on three tasks: Striking a puck with unknown physical properties toward an unreachable goal; Edge Pushing a box with unknown contents to the table edge, where incorrect property estimates cause it to fall; and the classic CartPole task. Task details are in Section B.1.

Baselines. We compare our method with the following baselines: Domain Randomization (DR) [10], which randomizes physical properties for generalization; DR+Stack, which augments DR with stacked observations to capture the temporal dynamics of objects influenced by physical properties; DR+LSTM, which integrates an LSTM into PPO to model temporal dependencies, similar to dynamic randomization with LSTM [13] but using PPO instead of RDPG; DR+Stack+Est, which extends DR+Stack with one head for action computation and another for property estimation; Student [23], which trains a student policy using DAgger; RMA [39], which trains to encode observations by regressing towards the privileged latent representation of physical properties; and UP-OSI [29], which simultaneously trains a privileged policy and an online property estimator.

5.1 Does the task-informed exploration approach improve task performance?

Ours. Fig. 3 summarizes the results for the one-shot Striking task. Our method achieves a 90.1% success rate, significantly outperforming all baselines, which reach at most 40%. Our policies achieve 92.3% success in exploration (see Section C.1) and 98.7% in task, demonstrating that the exploration policy estimates properties with sufficient accuracy for successful task performance. We confirmed consistent results on the Edge Pushing task (see Section C.2).

DR baselines. DR alone learns an average motion across properties and achieves only a 25.4% success rate. In methods with temporal information, DR+Stack and DR+LSTM, exploratory pushing motions emerge but attains only 35.4% and 23.3% task success, respectively. These methods rely on delayed task rewards, provided only after task execution, to evaluate the effectiveness of the exploration, making it difficult to associate the exploratory motions with the rewards. Adding rewards for property estimation, as in DR+Stack+Est, also achieves only 34.8% due to the challenge of balancing exploration and task execution rewards within a single policy.

Privileged baselines. The methods leveraging a privileged policy (Student, UP-OSI, and RMA) achieved success rates of 23.0%, 33.7%, and 31.0%, respectively. Since the privileged task policy lacks exploratory behavior, imitating it (Student) or rolling it out—whether using online estimation (UP-OSI) or latent encodings (RMA)—leads to task failure.

5.2 Do task-informed exploration rewards capture task-relevant physical properties?

Fig. 4 presents task sensitivity to errors in each physical property parameter. In the Striking task, the CoM in the y-direction and dynamic friction show a sharper decline in success rate as error increases, compared to less sensitive parameters such as static friction. This sensitivity model indicates that the former parameters are more relevant for performing the task, resulting in tighter estimation thresholds for exploration rewards. The results, aligning with our physics-based intuition, demonstrate that our method automatically identifies task-relevant properties. Similarly, in the Edge Pushing and CartPole tasks, each property exhibits distinct relationships, reflecting varying levels of relevance to task performance. See Section C.4 for further analysis on all tasks.

5.3 How do task-informed exploration rewards and uncertainty estimates contribute?

Task-informed exploration rewards. We compare the task-informed rewards from Eq. 3 against task-agnostic rewards on the Striking task. When thresholds are uniformly too high (i.e., $\epsilon=\max_{j}\epsilon_{\text{threshold},j}$), task success drops to 47.3%, failing to accurately estimate task-relevant properties. When thresholds are uniformly too low (i.e., $\epsilon=\min_{j}\epsilon_{\text{threshold},j}$), exploration success drops to 8.0%, requiring highly accurate identification of all properties. In contrast, task-informed thresholds achieve 92.3% exploration success and 90.1% task success, supporting our hypothesis that computing the highest estimation thresholds (to trigger exploration rewards) while maintaining sufficiently low thresholds (to obtain more accurate estimates) is crucial. See Section C.3 for complete results.

Uncertainty estimates. Fig. 5 shows that both estimation errors and uncertainties decrease as exploration progresses when rolling out the exploration policy 100 times for the Striking task. Further, Fig. 5 shows that low uncertainty in task-relevant property estimates leads to task success in sim2sim transfer (i.e., , rolling out in PyBullet). The dotted red line indicates the computed uncertainty threshold for policy switching. See Section C.6 for a full analysis, including uncertainty estimates of task-irrelevant properties. These findings support using uncertainty as a proxy for estimation error in physical setups and for determining when to switch to task execution.

5.4 How does our approach perform on a physical robot?

We deploy our method on the robot, with examples provided in the video and on the website. Our approach addresses sim-to-real challenges by learning to explore and estimate task-relevant physical properties of objects, and by enabling the task policy to adjust its motion based on the estimated properties.

On the Striking task, we evaluated on pucks with three distinct friction properties: ball bearings, nylon, and aluminum. During exploration, each puck exhibits distinct motion patterns due to differences in dynamic friction, as shown in Fig. 6. The online property estimator successfully capture these differences, with dynamic friction estimates converging to 0.09 for ball bearings, 0.12 for nylon, and 0.15 for aluminum, as the uncertainty drops.

Given the estimated properties, our method achieves 8/9 successful runs across three trials per puck. These results demonstrate that the learned exploration policy and online property estimator provide accurate property estimates, and the uncertainty estimates enable a policy switching, leading to one-shot task success. We provide detailed results for the Striking and Edge Pushing tasks in Appx. D.

6 Conclusion

We present a task-informed exploration RL approach that enables robots to perform exploratory motions for identifying task-relevant properties. Our approach automatically generates task-informed exploration rewards by modeling the sensitivity of a privileged task policy to estimation errors in each property. Additionally, we introduce an uncertainty-based mechanism to transition from exploration to task execution once the property estimates achieve sufficient accuracy for a given task. We evaluate our approach on the Striking and the Edge Pushing tasks with objects of varying friction and center of mass, demonstrating significantly improved task performance over alternative methods. Our analysis shows that the exploration rewards capture the relative importance of physical properties in two manipulation tasks and the classical CartPole example. Finally, we validated our approach in a physical setup, showing that the robot successfully performs exploration followed by task execution using property and uncertainty estimates.

Limitations

Correlated properties for exploration reward design. Our method for obtaining the task-informed exploration rewards, by fitting uni-modal functions independently for each property, assumes uncorrelated effects of the properties’ estimation errors on the reduction of task performance. The correlated case would require designing and testing more complex reward functions.

Geometry and complex dynamics. In this work, we assumed a fixed shaped object. Generalizing over different shapes requires finding suitable input representations for the observed shape when learning the privileged task policy. Extending our approach to represent more complex dynamics—such as non-uniform friction, object deformation, or viscosity, poses the same representation challenge due to the high-dimensionality of the parameter space. Additionally, simulation of such dynamics is computationally expensive, making it demanding for learning policies using RL.

Sim2real model mismatch. Despite our method explicitly handling the sim2real transfer by learning to explore and estimate the physical property parameters of the object, we still observed distinct behavior in the highly dynamic striking task when the puck had a shifted center of mass, indicating a significant sim2real mismatch in the dynamic model itself. While our method can handle sim-to-real gaps due to parameter mismatch, it is unable to handle mismatches in the dynamic model itself, as it learns parameter estimators in simulation.

**Sensor modalities. ** Finally, in our experiments, we use only kinematic observations to infer property parameters. It remains unexplored how to best utilize other sensing modalities, such as force-torque, tactile, audio, and vision to identify a wider range of properties and adapt to more diverse tasks and environments.

Acknowledgments

This work is supported by the JST Moonshot R&D (Grant No. JPMJMS2031). We appreciate all reviewers for the valuable feedback.

Appendix A Method

A.1 Task-informed exploration framework

Our proposed approach utilizes the task policy $\pi_{task}$ to autonomously discover task-informed exploration rewards for training the exploration policy $\pi_{exp}$, without manual design. These rewards enable us to learn an exploration policy that generates informative motions to estimate the physical properties relevant to the task. Additionally, we compute the uncertainty of the physical property estimates and use them as a criterion to switch from the exploration to the task phase, as a surrogate of a estimation error. The task policy then uses the last estimate of the physical properties $\hat{\phi}$ from the exploration phase as input. Fig. 7 outlines the proposed method.

A.2 Training the physical property and uncertainty estimator

In the property estimator ensemble $f_{\phi}$, each neural network $i$ outputs a predicted mean $\hat{\phi}_{i,t}$ and covariance $\hat{\Sigma}_{i,t}$, given the current state $s_{t}$. To train each network, we model the conditional distribution of the true physical properties as a multivariate Gaussian parameterized by the predicted mean and covariance, i.e. $p(\phi|s_{t})=\mathcal{N}(\hat{\phi}_{i,t},\hat{\Sigma}_{i,t})$. Therefore, we use the corresponding negative log-likelihood (up to an additive constant) as the loss criterion:

[TABLE]

Appendix B Experimental setup

B.1 Tasks

Striking. The goal of this task is to strike a puck into a given target area beyond the robot’s reachable workspace. Fig. 8(a) shows the simulated environment, and Fig. 8(b) provides a schematic representation, highlighting the non-overlapping yellow workspace and green target area. This highly dynamic task exemplifies the scenario where the robot must adjust its motion based on the physical properties of the object and is unable to recover from failure once the puck becomes unreachable, motivating the need to perform exploratory motions before executing the main task. For each trial, the puck and the robot start from a fixed position, and the task is successful if the puck arrives within a $20\,$\mathrm{cm}$\times 20\,$\mathrm{cm}$$ square at the goal target coordinates $(p_{x}^{\text{target}},p_{y}^{\text{target}})$. The state $s_{t}$ consists of the puck’s position $(p_{x}^{\text{puck}},p_{y}^{\text{puck}})$ and orientation $\theta_{\text{puck}}$, and the robot’s pusher position $(p_{x}^{\text{pusher}},p_{y}^{\text{pusher}})$, and the action $a_{t}$ consists of the pusher velocity $(v_{x}^{\text{pusher}},v_{y}^{\text{pusher}})$. The properties $\phi$ contain: static friction, dynamic friction, restitution, center of mass (CoM) in the $x$ and $y$ directions, and mass.

Edge Pushing. This task uses the same setup, randomized physical properties, state, and action definitions as the Striking task. The goal is to push a box with unknown contents to a target position at the edge of the table. Inaccurate estimates of the box’s physical properties can lead to catastrophic and irreversible failure—specifically, the box falling off the table. Fig. 9(a) shows the simulated environment, and Fig. 9(b) illustrates the green target position. The task succeeds if the box arrives within $10,$\mathrm{cm}$$ of the target coordinates $(p_{x}^{\text{target}},p_{y}^{\text{target}})$ without falling off the table.

CartPole. Additionally, we evaluate our approach for modeling task sensitivity to estimation error on the classic CartPole benchmark example from Isaac Lab [40] to assess its generalizability to tasks involving a different set of physical properties. Fig. 10(a) shows the simulated environment, and Fig. 10(b) provides a schematic representation. For each trial, the pole starts from an angle sampled from $\mathcal{U}[-0.25,0.25]\,$\mathrm{rad}. The task is successful if the pole remains upright within a tolerance of $\pi/2\,$\mathrm{rad} for 5 $\mathrm{s}$. The state consists of the current cart position $p_{x}$, cart velocity $\dot{p}_{x}$, pole angle $\theta$, and pole angular velocity $\dot{\theta}$, and the action $a_{t}$ specifies the force $\tau$ applied to the cart along the $x$ direction. The properties $\phi$ contain: cart joint friction, pole joint friction, cart mass, and pole mass.

B.2 Simulation setup

We develop Striking and CartPole simulation environments using Isaac Lab [40, 41], as depicted in Fig. 8(a) and  Fig. 10(a). We use an NVIDIA GeForce RTX 2080 SUPER GPU and NVIDIA GeForce RTX 4090 throughout the experiments.

Striking. We abstract the robot as a cylindrical pusher to accelerate training, and model the puck as a fixed-size cylinder. We randomize the puck’s physical properties $\phi$, by sampling from the distributions specified in Table 2. The environment operates at a control frequency of 24 Hz, with a maximum episode duration of $H=300$ steps or 12.5 seconds. Furthermore, we utilize PyBullet [42] to evaluate the performance of our uncertainty estimation by rolling out the policy and estimator on a different physics engine from the one used for training.

Edge Pushing. This environment uses the same setup as Striking, with the puck replaced by a fixed-size box.

CartPole. We randomize the CartPole properties $\phi$, by sampling from the distributions specified in Table 2. The environment operates at a control frequency of 60 Hz, with a maximum episode duration of $H=300$ steps or 5 seconds.

B.3 Training setup

Privileged task policy. We train the privileged task policy $\pi_{task}$ using Proximal Policy Optimization (PPO) [43] with a continuous action space, and stack the last 5 states, $\{s_{t},s_{t-1},\dots,s_{t-4}\}$, to capture temporal information. For the Striking and Edge Pushing tasks, the policy function consists of a neural network architecture with four linear layers of sizes 256, 128, 64, and 2, followed by Tanh non-linearities, and the value function consists of the same architecture, but with the final linear layer (1) that outputs the state value prediction.  Table 3 provides a summary of the key hyperparameters and training settings. For the CartPole example we use the default training settings.

Task rewards. For the Striking and Edge Pushing tasks, the robot receives a positive reward of $r_{t}=+30$ for positioning the object (the puck or the box) within the target area and a penalty of $r_{t}=-15$ if either the object or the robot’s end effector violates the workspace boundaries. For the CartPole task, we define the reward function as $r_{t}=+1$ when the pole stays within $\pi/2$ rad of the upright position and the cart remains within the track limits. Violating either condition results in $r_{t}=-2$ and immediate episode termination as a failure.

Exploration policy. We also use PPO to train the exploration policy $\pi_{exp}$, consisting of an LSTM layer with 128 hidden units, followed by three fully connected layers with 256, 64, and 2 units, and Tanh non-linearities. The value function consists of the same architecture, but with the final linear layer (1) that outputs the state value prediction.  Table 3 summarizes the key hyperparameters.

Physical property and uncertainty estimator. Each network in the ensemble consists of two LSTM layers with 128 hidden units, followed by a fully connected layer that outputs $2\times\|\phi\|$ units, where $\|\phi\|$ is the dimensionality of the physical properties being estimated. The first $\|\phi\|$ units, to which we apply a Tanh nonlinearity, correspond to the predicted mean of the physical properties $\hat{\phi}_{i,t}$. The remaining $\|\phi\|$ units correspond to the predicted natural logarithm of the diagonal elements of the covariance matrix $\hat{\Sigma}_{i,t}$. We use the natural logarithm to improve numerical stability during training and assume a diagonal covariance matrix to simplify parametrisation and reduce the dimensionality of the output. To form the ensemble, we train five networks with different random initialization weights.

Noise. We add two types of noise to the observations: step-wise noise, applied independently at each timestep to simulate sensor noise, and episodic noise, which remains constant throughout the episode to simulate calibration errors.  Table 4 summarizes the distributions of the observation noise.

B.4 Hardware setup

We evaluate our approach on a physical setup using a KUKA iiwa robot arm equipped with a pusher extension, as shown in Fig. 11. We use a Vicon motion capture system to track the object’s current pose, and map the policy actions (i.e., pusher velocities) to robot joint configurations using inverse kinematics. We evaluate two hardware tasks: Striking a puck with three friction levels and three center-of-mass locations (Fig. 11), and Edge Pushing a box of eggs placed on one of two sides (Fig. 2).

Appendix C Additional results

C.1 Exploration policy training — Striking

**Simultaneous training of exploration policy and property estimator. ** Fig. 12 shows the training performance of the exploration policy $\pi_{\text{exp}}$, trained using task-informed exploration rewards and the estimator loss defined in Eq. 4, for the striking task. The results show the average over three random seeds, with shaded regions indicating the standard deviation. We demonstrate that, despite the exploration rewards being non-stationary due to their dependence on the simultaneously trained estimator, on-policy RL achieves stable training and consistently converges to an exploration success rate above 90% across seeds.

**Alternative system identification approaches. ** Our framework supports alternative system identification modules, as long as they can output both estimates and uncertainty. While the LSTM-based property estimator is sufficient for our short exploration horizon, we also tested a Transformer-based estimator, achieving similar exploration success (Fig. 13). Our method can accommodate such models for tasks requiring longer temporal dependencies during exploration or multi-modal inputs. Additionally, although we chose a learned property estimator for fast, online inference and ease of integration into an RL pipeline without additional simulation queries, investigating alternative modules (e.g., sim-in-the-loop approaches) is promising future work.

C.2 Edge Pushing baseline comparison — Edge Pushing

C.3 Task-agnostic vs. task-informed exploration rewards

C.4 Sensitivity analysis — Striking, Edge Pushing, CartPole

While our approaches can apply any uni-modal functions, we applied three different uni-modal distributions, the Gaussian, Beta, and Gamma distributions, to model the sensitivity of each property to each task.

C.5 Estimation error during exploration policy training

Fig. 15 illustrates how the Root Mean Squared Error (RMSE) of the property estimates at the end of the exploration episode evolves during training. We report mean and standard deviation (SD) across the training environments. With the task-informed exploration policy, the estimation error for relevant physical properties, such as CoM and dynamic friction, decreases, while the error for less relevant properties remains high. By the end of training, the estimation errors for all properties fall below the thresholds indicated by the dotted lines. In contrast, task-agnostic exploration policies result in high estimation errors across all physical properties. The exploration reward with high estimation thresholds for all properties (Task-agnostic (Max.)) proves non-informative. Due to the high thresholds, even large estimation errors fall below them, failing to incentivize accurate property estimation. On the other hand, setting thresholds too low for all properties (Task-agnostic (Min.)) makes it difficult to achieve the required thresholds, leading to sparse rewards. Additionally, training becomes more challenging, as it requires learning more complex behaviors to accurately estimate all properties.

C.6 Can we use uncertainty for policy switching?

Relationship between uncertainty and estimation error. First, we evaluate whether the estimated uncertainty reflects the estimation errors by rolling out the exploration policy 100 times in Isaac Lab.  Fig. 16 shows the mean and standard deviation of the estimation errors and uncertainties over the first 14 timesteps ($\approx 0.6$ seconds) of the exploration episodes. The plots demonstrate that as the exploration progresses, both the estimation errors and uncertainties decrease for task-relevant properties, such as the CoM in the y-direction and dynamic friction. This result suggests that the uncertainty estimates effectively reflect the estimation error, supporting our approach of using uncertainty as a surrogate when the estimation error is inaccessible at test time.

Relationship between uncertainty and task success. Next, we evaluate the relationship between uncertainty estimates and task success. For this evaluation, we roll out the task policy 100 times in PyBullet using estimates from the exploration policy, running for the maximum episode length. We use a simulator different from the one used for training to better assess uncertainty when there is a domain gap between training and testing, as we intend to use these uncertainties in a physical setup.  Fig. 17 presents box plots of uncertainty at the end of the exploration episode for successful and failed trials. The plots show that low uncertainty in task-relevant property estimates leads to task success, while high uncertainty results in failure. These findings suggest that uncertainty estimates during exploration predicts task outcomes and indicates when property estimates are sufficiently accurate to transition from exploration to the task phase.

Appendix D Robot experiments

D.1 Striking

Varying surface friction. We provide dynamic friction estimates and its uncertainty for all trials on pucks with different levels of friction.  Fig. 18 shows that the dynamic friction estimates consistently converge to 0.09 for ball bearings, 0.12 for nylon, and 0.15 for aluminum, with uncertainty dropping below 0.056, the uncertainty thresholds computed in Fig. 17.

Shifted center of mass. We also evaluated pucks with three varying locations of the center of mass, as shown in the second column in Fig. 11. When the CoM is in the center, the robot successfully estimates its location and completes the task, achieving 3/3 successful runs. However, for pucks with shifted CoM—either to the left or right—exploration fails due to inaccurate estimates and consistently high uncertainty, as shown in Fig. 19. We observe that, while shifts in the CoM primarily cause spinning with minimal trajectory deviation in the Isaac Lab training environment, they result in more pronounced trajectory divergence in the physical setup. This discrepancy likely arises because the simulation models the puck-table contact as a point contact, whereas real-world contact involves full surface contact, altering the friction distribution. Bridging this sim-to-real gap by modeling such contact in simulation or utilizing a small amount of real-world data to learn unmodeled dynamics is a promising direction for future work.

D.2 Edge Pushing

Shifted center of mass. For the Edge Pushing task, we tested the robot with two different center-of-mass (CoM) locations by placing the eggs on either side of the egg cartons. When we shifted the CoM to the left, the robot estimated its location 5.8cm off-center to the left, achieving 5/5 successful runs. When we shifted the CoM to the right, the robot estimated its location 5.8cm off-center to the right, achieving 3/5 successful runs. Fig. 20 shows that the robot’s CoM estimates converge to the correct values, with uncertainty dropping below 0.018—the computed threshold.