TReF-6: Inferring Task-Relevant Frames from a Single Demonstration for One-Shot Skill Generalization

TL;DR

TReF-6 introduces a method to infer a task-relevant spatial frame from a single demonstration, enabling robots to generalize skills across different scenes by combining geometric inference with semantic grounding.

Contribution

The paper presents a novel approach to infer a simplified 6DoF task frame from a single trajectory, integrating geometric and semantic information for improved one-shot skill generalization.

Findings

Robust in simulation despite trajectory noise

Supports real-world manipulation tasks with diverse objects

Enables one-shot imitation learning that preserves task intent

Abstract

Robots often struggle to generalize from a single demonstration due to the lack of a transferable and interpretable spatial representation. In this work, we introduce TReF-6, a method that infers a simplified, abstracted 6DoF Task-Relevant Frame from a single trajectory. Our approach identifies an influence point purely from the trajectory geometry to define the origin for a local frame, which serves as a reference for parameterizing a Dynamic Movement Primitive (DMP). This influence point captures the task's spatial structure, extending the standard DMP formulation beyond start-goal imitation. The inferred frame is semantically grounded via a vision-language model and localized in novel scenes by Grounded-SAM, enabling functionally consistent skill generalization. We validate TReF-6 in simulation and demonstrate robustness to trajectory noise. We further deploy an end-to-end pipeline on real-world manipulation tasks, showing that TReF-6 supports one-shot imitation learning that preserves task intent across diverse object configurations.

Figures

Tables

Table 1. Table 1: Comparison of Spatial influence Inference Methods under Different Noise Levels over 50 Random Seeds in 3D. Report in MEDE: Mean ± \pm Std

Method	0% Noise	10% Noise	30% Noise	50% Noise	80% Noise
Directional Consistency Score	$0.0263\pm 0.0392$	$0.0560\pm 0.0276$	$0.1688\pm 0.0831$	$0.3044\pm 0.1808$	$0.5814\pm 0.3722$
Directional Consistency Score (Random Initialization)	$0.1884\pm 0.8217$	$0.2794\pm 1.0735$	$0.3076\pm 0.6416$	$0.4828\pm 1.1278$	$0.7945\pm 1.2573$
Inverse Dynamics Triangulation	$0.0127\pm 0.0157$	$0.1111\pm 0.0818$	$0.4009\pm 0.2589$	$0.6540\pm 0.4020$	$0.9474\pm 0.5689$
Cosine Similarity	$1.8483\pm 2.2455$	$1.8892\pm 2.3067$	$1.4667\pm 2.0082$	$1.1981\pm 1.7585$	$1.7815\pm 2.0691$
Quadratic Residual	$0.3311\pm 0.2217$	$0.3397\pm 0.2225$	$0.4147\pm 0.2441$	$0.5191\pm 0.2868$	$0.7482\pm 0.4037$

Table 2. Table 2: Comparison of Spatial influence Inference Methods under Different Noise Levels over 50 Random Seeds in 2D. Report in MEDE: Mean ± \pm Std

Method	0% Noise	10% Noise	30% Noise	50% Noise	80% Noise
Directional Consistency Score	$0.0391\pm 0.0289$	$0.0584\pm 0.0344$	$0.1473\pm 0.1903$	$0.2174\pm 0.1725$	$0.4965\pm 1.0663$
Inverse Dynamics Triangulation	$0.0128\pm 0.0169$	$0.0739\pm 0.0642$	$0.2817\pm 0.2396$	$0.4356\pm 0.3194$	$0.6179\pm 0.4023$
Cosine Similarity	$1.7646\pm 2.0060$	$1.5219\pm 2.0972$	$1.4158\pm 2.2923$	$1.6448\pm 2.2317$	$1.5106\pm 1.8028$
Quadratic Residual	$0.2275\pm 0.1601$	$0.2432\pm 0.1475$	$0.2753\pm 0.1612$	$0.3218\pm 0.1919$	$0.4468\pm 0.2454$

Table 3. Table 3: Performance of Sequential influence Inference under Varying Noise Levels and Switch Ratios. Reported in MEDE (Mean Euclidean Distance Error) over 50 seeds.

Noise Level	Switch Ratio (p1 / p2)	MEDE (p1)	MEDE (p2)	Overall Mean Error
0%	30 / 70	$0.0551\pm 0.0404$	$0.0541\pm 0.0696$	$0.0546\pm 0.0359$
	50 / 50	$0.0622\pm 0.0886$	$0.0800\pm 0.1255$	$0.0711\pm 0.0760$
	70 / 30	$0.0720\pm 0.0654$	$0.4450\pm 0.5543$	$0.2585\pm 0.2733$
10%	30 / 70	$0.0936\pm 0.0424$	$0.0949\pm 0.0574$	$0.0943\pm 0.0362$
	50 / 50	$0.1131\pm 0.1572$	$0.1337\pm 0.1504$	$0.1234\pm 0.1041$
	70 / 30	$0.0888\pm 0.0484$	$0.4822\pm 0.4958$	$0.2855\pm 0.2459$
30%	30 / 70	$0.2360\pm 0.1219$	$0.2583\pm 0.1402$	$0.2471\pm 0.1069$
	50 / 50	$0.2378\pm 0.1643$	$0.2799\pm 0.2037$	$0.2588\pm 0.1331$
	70 / 30	$0.1955\pm 0.1029$	$0.6322\pm 0.4831$	$0.4138\pm 0.2442$

Table 4. Table 4: Success Rates of Baseline DMP vs. Our Method for Peg-in-hole Dropping Task. Reported over 15 variations.

Metric	Baseline DMP	Ours (Gap)
Overall Success Rate	$20.0\%$	$53.3\%$ ($+33.3\%$)
Grasping Success Rate	$80.0\%$	$86.7\%$ ($+6.7\%$)
Execution Success (Given Grasp)	$25.0\%$	$61.5\%$ ($+36.5\%$)
Success Rate by Object:
Red Cube	$1/5$	$\mathrm{𝟒}/\mathrm{𝟓}$ ($+\mathrm{𝟑}$)
Blue Cap	$1/5$	$\mathrm{𝟐}/\mathrm{𝟓}$ ($+\mathrm{𝟏}$)
Green Ring	$1/5$	$\mathrm{𝟐}/\mathrm{𝟓}$ ($+\mathrm{𝟏}$)
Success Rate by Rod:
Red Rod	$\mathrm{𝟑}/\mathrm{𝟑}$	$\mathrm{𝟑}/\mathrm{𝟑}$ ($+0$)
Blue Rod	$0/3$	$\mathrm{𝟏}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)
Yellow Rod	$0/3$	$\mathrm{𝟑}/\mathrm{𝟑}$ ($+\mathrm{𝟑}$)
Purple Rod	$0/3$	$0/3$ ($+0$)
Green Rod	$0/3$	$\mathrm{𝟏}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)

Table 5. Table 5: Success Rates of Baseline DMP vs. Our Method for Cabinet Door Opening Task. Reported over 12 variations.

Metric	Baseline DMP	Ours (Gap)
Overall Success Rate	$8.3\%$	$66.7\%$ ($+58.3\%$)
Grasping Success Rate	$16.7\%$	$75.0\%$ ($+58.3\%$)
Execution Success (Given Grasp)	$50.0\%$	$88.9\%$ ($+38.9\%$)
Success Rate by Orientation:
Frontal	$1/4$	$\mathrm{𝟑}/\mathrm{𝟒}$ ($+\mathrm{𝟐}$)
Left Angled	$0/4$	$\mathrm{𝟑}/\mathrm{𝟒}$ ($+\mathrm{𝟑}$)
Right Angled	$0/4$	$\mathrm{𝟐}/\mathrm{𝟒}$ ( $+\mathrm{𝟐}$)
Success Rate by Hinge Position:
Hinge at Right	$1/3$	$\mathrm{𝟑}/\mathrm{𝟑}$ ($+\mathrm{𝟐}$)
Hinge at Left	$0/3$	$\mathrm{𝟐}/\mathrm{𝟑}$ ($+\mathrm{𝟐}$)
Hinge at Bottom	$0/3$	$\mathrm{𝟑}/\mathrm{𝟑}$ ($+\mathrm{𝟑}$)
Hinge at Top	$0/3$	$0/3$ ($+0$)

Table 6. Table 6: Success Rates of Baseline DMP vs. Our Method for Surface Wiping. Reported over 9 variations.

Metric	Baseline DMP	Ours (Gap)
Overall Success Rate	$33.3\%$	$66.7\%$ ($+33.3\%$)
Grasping Success Rate	$100.0\%$	$77.8\%$ ($-22.2\%$)
Execution Success (Given Grasp)	$33.3\%$	$85.7\%$ ($+52.4\%$)
Success Rate by Stain Color:
Blue	$1/3$	$\mathrm{𝟐}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)
Black	$1/3$	$\mathrm{𝟐}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)
Red	$1/3$	$\mathrm{𝟐}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)
Success Rate by Surface Tilted Angle:
${45}^{\circ }$	$\mathrm{𝟑}/\mathrm{𝟑}$	$2/3$ ($-1$)
${90}^{\circ }$	$0/3$	$\mathrm{𝟑}/\mathrm{𝟑}$ ($+\mathrm{𝟑}$)
${30}^{\circ }$	$0/3$	$\mathrm{𝟏}/\mathrm{𝟑}$ ($+\mathrm{𝟏}$)

Equations

$$\mathcal{S}(p)=-\frac{1}{T}\sum _{t=1}^T\left|\left|\frac{p-x_t}{||p-x_t||+ϵ}-{\ddot{x}}_t\right|\right|$$

$${\left|\left|\frac{p-x_t}{||p-x_t||+ϵ}-{\ddot{x}}_t\right|\right|}^2$$

$$⟹\frac{\partial }{\partial \theta }{\left|\left|\frac{p-x_t}{||p-x_t||+ϵ}-{\ddot{x}}_t\right|\right|}^2$$

$$\Delta x_t$$

$${\text{Score}}_{\text{cos}}(p)=\frac{1}{T}\sum _{t=1}^T\left|\frac{(p-x_t{)}^{\top }{\ddot{x}}_t}{||(p-x_t)||\cdot ||{\ddot{x}}_t||+ϵ}\right|$$

$${\text{Score}}_{\text{quad}}(p)=\frac{1}{T}\sum _{t=1}^T||(p-x_t)-{\ddot{x}}_t|{|}^2$$

$$\underset{p\in {\mathbb{R}}^3}{\min }\sum _{t=1}^T{\left|(I-{\hat{a}}_t{\hat{a}}_t^{\top })(x_t-p)\right|}^2$$

$${\text{DMP}}_{\text{pos}}(x_0^{\text{new}})={\left\{y_t|\begin{array}{rl}\tau {\ddot{y}}_t& ={\alpha }_z\left({\beta }_z(g^{\text{deploy}}-y_t)-\tau {\dot{y}}_t\right)+f(s_t),\\ \tau {\dot{y}}_t& ={\ddot{y}}_t,\tau {\dot{s}}_t=-{\alpha }_s{s}_t,\\ f(s)& =\frac{{\sum }_i{\psi }_i(s)w_i}{{\sum }_i{\psi }_i(s)}s{g}^{\text{deploy}},g^{\text{deploy}}=\Delta x_T\cdot \frac{\Vert p^{\ast }-x_0^{\text{new}}\Vert }{\Vert p^{\ast }-x_0^{\text{demo}}\Vert }\end{array}\right\}}_{t=1}^T$$

$${\text{DMP}}_{\text{quat}}(q_0^{\text{new}})={\left\{y_t|\begin{array}{rl}\tau {\ddot{y}}_t& ={\alpha }_z\left({\beta }_z(\Delta q_T-y_t)-\tau {\dot{y}}_t\right)+f(s_t),\\ \tau {\dot{y}}_t& ={\ddot{y}}_t,\tau {\dot{s}}_t=-{\alpha }_s{s}_t,\\ f(s)& =\frac{{\sum }_i{\psi }_i(s)w_i}{{\sum }_i{\psi }_i(s)}s\Delta q_T\end{array}\right\}}_{t=1}^T$$

Full text

Keywords: Spatial Reference Frames, One-Shot Imitation Learning, Dynamic Movement Primitives

1 Introduction

Robots are increasingly expected to operate in dynamic, human-centered environments, whether assisting in homes [1], collaborating in warehouses [2], or supporting kitchen workflows [3]. These settings demand generalizable behavior: adapting motion to unseen objects, adjusting to new object placements and orientations, and aligning with different surface orientations. While humans perform such adaptations effortlessly, robots struggle to generalize skills from limited demonstrations.

Since training data can never fully prevent out-of-distribution (OOD) scenarios, researchers increasingly turn to extract structural representations, such as rigid-body poses [4] or keypoints [5, 6, 7], which provide a more stable foundation for generalization in manipulation tasks. Recent work has developed generalizable techniques for generalizing a task in terms of goal poses [8, 9, 10], sub-goals [11], or contact interactions [12]. There has been less work, however, on generalizing the spatial constraints encoded in the trajectory itself, such as the curvature of opening the door indicating a hinge constraint besides just the handle position. In practice, the shape of a human demonstration reflects more than just start and end points. It encodes implicit constraints, such as obstacle avoidance [13], mechanical limitations (e.g. hinge constraints) [14], or ergonomic preferences [15]. For any two points in space, infinitely many paths exist, yet demonstrators tend to follow specific, repeatable curves. Prior works in [16] have shown that humans opt for these repeatable trajectories due to their similarities in more condensed, latent geometric structure. We believe that inferring these latent structures could be informative in generalizing trajectories to unseen objective configurations. Furthermore, we expect that they may correspond to semantic features that are detectable by vision-language models (VLMs) [17]. This motivates our central research question: Can we use a single demonstration to infer a task-relevant spatial reference frame that is (1) semantically identifiable (anchors to scene features) and (2) functionally meaningful (preserves constraints), enabling generalization across object poses and configurations?

In this work, we propose TReF-6 (Figure 1), a framework that infers a task-relevant, 6DoF reference frame from a single demonstration that enables simple dynamical controllers such as DMPs to generalize their motion robustly. Our contributions include:

Formalizing the problem of inferring a task-relevant frame as an optimization over its geometric consistency to the trajectory dynamics; 2. 2.

An efficient optimization algorithm that is robust to trajectory noise and does not rely on object priors, human labels, or dense annotations; and 3. 3.

Simulated and physical robot evaluations that demonstrate our method’s ability to generalize to new spatial variations of demonstrated tasks.

2 Related Work

Dynamic Movement Primitives (DMPs) encode a demonstrated trajectory as a stable dynamical system [18, 19] that converges toward a goal. DMPs can generalize the shape of a demonstrated motion to target a new goal pose, yet are functionally agnostic to the surrounding environment unless manually augmented with additional force-like terms that account for external influences [20] or goal and sub-goal related task frames [21]. Without access to an appropriate task-specific reference, DMPs are trained over just a single trajectory, so by definition they “overfit” to the environmental configuration and constraints that were present during the initial demonstration.

Data-driven approaches such as TP-GMMs [22] and their extensions [23] involve conditioning a demonstrated trajectory on external task parameters. These approaches improve adaptation across different object poses, placements, and orientations, but require extensive training data (including synthetic data) to generalize effectively [22, 24], or assume that relevant task-relevant frames, such as those aligned with object or interaction axes, are provided or can be easily extracted [25].

Affordance learning focuses on the actionable possibilities in an environment, such as regions on an object that can be grasped, pushed, or opened [26], capturing the physical interactions that enable task completion. Recent efforts in this field have focused on visually identifying object surfaces that support affordances based on robot-collected data [27] or human demonstration videos [28, 29]. Others use depth data to address occlusion and collision constraints in articulated objects [30]. While these approaches have shown promising results, they involve training visual backbones with target object types or multi-view RGB inputs. In real-world scenarios where a robot must interact with unseen novel objects, especially under varying camera viewpoints with crowded objects and random occlusions, such reliance becomes a bottleneck in terms of data requirements and system robustness.

One-shot imitation learning targets this challenge of adapting to novel task constraints or objects. This may involve grounding the demonstration using external context—language descriptions or videos of related tasks to make sense of the action in a broader setting [31]. Other methods focus on low-level structure, aligning object parts or motion trajectories across different scenes to find transferable patterns [32, 8, 33] or with rich contact information [12]. Recent work also looks for structure that is consistent across different environments, for example, regions in 3D space that consistently guide behavior across tasks [34], or prior demonstrations retrieved based on graph-based similarity metrics [35]. These approaches enable a robot to quickly generalize its task knowledge to novel constraints and objects, and reinforce the importance of a reasoning module that correctly and efficiently identifies task-relevant features for one-shot learning [36].

Although efficient at generalization, one-shot methods still require a large amount of data at training to acquire the ability to learn-to-learn and generalize from a single new example. Compared to end-to-end one-shot policy learning approaches, DMPs offer advantages in stability, data efficiency, and ease of adaptation once a transferable frame is available. This motivates our focus on building a minimal-assumption method to automatically extract a task-relevant spatial frame, enabling classical DMPs to generalize motion trajectories without the need for large-scale task distributions, external labels, or object CAD models.

3 Methodology

We introduce TReF-6111GitHub repository: https://github.com/iqr-lab/tref-6, a trajectory-based framework for inferring a task-relevant local frame from a single demonstration for skill generalization. Unlike prior work that relies on large-scale training data or assumes external knowledge about object geometry, our method leverages a novel optimization-based formulation to infer latent influence point from motion dynamics, which is general-purpose and operates on a single trajectory. The end-to-end pipeline consists of three stages:

Influence Point Inference: Optimize a directional consistency score to identify a 3D spatial point that best captures the trajectory’s dynamics. 2. 2.

Semantic Grounding: Refine the point by aligning it with a semantically relevant and spatially aligned visual feature identified by a VLM, and extract a local frame based on surface normals and interaction direction. 3. 3.

DMP Reparameterization: Transform the trajectory into the inferred local frame, fit DMPs, and reuse them in novel scenes by extracting a new frame.

3.1 Influence Point Inference: Defining a Directional Consistency Score

We formulate the frame inference problem as identifying a coordinate transformation informed by the trajectory. We hypothesize that the demonstrated motion is shaped by a dominant spatial constraint like rotation around a hinge (axis constraint), movement along a shelf (plane constraint), or alignment toward a socket (point constraint). Rather than designing separate models for each type, we infer a single 6-DoF frame centered at a latent point $p\in\mathbb{R}^{3}$ that reflects the underlying structure influencing the motion, serving as a general-purpose reference for motion generalization.

Given a single trajectory $\{x_{t}\}^{T}_{t=1}\in\mathbb{R}^{3}$, we hypothesize that the motion can be captured by a position-only, latent influence pointing from $x_{t}$ to a fixed point $p\in\mathbb{R}^{3}$. We optimize $p$ for both temporal consistency and directional agreement, whether the observed acceleration aligns consistently toward a candidate point, inspired by prior work in shared-control robotics to disambiguate user intent [37]. Specifically, we define the directional consistency score, $\mathcal{S}:\mathbb{R}^{3}\rightarrow\mathbb{R}$, which compares the predicted direction from $x_{t}$ to $p$ with the acceleration, $\ddot{x}_{t}$. We define $\mathcal{S}(p)$ as:

[TABLE]

where $\epsilon>0$ is a small positive constant. A higher $\mathcal{S}(p)$ value indicates stronger consistency between the trajectory’s underlying dynamics and the candidate point $p$. A 2D example of the score landscape is shown in Appendix Figure 9, where regions around the ground-truth candidate point of influence have higher $\mathcal{S}(p)$ values. By relying on the second derivative, the score inherently reflects the temporal dynamics of the trajectory. Since direction difference is normalized, $\mathcal{S}$ provides an objective that is robust to variations in force magnitude.

3.2 Influence Point Inference: Optimizing for Directional Consistency

We can then estimate the latent intent point by solving for $p^{*}=\arg\max_{p\in\mathbb{R}^{3}}\mathcal{S}(p)$. Due to vector normalization and temporal error aggregation, we find that the score landscape exhibits many local optima and extensive flat regions, especially under noise in acceleration direction, as illustrated in Appendix Figure 5, making the result highly sensitive to initialization. Empirically, we find that choosing an initialization point with $(1)$ sharp gradients and $(2)$ proximity to the ground truth, is essential for successful optimization - an insight aligned with prior works in optimization [38, 39]. Formally, if we denote the small angular deviation between $\frac{p-x_{t}}{||p-x_{t}||}$ as $\theta$, we can apply the law of cosines to approximate the squared residual and its partial derivative as follows:

[TABLE]

where the change in residual magnitude due to the deviations in $\theta$ is proportional to $||\ddot{x}_{t}||$. We initialize the optimization near these regions by computing $p_{0}\sim\frac{1}{k}\sum_{\tau\in\mathcal{I}}x_{j}+\mathcal{N}(0,\sigma^{2}\mathbf{I})$, where $\mathcal{I}$ is the top$-k$ timesteps with the largest $||\ddot{x}_{t}||$ magnitudes. This initialization places the optimization near points of sharper gradients. We empirically show in Section 4.1 that this approach significantly improves convergence and solution quality. Due to the non-smoothness of normalization near $p\approx x_{t}$, we approximate gradients using central finite differences and use Adam [40] to smooth fluctuations in the gradient and accelerate convergence. Algorithm 1 details the entire optimization algorithm.

3.3 Semantic Grounding

We then use the optimized influence point $p^{*}$ as the origin of the task-relevant 6DoF Frame. To ensure the frame is semantically meaningful and transferable across scenes, we refine this point by aligning it with visual features identified by GPT-4o [41]. Specifically, we implement a two-phase querying process. In the first phase, we prompt the model to generate a high-level task label based on the initial state RGB image overlaid with the demonstration trajectory. In the second phase, using the predicted task label, we query GPT-4o again with the same RGB image now overlaid with the inferred influence point to identify the visual features associated with both the influence point and the interaction point (where the robot first begins its interact with the environment). We provide the full prompts in Appendix 9.8. The model returns a natural language description of the features, which we use to guide segmentation via Grounded-SAM and refine the point. Once the refined point is established, we compute the surface normal at that location to define the $z$-axis of the frame. We then define the $yz$-plane using the $z$-axis and the vector from the refined point to the interaction point. The $x$-axis is then computed as the unit vector orthogonal to this plane. Finally, the $y$-axis is obtained as the cross product of the $z$- and $x$-axes. This captures both local geometry (via the surface normal) and task-relevant directionality (via the interaction point), as shown in Figure 2.

3.4 DMP Reparameterization

To enable environment-adaptive generalization, we apply the standard DMP formulation in a task-relevant frame centered at the refined influence point $p^{*}$. Rather than modifying the DMP dynamics, we reparameterize the demonstrated trajectory by transforming it from the robot base frame, referred to as the world frame, into the task-relevant frame. Given a demonstration of length $T$, the Cartesian position $x_{t}$ and quaternion orientation $q_{t}$ at time $t$ of the trajectory are transformed from the world frame into the inferred task-relevant frame. We then compute the relative motion from the starting pose in this local frame:

[TABLE]

where $x_{0},q_{0}$ denote the initial end-effector pose relative to the influence point, and $\otimes$ is quaternion multiplication. This initial pose corresponds to the location of the tool at the beginning of task execution (e.g., where a brush first makes contact with the robot). We then fit Cartesian and quaternion DMPs over $\Delta x_{t}$ and $\Delta q_{t}$, respectively. At deployment, a new influence point $p^{*}$ and associated task-relevant frame are inferred, and the new starting pose $(x^{\text{new}}_{0},q^{\text{new}}_{0})$ is computed accordingly. The DMPs then roll out relative motions with respect to $(x^{\text{new}}_{0},q^{\text{new}}_{0})$. The definition of the roll out functions $\text{DMP}_{\text{pos}}(x^{\text{new}}_{0})$ and $\text{DMP}_{\text{quat}}(q^{\text{new}}_{0})$ is deferred to Appendix 9.6. The generated trajectory is finally mapped back into the world frame using the inverse transformation.

4 Experiment

To evaluate whether our proposed method could infer a task-relevant spatial reference frame that is (1) semantically identifiable and (2) functionally meaningful from a single demonstration, we designed controlled 3D simulations with known influence point to assess inference precision under varying levels of directional and magnitude noise in acceleration, using Mean Euclidean Distance Error (MEDE) between the inferred and ground-truth influence point as the metric. In real-world tasks, we assess whether the inferred frame can be semantically grounded and enable one-shot skill generation. While most sophisticated baselines such as affordance-based imitation or goal-conditioned policies exist, they typically rely on object CAD models [8], extensive training [34], or rich contact information [12], which are unavailable in our setting. We thus benchmark against privileged DMPs, which have access to additional information such as object positions or operate in environments that mimic the demonstration setup, defined in Section 4.2, as the strongest feasible baseline. Improvement over the baseline, measured by task success under an OOD environment, indicates that the inferred frame captures structure critical for generalizing the demonstrated behavior.

4.1 Simulated Environment

In real-world settings, demonstrations are inherently noisy, and our method relies on the second-order trajectory, which amplifies noise. Therefore, to test the robustness of our method with noisy demonstrations, we design a 3D simulation where the ground-truth influence point and motion dynamics are fully controllable. A point mass, simulating a robot end-effector, starts at the origin with a random velocity sampled from $[-0.5,0.5]^{3}$, and is influenced by a randomly placed, ground-truth influence point $\hat{p}$ within a bounded region $[-5,5]^{3}$. A directionally noisy force points from the particle to the influence, scaled by a coefficient $\alpha_{t}\sim\mathcal{U}(0,||\hat{p}-x_{t}||)$, mimicking diminishing attraction. We add a constant Gaussian noise to both direction and magnitude of control. The state evolves throughout a locally linearized dynamics for $100$ steps.

We evaluated our method with and without random initialization along with three other scoring objectives and one inverse dynamics baseline for inferring the spatial influence point $p$. All methods were tested under the same simulated conditions. We reported the MEDE between the predicted and ground-truth influence points over $50$ randomized seeds. These baselines were selected to represent diverse but plausible strategies for trajectory-based influence point inference without requiring access to ground-truth labels, including a physically grounded method (inverse dynamics triangulation), optimization over residual scores (quadratic residual score), and directional alignment (cosine similarity score).

Directional Consistency Score (ours) achieved the best overall performance across all noise levels, maintaining low error and variance even as the force signal became increasingly corrupted, consistently outperforming random initialization baseline across all noise settings, with an average of $55.0\%$ reduction in error and an average of $86.8\%$ reduction in variance. Appendix Figure 7 illustrates a representative worst-case outcome under $50\%$ noise using random initialization. In contrast, across all baselines, we observed distinct failure modes under extreme noisy conditions. Inverse Dynamics Triangulation was highly sensitive to perturbations in the direction of the force vector, as it relied on accurate ray intersections from $\ddot{x}_{t}$ orientations. Cosine Similarity Score suffered in 3D settings due to vanishing gradients, leading to poor convergence and high variance. Quadratic Residual Score broke down when the force magnitudes were irregular or noisy—as was the case in our simulations where magnitudes were modulated stochastically. In contrast, our proposed Directional Consistency Score remained robust by comparing normalized force predictions directly with observed accelerations, achieving the best performance across varying noise levels.

Figure 3 summarizes these results. Mathematical formulations of Cosine Similarity Score, Quadratic Residual Score, and Inverse Dynamics Triangulation are summarized in Appendix 9.2. More detailed analysis and discussion of each approach as well as a comprehensive simulation experiment results are summarized in Appendix 9.4.

4.2 Real-World Experiment

Our real-world experiments aim to validate that TReF-6 enables functionally meaningful one-shot skill generalization, even when paired with simple downstream controllers such as DMPs. We evaluate its effectiveness across three real-world manipulation tasks on a 7-DoF Kinova Gen3 robot where trajectory shape and alignment are critical: (1) peg-in-hole dropping, (2) cabinet door opening, and (3) surface wiping. Each task includes one demonstration and multiple OOD variants for evaluation. The peg-in-hole task tests for semantic transferability and precision, with variations in object shape and color, as well as rod color and height. The door-opening task examines generalization across spatial positioning and rotations, varying cabinet position, hinge placement, and cabinet orientation. The wiping task evaluates adaptability to surface tilt and the ability to maintain continuous surface contact without excessive force, with changes in stain appearance and the flatness of the board.

To isolate the contribution of our method, we adapt DMPs as the shared motion controller and compare executions with and without our inferred local reference frame. Since vanilla DMPs lack semantic grounding, we provide a privileged setup for baseline DMPs: objects and rods in the peg-in-hole task are placed in the original demonstration locations; for door opening, the handle position is explicitly specified; and for wiping, the whiteboard brush and tilted surface are arranged to match the demonstration.

(1) Peg-in-hole Dropping: In this task, the robot is provided with a single demonstration, illustrated in Appendix Figure 6, and both methods successfully reproduce the demonstrated behavior in the original setting. We intentionally designed the task to be challenging and tightly constrained: as illustrated in Appendix Figure 11, slight misalignments between the object and the rod lead to failure. Our method maintains a high success rate under variations, as shown in Figure 4. While the baseline achieves a similar success rate in the pickup phase due to its privileged setup, its performance in completing the full task, assuming the object has been picked up, is significantly lower than ours.

(2) Cabinet Door Opening: The robot is provided with a single demonstration, illustrated in Appendix Figure 6, where it opens a cabinet door directly in front of it, left to right. Successful task execution requires rotating the handle around a hinge axis - a simple linear pulling would result in failure, as it would drag the entire cabinet without opening the door. Both the baseline and our method reproduce the demonstrated behavior in the original setting. However, DMP fails to adapt its motion to the new hinge geometry or cabinet orientation. In contrast, our method generalizes the motion successfully across multiple variants with an overall success rate of $66.7\%$ as shown in Figure 4.

(3) Surface Wiping: In this task, the robot is provided with a single demonstration, illustrated in Appendix Figure 6. A successful execution requires wiping out the ink on the whiteboard while maintaining contact with the surface, without deforming it to the point of touching the threshold placed behind. Both the baseline and our method reproduce the demonstrated behavior under the original setup. As shown in Appendix Table 6, the baseline performs reliably in this setting and under stain color changes. However, its motion does not adapt to different surface orientations. In contrast, our method achieves a much higher overall success rate of $66.7\%$ across tilted surface variants. When evaluating only trails where the brush was successfully grasped, our method achieves $85.7\%$ success, outperforming the baseline by $52.4\%$.

5 Discussion and Analysis

Overall, our real-world evaluations show that TReF-6 enables functionally meaningful generalization across spatial variations with only a single demonstration by providing a semantically transferable task-relevant frame. We now highlight the key takeaways of our results.

Task-Specific Strength.

Across all three tasks, TReF-6 exhibits consistent improvements over baseline DMPs by leveraging task-specific spatial cues. In the peg-in-hole dropping task, it adapts to changes in rod height and color, successfully adjusting the hook trajectory, while baseline DMP fail to hook the rod securely before release. In the door opening task, TReF-6 correctly infers the arc direction and hinge orientation, unlike baseline which follows a fixed left-to-right arc. In the surface wiping task, TReF-6 realigns motion to maintain contact to different tilted surfaces, which the baseline cannot handle.

Partial Success Despite Failures.

Even when task execution is not fully successful, TReF-6 often produces structurally meaningful motions. Failures in peg-in-hole dropping tasks arise primarily from color misclassifications (e.g., identifying red or blue rods as a purple rod), yet the behavior remains robust enough to complete partial goals (e.g. reaching the wrong rod). Most failures in door opening task arise from unreachable grasp pose, particularly when the handle is located near the table surface, leading the robot to hit kinematic limits. When grasping fails, the resulting motion still preserves arc structure around the hinge, reflecting meaningful generalization without success in the full task.

Performance Correlates to Perception Accuracy.

The quality of generalization is tightly coupled with the reliability of depth perception. One example frame extraction failure case of dropping on the green rod that caused by depth noise is visualized in Appendix Figure 12. Failure in the mirrored variant of door opening task is also attributed to poor depth perception when the door surface is nearly orthogonal to the camera, as shown in Appendix Figure 13. The lowest performance on surface wiping occurs at the $30^{\circ}$ tilt, as shown in Appendix Figure 10. We hypothesize that shallow tilts, oblique angles, and small cross-section area of object lead to noisier or incomplete depth maps, leading to frame extraction failure. However, these perception challenges, such as segmentation errors or depth noise, may be mitigated in the future through advances in 3D perception and improvements to grounding models like Grounded-SAM, especially considering that we did not fine-tune the model for our tasks.

6 Conclusion

We presented a novel framework to augment DMPs’ generalization capabilities: inferring a task-relevant spatial reference frame from a single demonstration that is both semantically identifiable and functionally meaningful. Unlike prior approaches that rely on object priors, multiple demonstrations, or predefined frames, our method extracts a full 6-DoF reference frame directly from the geometry of the demonstrated trajectory. By anchoring this inferred frame to semantic scene features, we enable skill generalization across diverse object poses and spatial configurations. Our physical robot experiments validate that our approach preserves the structural constraints of the task while adapting to challenging environment variations. These results highlight the promise of geometry-driven, semantically grounded reference frames as a foundation for scalable imitation learning. Moreover, our framework is agnostic to the type of downstream motion primitives; future works therefore could explore beyond DMPs such as probabilistic movement primitives and kernel-based models. Future works can also extend our approach with richer object representations to enable grasping strategies across diverse geometries.

7 Limitations

Since TReF-6 relies on Grounded-SAM for localization, semantic mis-segmentation occasionally occurs. Since segmentation is not fine-tuned and operates out-of-the-box, addressing such external semantic ambiguities remains future work.

TReF-6 extracts a single influence point to define a task-relevant 6DoF frame. While more complex spatial constraints could theoretically require richer representations, our experiments demonstrate that a single frame abstraction is sufficient to generalize across diverse atomic tasks in practice.

TReF-6 focuses on motion generation after an object has been acquired, modeling the tool or object as a single representative point. It does not address grasp planning or complex contact interactions, and thus assumes that a feasible grasp has already been achieved prior to motion execution. Despite this simplification, experiments demonstrate that TReF-6 achieved near-perfect success rates in task execution once the object is properly grasped. Future work will focus on extending the framework to integrate grasp planning, enabling the system to jointly reason about how to grasp and how to perform the task within the same spatial frame.

8 Acknowledgments

The authors would like to thank members of the Yale Inquisitive Robotics Lab and Qian Wang for their feedback and contributions.

9 Appendix

9.1 Loss Landscape

9.2 Definition of Baselines

A. Cosine Similarity Score

We define the cosine similarity score as:

[TABLE]

While still non-convex, cosine similarity score is effective in 2D due to a constrained directional space and sharp gradient alignment. Its performance degrades in 3D settings where directional ambiguity, vanishing gradients, and the lack of magnitude information significantly impair its ability to infer correct influence points.

B. Quadratic Residual Score

We define the quadratic residual score as:

[TABLE]

Despite this objective being convex and easy to optimize, it assumes the magnitude of the observed acceleration matches that of the position-based prediction, which is often violated in realistic demonstrations and results in low accuracy even in simulated environment.

C. Inverse Dynamics Triangulation

Under the Newtonian assumption $\vec{f}_{t}=m\cdot\vec{a}_{t}$, the inverse dynamics triangulation method interpret the direction of the acceleration vector at each timestep as a ray pointing toward the latent affordance point. Then, the triangulation process estimates the point in space that minimizes the orthogonal distance to all such rays:

[TABLE]

where $\hat{a}_{t}$ is the unit acceleration direction at timestep $t$, and $(I-\hat{a}_{t}\hat{a}_{t}^{\top})$ is the projection matrix orthogonal to that direction.

While effective under clean conditions, this method is highly sensitive to noise in the direction of $\vec{a}_{t}$, especially in higher dimensions where ray intersections are less geometrically constrained. As such, it performs well at low noise levels but deteriorates rapidly when acceleration signals are noisy or inconsistent.

9.3 Demonstrations for Each Task

9.4 Simulation Experiment Details

9.4.1 3D Baseline Analysis

Directional Consistency Score (ours) achieved the best overall performance across all noise levels tested. Even as the force signal became increasingly corrupted, it maintained both low error and low variance: MEDE raises modestly from $0.0560\pm 0.0276$ at $10\%$ noise to $0.5814\pm 0.3722$ at $80\%$ noise. Despite its non-convexity, our method yielded stable and accurate estimates when paired with structured initialization and optimization.

Inverse Dynamics Triangulation performed well at low noise levels: $0.1111\pm 0.0818$ at $10\%$ and achieved the best performance over other methods at $0\%$ noise level with MEDE $0.0127\pm 0.0157$. However, it degraded more rapidly as the noise increases, reaching $0.9747\pm 0.5689$ at $80\%$ noise. We anticipated this behavior since this approach assumes clean Newtonian force signals and becomes unstable when acceleration vectors fluctuate or intersect imprecisely.

Cosine Similarity Score aligned force directions but disregarded magnitude. In 3D, where the directional ambiguity is high and the gradients flatten near $\cos(\theta)\approx 1$, the method struggled to guide the optimization effectively. Interestingly, increasing the level of noise improved its performance, and it reached its best performance at the noise level $50\%$ with the MEDE of $1.1981\pm 1.7585$. The cause would be the injected noise that pushes the optimization out of local minima. It produced the highest variance overall and failed catastrophically at times, especially under low noise.

Quadratic Residual Score offered a convex alternative, minimizing the L2 difference between $p-x_{i}$ and $a_{t}$. However, this formulation implicitly assumes that the observed acceleration vectors have magnitudes that are consistent with the inferred direction vectors. In our simulation, we intentionally violated this assumption by scaling the force magnitude as the distance $||p-x_{t}||$ multiplied by a stochastic coefficient drawn from a uniform distribution. This design mimicked real-world conditions, where human-applied forces are irregular and not strictly distance-proportional. As a result, the quadratic score performed poorly, with a MEDE of $0.3311\pm 0.2217$ even in the noise-free setting.

The results were shown in Appendix Table 1.

9.4.2 Lower-Dimensional Intuition

To better understand the behaviors of different scoring functions, we also conducted a controlled 2D version of the influence inference experiment with setup mirrored the 3D scenario. We evaluated the same four methods (Directional Consistency Score, Inverse Dynamics Triangulation, Cosine Similarity, and Quadratic Residual) across 50 seeds under different directional noise. The results were summarized in Appendix Table 2.

In the 2D setting, both the Directional Consistency Score (ours) and Inverse Dynamics Triagnulation demonstrated strong performance under low to moderate noise. Notably, the Directional Consistency Score achieved the most robust accuracy across $10\%$, $30\%$, and $50\%$ noise levels, which confirmed its resilience to force perturbations even in lower-dimensional dynamics. The increased standard deviation of the Directional Consistency Score at $80\%$ noise could be attributed to degraded initialization under extreme noise, where the top-$k$ acceleration magnitudes no longer reliably reflected actual force directions. As a result, optimization was more likely to begin near outliers or flat-gradient regions, leading to wider variability in outcomes.

Inverse Dynamics Triangulation again achieved the lowest MEDE of $0.0128\pm 0.0169$ in the noise-free setting. Although its performance degraded with increasing noise, the deterioration was more moderate than in the 3D case due to reduced directional ambiguity.

Quadratic Residual Score, as expected, remained stable due to its convex formulation. While it outperformed other methods at noise level $80\%$ with a MEDE of $0.4468\pm 0.2454$, this performance was not significantly better than that of the Directional Consistency Score, which achieved a comparable MEDE of $0.4965\pm 1.0663$.

9.4.3 Inference from Partial Trajectories

To assess how early the spatial intent can be reliably inferred, we evaluated our method under the constraint of partial observation. Specifically, we provided only the first $T$ timesteps of the demonstration trajectory $\{x_{t},\ddot{x}_{t}\}_{t=1}^{T}$ to the inference algorithm, and vary $T$ from $1$ to $100$. This setting emulated early recognition in human intentions when interacting with robots where only partial trajectory is observed.

Appendix Figure 8 plotted the mean inference error ($\pm 1$ std) of the fitted point across $50$ random seeds under varying trajectory lengths, for five different noise levels: $0\%$, $10\%$, $30\%$, $50\%$, and $80\%$.

Across all noise levels, we observed a significant inference error drop within the first $20-30$ timesteps and gradually plateaus afterward. At low noise ($0\%$, $10\%$), as few as $25$ steps sufficed to reliably identify the influence point. Under heavier noise ($50\%$, $80\%$), longer observation windows were required, but the Directional Consistency Score still converged to reasonable estimates under $40$ steps.

As expected, the variance was significantly higher when fewer steps are available, even under a noise-free setting, showing the ambiguity in intent when the motion was just beginning. As more of the trajectory was revealed, the system became increasingly accurate, suggesting that the method captured the cumulative structure of intention encoded in second-order motion.

9.4.4 Sequential Influence Inference

We further evaluated our method in a setting where a trajectory was governed by two spatial influence points $p_{1}$ and $p_{2}$ in sequence. The transition between the two points was defined by a known switch step, allowing the inference algorithm to segment the trajectory accordingly. For each segment, we fitted a separate influence point using our method and reported the MEDE for each inferred point.

Appendix Table 3 presented the results across three noise levels ($0\%$, $10\%$, and $30\%$) and three switch ratios: $30/70$, $50/50$, and $70/30$ corresponding to the percentage of trajectory length influenced by $p_{1}$ vs. $p_{2}$. Each configuration was averaged over 50 random seeds.

Across all settings, we observed a longer influence period of a point generally yields a lower MEDE. This result aligned with intuition that longer segments provide more consistent motion patterns to constrain the optimization.

Interestingly, we found that $30/70$** split consistently achieved the lowest overall mean error** across noise levels. This behavior could be explained by the inherent asymmetry in how motion dynamics evolved across segments. In our simulation, the first segment began with an initial velocity uniformly sampled from $[-0.5,0.5]^{3}$, representing a moderate and unbiased starting momentum. In contrast, the second segment inherited the velocity from the first, which may accumulate significant directional bias by the time of switching. Consequently, estimating $p_{2}$ required more trajectory context.

In real-world applications such as manipulation or teleoperation, this challenge may be less significant. For instance, a human operator may pause or reorient the robot between sub-tasks, effectively resetting the velocity and making both phases more distinguishable. Our simulation represented a worst-case continuity scenario and that influence inference in practice could be even more robust given appropriately segmented demonstrations.

9.5 Real World Experiment Complete Results

9.6 DMP Roll Out Definition

Position DMP:

[TABLE]

Orientation DMP:

[TABLE]

9.7 Score Visualization

9.8 Prompt to VLM

After we obtained the inferred influence point, it was projected on the initial RGB start state image. To extract semantic information about the task and the relevant environmental features, we adopt a two-phase querying strategy using a Vision-Language Model (VLM).

In the first phase, given the full demonstration trajectory overlaid on the start-state image, the VLM is queried to infer a concise task label that captures the primary objective of the motion. In the second phase, given the projected influence point on the same image and the previously inferred task label, the VLM is queried to identify the fine-grained environmental feature corresponding to the influence point and suggest an appropriate robot interaction location.

Few text-based examples are given to restrict the output to fine-grained details, without image-text pairs or in-context visual examples. The VLM is used solely as a tool for semantic extraction of task-relevant features, without any task-specific retraining or fine-tuning.

For the experiments conducted in this work, we employ GPT-4o [41], one of the latest publicly available Vision-Language Models at the time of the experiments. Due to the rapid advancement of vision-language models, the proposed pipeline is designed to directly benefits from future models with improved visual reasoning capabilities.

We provide the full text of the prompts used for querying the VLM in each phase below.

Phase 1: Task Label Inference

Instructions: You are a motion understanding expert. Provided is an image showing the initial state of a robot task, overlaid with the robot’s full demonstration trajectory.

Based on the scene context and the projected trajectory, identify the task that the robot is attempting to perform.

Respond with a concise task label that captures the primary objective of the robot’s action. Focus solely on the relevant interaction. Ignore unrelated background objects.

Examples of valid task labels include: “place object on shelf,” “connect two components,”“ align tool with fixture,” “adjust object position,” or “slide object along surface.”

Your output should be a short phrase describing the task.

Phase 2: Fine-Grained Feature Identification

Instructions: You are an environment reasoning expert. Provided is an image of the environment annotated with a projected spatial influence point, inferred from the robot’s demonstration.

The task is $\{\text{task label}\}$. Your goal is to reason about fine-grained, task-relevant environmental features based on the task and influence point. In particular:

• •

Detect precise structural features that are critical for completing the task.

• •

Avoid vague descriptions like “cabinet” or “box.” Instead, refer to specific parts, boundaries, or interaction affordances.

Based on the influence point and the task:

Describe the fine-grained environmental feature that the projected point corresponds to that is related to the task. Be specific about the geometry, material, or function if relevant. 2. 2.

Specify where the robot should grasp or interact with the object to successfully complete the task, using fine-grained features as references. Be specific about the geometry, material, or function if relevant.

Do not mention unrelated scene elements.

Provide two short phrases: one for the influence point description and one for the grasp location.

Example Outputs

• •

Task label: Open cabinet door

• •

Fine-grained description:

• –

(1) The projected point corresponds to the vertical seam between the cabinet door and the frame, near the edge where the door can be pulled open.

• –

(2) The robot should grasp the small metal door handle located along the seam to successfully open the cabinet door.

9.9 Real World Task Details and Failure Example Visualizations