Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online

Yishu Li; Xinyi Mao; Ying Yuan; Kyutae Sim; Ben Eisner; David Held

arXiv:2509.00271·cs.RO·September 3, 2025

Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online

Yishu Li, Xinyi Mao, Ying Yuan, Kyutae Sim, Ben Eisner, David Held

PDF

Open Access

TL;DR

This paper presents a history-aware verifier that improves online interaction decisions by reasoning about past interactions, significantly enhancing performance in ambiguous scenarios involving robots and complex objects.

Contribution

The paper introduces a novel history-aware verifier that explicitly decouples action generation from verification, improving decision-making in ambiguous online interactions.

Findings

01

Empirical results show improved interaction success rates across multiple environments.

02

Theoretically, employing a verifier enhances expected action quality.

03

Demonstrates effectiveness over baseline methods in real-world and simulated tests.

Abstract

We introduce a novel History-Aware VErifier (HAVE) to disambiguate uncertain scenarios online by leveraging past interactions. Robots frequently encounter visually ambiguous objects whose manipulation outcomes remain uncertain until physically interacted with. While generative models alone could theoretically adapt to such ambiguity, in practice they obtain suboptimal performance in ambiguous cases, even when conditioned on action history. To address this, we propose explicitly decoupling action generation from verification: we use an unconditional diffusion-based generator to propose multiple candidate actions and employ our history-aware verifier to select the most promising action by reasoning about past interactions. Through theoretical analysis, we demonstrate that employing a verifier significantly improves expected action quality. Empirical evaluations and analysis across…

Figures40

Click any figure to enlarge with its caption.

Tables5

Table 1. Table 1: Failure Rate % ( ↓ \downarrow ) : Lower is better. AVG c = average over categories, AVG s = average over all samples. All 22 categories are treated as train categories, and we test on unseen instances. We run each object for 30 steps in simulation, and 30 samples are given for the verifier to choose from at each step. We see HAVE’s overall performance increase compared with baselines and ablation choices, as well as a more balanced ability across categories. Icon-to-text correspondence in Fig. 16 .

Table 2. Table 3: Failure Rate % ( ↓ \downarrow ) on Held-out Categories : Lower is better. AVG c is the category-wise average, AVG s is sample-wise average. HAVE provides substantial improvements over baselines without a verifier.

Table 3. Table 4: Success Rate on MimicGen [ 37 ] : benchmark tasks ( ↑ \uparrow ). We train on 3 seeds for each task.

High-level Type	Square D2	Nut Assembly D0	Threading D2
TAX3D	0.29 $\pm$ 0.03	0.11 $\pm$ 0.08	0.10 $\pm$ 0.03
TAX3D+Verifier	0.41 $\pm$ 0.12	0.15 $\pm$ 0.13	0.19 $\pm$ 0.07

Table 4. Table 6: Overfitting Analysis (Failure Rate %; Averaged across class)

	General	Doors	Held-out	Uneven
Train set	1.84	2.96	3.69	12.20
Test set	2.26	16.77	16.92	12.93

Table 5. Table 7: Capacity Analysis on Held-Out (Failure Rate %); Metric presented as A V G sample AVG_{\text{sample}} ( A V G class AVG_{\text{class}} )

	Original	Smaller	Bigger
Train categories	2.49 (3.69)	2.86 (3.37)	2.16 (2.54)
Unseen categories	11.31 (16.92)	11.03 (18.58)	12.70 (20.57)

Equations84

\displaystyle p_{G}=P\bigl{(}R_{\text{gt}}(a)=1\,\bigm{|}\,a\sim\pi_{G}\bigr{)}

\displaystyle p_{G}=P\bigl{(}R_{\text{gt}}(a)=1\,\bigm{|}\,a\sim\pi_{G}\bigr{)}

p_{V} = P (V (a) = R_{gt} (a))

E [R_{gt} (a_{w/ver})]

E [R_{gt} (a_{w/ver})]

+ P (\forall i, V (a_{i}) = 0) \cdot E [R_{gt} (a) ∣ V (a) = 0]

= (1 - (1 - Q)^{N}) \cdot \frac{P ( R = 1 , V = 1 )}{P ( V = 1 )}

+ (1 - Q)^{N} \cdot \frac{P ( R = 1 , V = 0 )}{P ( V = 0 )}

P (R = 1, V = 1) = p_{G} \cdot p_{V}, P (R = 1, V = 0) = p_{G} \cdot (1 - p_{V}) .

P (R = 1, V = 1) = p_{G} \cdot p_{V}, P (R = 1, V = 0) = p_{G} \cdot (1 - p_{V}) .

E [R_{gt} (a_{w/ver})] = (1 - (1 - Q)^{N}) \cdot \frac{p _{G} p _{V}}{Q} + (1 - Q)^{N} \cdot \frac{p _{G} ( 1 - p _{V} )}{1 - Q} .

E [R_{gt} (a_{w/ver})] = (1 - (1 - Q)^{N}) \cdot \frac{p _{G} p _{V}}{Q} + (1 - Q)^{N} \cdot \frac{p _{G} ( 1 - p _{V} )}{1 - Q} .

E [R_{gt} (a_{w/ver})] > E [R_{gt} (a_{naive})] = p_{G} .

E [R_{gt} (a_{w/ver})] > E [R_{gt} (a_{naive})] = p_{G} .

p_{G} ⟺ d i v i d e b y

(1 - (1 - Q)^{N}) \cdot \frac{p _{V}}{Q} + (1 - Q)^{N} \cdot \frac{1 - p _{V}}{1 - Q} > 1

(1 - (1 - Q)^{N}) \cdot \frac{p _{V}}{Q} + (1 - Q)^{N} \cdot \frac{1 - p _{V}}{1 - Q} > 1

⟺

⟺

⟺

⟺

⟺

⟺

⟺

⟺

\frac{p _{V}}{Q} > 1 ⟺ p_{V} > Q .

\frac{p _{V}}{Q} > 1 ⟺ p_{V} > Q .

p_{V} > p_{G} p_{V} + (1 - p_{G}) (1 - p_{V}),

p_{V} > p_{G} p_{V} + (1 - p_{G}) (1 - p_{V}),

p_{V} - p_{G} p_{V}

p_{V} - p_{G} p_{V}

⟺ p_{V} (1 - p_{G})

p_{V} > 1 - p_{V} ⟺ p_{V} > 0.5.

p_{V} > 1 - p_{V} ⟺ p_{V} > 0.5.

\displaystyle p_{G}=P\bigl{(}R_{\text{gt}}(a)=1\,\bigm{|}\,a\sim\pi_{G}\bigr{)}

\displaystyle p_{G}=P\bigl{(}R_{\text{gt}}(a)=1\,\bigm{|}\,a\sim\pi_{G}\bigr{)}

p_{V 1} = P (V (a) = R_{gt} (a) = 1 ∣ R_{gt} (a) = 1)

p_{V 0} = P (V (a) = R_{gt} (a) = 0 ∣ R_{gt} (a) = 0)

E [R_{gt} (a_{w/ver})]

E [R_{gt} (a_{w/ver})]

+ P (\forall i, V (a_{i}) = 0) \cdot E [R_{gt} (a) ∣ V (a) = 0]

= (1 - (1 - Q)^{N}) \cdot \frac{P ( R = 1 , V = 1 )}{P ( V = 1 )}

+ (1 - Q)^{N} \cdot \frac{P ( R = 1 , V = 0 )}{P ( V = 0 )}

Q = P (V (a) = 1) = p_{G} \cdot p_{V 1} + (1 - p_{G}) \cdot (1 - p_{V 0}),

Q = P (V (a) = 1) = p_{G} \cdot p_{V 1} + (1 - p_{G}) \cdot (1 - p_{V 0}),

P (R = 1, V = 1) = p_{G} \cdot p_{V 1}, P (R = 1, V = 0) = p_{G} \cdot (1 - p_{V 1}) .

P (R = 1, V = 1) = p_{G} \cdot p_{V 1}, P (R = 1, V = 0) = p_{G} \cdot (1 - p_{V 1}) .

E [R_{gt} (a_{w/ver})] = (1 - (1 - Q)^{N}) \cdot \frac{p _{G} p _{V 1}}{Q} + (1 - Q)^{N} \cdot \frac{p _{G} ( 1 - p _{V 1} )}{1 - Q} .

E [R_{gt} (a_{w/ver})] = (1 - (1 - Q)^{N}) \cdot \frac{p _{G} p _{V 1}}{Q} + (1 - Q)^{N} \cdot \frac{p _{G} ( 1 - p _{V 1} )}{1 - Q} .

E [R_{gt} (a_{w/ver})] > E [R_{gt} (a_{naive})] = p_{G} .

E [R_{gt} (a_{w/ver})] > E [R_{gt} (a_{naive})] = p_{G} .

p_{G} ⟺ d i v i d e b y

(1 - (1 - Q)^{N}) \cdot \frac{p _{V 1}}{Q} + (1 - Q)^{N} \cdot \frac{1 - p _{V 1}}{1 - Q} > 1

(1 - (1 - Q)^{N}) \cdot \frac{p _{V 1}}{Q} + (1 - Q)^{N} \cdot \frac{1 - p _{V 1}}{1 - Q} > 1

⟺

⟺

⟺

⟺

\frac{p _{V 1}}{Q} > 1 ⟺ p_{V 1} > Q .

\frac{p _{V 1}}{Q} > 1 ⟺ p_{V 1} > Q .

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Taxonomy

TopicsMultimodal Machine Learning Applications · Robot Manipulation and Learning · Reinforcement Learning in Robotics

Full text

\doparttoc\faketableofcontents

Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online

Yishu Li1, Xinyi Mao2, Ying Yuan1, Kyutae Sim1, Ben Eisner1, David Held1

1Robotics Institute, Carnegie Mellon University

2Computer Science and Technology, Tsinghua University

Abstract

We introduce a novel History-Aware VErifier (HAVE) to disambiguate uncertain scenarios online by leveraging past interactions. Robots frequently encounter visually ambiguous objects whose manipulation outcomes remain uncertain until physically interacted with. While generative models alone could theoretically adapt to such ambiguity, in practice they obtain suboptimal performance in ambiguous cases, even when conditioned on action history. To address this, we propose explicitly decoupling action generation from verification: we use an unconditional diffusion-based generator to propose multiple candidate actions and employ our history-aware verifier to select the most promising action by reasoning about past interactions. Through theoretical analysis, we demonstrate that employing a verifier significantly improves expected action quality. Empirical evaluations and analysis across multiple simulated and real-world environments including articulated objects, multi-modal doors, and uneven object pick-up confirm the effectiveness of our method and improvements over baselines. Our project website is available at: https://liy1shu.github.io/HAVE_CoRL25/.

Keywords: Ambiguities, Multi-modality, Generation-Verification

1 Introduction

In the real world, objects that appear visually similar can exhibit vastly different physical behaviors. A closed door might look identical regardless of whether it opens by pushing or pulling, and two boxes with the same outward appearances may demand different manipulation strategies depending on how their inside contents are distributed. A visual policy without history-based adaptation, no matter how large, would struggle to disambiguate such cases. When encountering a new object, we want a robot system to explore in a trial-and-error manner, observe the outcomes of its interactions, and adapt its strategy accordingly. Our goal is to enable this kind of history-aware disambiguation, where the robot continuously updates its internal model to improve future behavior.

Ideally, we can achieve this adaptation simply with a history-conditioned generator policy. However, our experiments with a history-conditioned diffusion model led to suboptimal performance. Indeed, recent work has demonstrated the theoretic and empirical difficulty of training a policy (via supervised fine-tuning) under settings in which the ground-truth solution traces are heterogeneous for the same input [1]. This prior work demonstrates (theoretically and empirically) the benefits of instead using verifier-based methods, even with learned rewards.

Inspired by this work, we propose, for ambiguous object manipulation, to train a history-aware verifier that takes in the history interactions and estimates a reward for action proposals sampled from a probabilistic action generator. We demonstrate, theoretically and empirically, the benefits of this approach compared to using only a history-conditioned generator policy. We show theoretically that using any better-than-random-chance verifier to select an action proposed by a generator policy can improve the policy performance. We also demonstrate the effectiveness of our approach in different simulation and real world environments with fundamental ambiguity including opening ambiguous articulated objects, ambiguous doors, and picking up objects with an uneven mass distribution. We also perform ablations and analysis to better understand the importance of various design choices of our proposed method. The contributions of our paper include:

(1)

A novel verifier structure that reasons about past interactions for disambiguation. 2. (2)

Theoretical analysis of using a verifier to select actions. 3. (3)

Simulated and real world experiments in multiple environments with fundamental ambiguities, such as opening ambiguous articulated objects, ambiguous doors, and lifting objects with uneven mass distributions.

2 Problem Statement and Assumptions

In this paper, we aim to tackle the problem of learning a policy in a Partially Observable Markov Decision Process (POMDP), in which the agent does not observe the true state but instead receives an observation that provides partial information about the state. In such a case, the history of observations and actions with the environment is essential for being able to estimate the true underlying state of the environment, which is important for effective decision-making. Therefore, leveraging past interactions—both successful and failed—can help an agent make more informed decisions.

Formally, the environment is characterized by a distribution over POMDPs. More concretely, consider a POMDP defined over a state space $\mathcal{S}$ , an action space $\mathcal{A}$ , and an obervation space $\mathcal{O}$ , with a transition function $\mathcal{T}:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$ , where $\Delta(\mathcal{S})$ denotes the set of probability distributions over $\mathcal{S}$ , and $\mathcal{T}(s^{\prime}\mid s,a)$ denotes the probability of transitioning to state $s^{\prime}$ after taking action $a$ in state $s$ . For each POMDP, assume that the (unknown) transition function $\mathcal{T}$ is sampled from a distribution $\mathcal{P}_{\mathcal{T}}$ over possible transition dynamics, that is, $\mathcal{T}\sim\mathcal{P}_{\mathcal{T}}$ . For example, consider a family of door-opening tasks, where the agent cannot determine from a single observation $o\in\mathcal{O}$ whether a door should be pushed or pulled. However, through interactions with the environment, the agent can use the history of observations and actions to implicitly infer the underlying transition dynamics of the current instance of the POMDP. Our task for the agent to use this history of observations and actions to discover the optimal policy for each POMDP.

3 Related Works

Interactive Perception: Prior works have explored how to use interactions to actively acquire informative observations to infer about hidden physical properties or dynamics in various robotics contexts [2, 3] from articulation [4] to deformable objects [5]. More recently, in-context learning methods have been used to adapt robot behavior online by conditioning on recent interactions [6, 7] through reinforcement learning [8, 9] or directly modeling simulation parameters [10]. Our work builds on this idea, introducing a generator-verifier system that decomposes policy generation and verification, and explicitly evaluates action candidates using prior interactions, enabling efficient and robust online disambiguation in ambiguous, multi-modal settings.

Generation-Verification Paradigm: Many recent advances in machine learning build on a generation-verification paradigm, from adversarial learning in a GAN [11], a policy generator and reward model in reinforcement learning and RLHF [12, 13, 14, 15], to test-time scaling in LLMs [16, 17, 18, 19, 20, 1]. Many of these work [1, 21, 15] stress the importance of verification model from different theoretical views. In parallel, some safety-focused approaches in robotics [22, 23, 24, 25, 26, 27] have adopted a similar structure with a nominal generator / policy and a verifier / filter to ensure constraint satisfaction at runtime. Aligning with these prior ideas on a high level, we focus on the theoretical explanation and experimental analysis on the benefit of a system with both a generator and a verifier.

4 Theoretical Motivation

The main idea behind our approach is as follows: Given an (unobserved) state space of $S$ , an observation space $O$ , and an action space of $A$ , our proposed framework consists of a generative policy $\pi_{G}:O\times A\to[0,1]$ that models the distribution of actions given an observation, and a verifier (or learned reward function) $V:O\times A\to\mathbb{R}$ that evaluates each of the proposed actions and outputs an estimated (scalar) reward for each one. During inference, we sample a batch of $N$ actions $\{a_{1},\ldots,a_{N}\}$ from the generator, $a_{i}\sim\pi_{G}(s)$ . We predict the reward of each sampled action with the verifier $V(a_{i})$ and then execute the action with the highest estimated reward. Below we present some theoretical justifications for this approach.

4.1 Generation-Verification Gap

Our approach is inspired by the theoretical result of Setlur et al. [1], which describes a setting in which the correct solution traces for a given input are hetereogeneous (as defined in Property 5.2 in Setlur et al. [1]). This situation is analogous to our setting in which there is a partially observed environment; in such a case, there will be diverse optimal actions for the same observation (in which the true optimal action depends on the unobserved ground-truth state of the environment).

Setlur et al. [1] then show that, in such cases, training a generative policy to imitate a hetereogeneous expert dataset will induce a reward gap of $\Omega\Big{(}\sigma_{e}\sqrt{1/n}\Big{)}$ , where $n$ is the amount of expert data, and $\sigma_{e}$ is variance describing the heterogeneity in the expert solution traces (Theorem 5.4). In our problem setting (Sec. 2), the ground-truth differs given the same partial observation due to different underlying transition function, leading to a large $\sigma_{e}$ . On the other hand, they show that if we train a verifier to fit the reward terms in the expert dataset, we can achieve an expected error that contracts at a rate of $\widetilde{\mathcal{O}}_{H}\!\left(H/n\right)$ (Proposition 5.5), where $H$ is the horizon length. Therefore we can see that using a verifier is asymptotically more data-efficient as the reward gap contracts with the scale of $1/n$ while the reward gap in the generation model only contracts with a scale of $\sqrt{1/n}$ .

4.2 Verifier-based Selection

Our manipulation framework uses a verifier to select the best action sampled from an action generator, compared to just sampling from the generator once. Below we provide a theoretical justification for this idea. In the below analysis, we use the simplified “non-contextual” setting in which the generator and verifier are not conditioned on the state. Consider a task with binary rewards. The generator $\pi_{G}$ represents a distribution over the action space $A\subseteq\mathbb{R}^{n}$ where we can sample actions from, the ground truth reward model $R_{gt}:A\to\{0,1\}$ maps actions $a\in A$ to binary rewards, and the verifier $V:A\to\{0,1\}$ maps actions $a\in A$ to estimated binary rewards. In reality, both the generator and the verifier are trained models that make errors with some probability. We thus make the following assumptions:

The accuracies of the generator and the verifier are independent. 2. 2.

The generator has probability $p_{G}$ of sampling an action with reward 1: $P\left(R_{\text{gt}}(a)=1\mid a\sim\pi_{G}\right)=p_{G}$ . 3. 3.

The verifier has probability $p_{V}$ of predicting the correct reward for any action: $P\left(V(a)=R_{\text{gt}}(a)\right)=p_{V}$ , $\forall a\in A$ .

We compare the expected reward of 2 methods:

•

Method 1: Sample one action from the generator: $a_{\text{naive}}\sim\pi_{G}$ .

•

Method 2: Sample $\mathrm{N}$ actions ${a_{1},\ldots,a_{\mathrm{N}}}$ from the generator and use the verifier to choose the action with the largest estimated reward: $a_{\text{w/ver}}=a_{i^{*}}$ , where $i^{*}=\arg\max_{i\in\{1,\ldots,\mathrm{N}\}}V(a_{i})$ . If multiple actions have the same highest score predicted by the verifier, randomly pick one of them.

For Method 1, the expected reward can be computed as: $\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]=p_{G}\times 1+(1-p_{G})\times 0=p_{G}$

For Method 2, the expected reward can be computed as: $\mathbb{E}\left[R_{\text{gt}}(a_{\text{w/ver}})\right]=\left(1-(1-Q)^{\mathrm{N}}\right)\frac{p_{G}p_{V}}{Q}+(1-Q)^{\mathrm{N}}\frac{p_{G}(1-p_{V})}{1-Q}$ , $\text{where }Q=P\left(V(a)=1\right)=(1-p_{G})(1-p_{V})+p_{G}p_{V}.$ Then we can prove (refer to the Appendix A.1 for full proof) that with $\mathrm{N}>1$ and $p_{V}>0.5$ , we have $\mathbb{E}\left[R_{\text{gt}}(a_{\text{w/ver}})\right]>\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]$ . This demonstrates that our verifier is expected to be useful as long as it has an accuracy greater than 50%. As a numerical example, if we let $\mathrm{N}=2$ , $p_{G}=p_{V}=0.9$ , we have $\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]=0.9$ and $\mathbb{E}\left[R_{\text{gt}}(a_{\text{w/ver}})\right]=0.97$ , meaning that the failure rate (the probability of executing an action with 0 reward) is reduced from 10% to 3% even with only one extra sample. See Appendix A for theory the independence assumption, extension to continuous rewards, and numerical examples for both discrete and continuous cases.

5 Method

Based on the above motivation, we construct our method, which consists of an unconditional diffusion-based generator and a history-aware verifier. We demonstrate in our experiments that such an architecture leads to much better performance than a history-conditioned diffusion model without a verifier, as also suggested by the above theoretical analysis.

Given the action generator $G_{\phi}$ , at each timestep $t$ , we sample action proposals $a_{t}^{(0)},\dots,a_{t}^{(\textrm{N}-1)}$ based on the current observation $o_{t}$ . The history-aware verifier $V_{\theta}$ then outputs corresponding scores $s_{t}^{(0)},\dots,s_{t}^{(\textrm{N}-1)}$ for the proposed actions, considering the history of past actions and their outcomes, $(o_{0},a_{0},\dots,o_{t-1},a_{t-1},o_{t})$ . Finally, we select and execute the action with the highest estimated score $a_{t}^{(\hat{i})}$ , where $\hat{i}=\arg\max_{i}s_{t}^{(i)}$ . Below we describe the details of our approach.

5.1 Dataset Generation

We desire for our method to be able to quickly adapt its future actions after performing either high-reward or low-reward actions and observing the results. Thus, we generate an offline dataset for training where each sample consists of a history sequence and an action proposal set. We generate each history sequence by either sampling a random action (most likely leading to a failure) or using the ground truth action (obtained using privileged knowledge in the simulator) at each step. The action proposal set is constructed with the ground-truth action, random actions and history actions. We label the score for each action proposal and history actions; see Appendix B.2 for details.

5.2 History-Aware Verifier

We assume that we are given an action proposal $\hat{a}$ as well as history actions and observations. Specifically, to learn from history, we group the data based on the observation before an action and the resulting observation after an action: $\{(o_{i-1},a_{i},o_{i}),i\in[t-1]\}$ . We compute action embeddings $E_{\textrm{action}}:A\to\mathbb{R}^{n}$ (where $n$ is the encoded action’s feature dimension) to obtain an embedding $\hat{a}_{\textrm{emb}}=E_{\textrm{action}}(\hat{a})$ for the action proposal $\hat{a}$ as well as an embedding for each action in the action history $\{a_{\textrm{emb},i}=E_{\textrm{action}}(a_{i}),i\in[t-1]\}$ . We also encode each pair of subsequent observations $E_{obs}:O\times O\to\mathbb{R}^{m}$ (where $m$ is the observation feature dimension) to obtain observation embeddings: $\{f_{i}=E_{\textrm{obs}}(o_{i-1},o_{i}),i\in[t-1]\}$ . We then input the action embeddings for the proposal as well as the history through an action transformer to receive an updated action proposal embedding $\hat{a}_{\textrm{emb}}^{\prime}$ and new history action embeddings $a^{\prime}_{\textrm{emb},i}$ . We also input the observation embeddings into an observation transformer to receive updated observation history embeddings $f^{\prime}_{i}$ . The action proposal embedding $\hat{a}_{\textrm{emb}}$ is also passed through an unconditional action encoder to obtain an unconditional evaluation of the proposed action without conditioning on history: $\hat{v}_{u}=E_{\textrm{uncond}}(\hat{a}_{\textrm{emb}})$ (used below).

Given a history of length $t$ , we perform an explicit dot-product attention operation with $\hat{a}_{\textrm{emb}}^{\prime}$ as the query ( $Q$ ), action history embeddings $\{a^{\prime}_{\textrm{emb},i}\}_{i=1}^{t}$ as keys ( $K$ ) and the observation history embeddings $\{f^{\prime}_{i}\}_{i=1}^{t}$ as values ( $V$ ): $\hat{v}=\text{softmax}\left(\left[QK^{\top},\mu_{qk}\right]/\sqrt{d_{k}}\right)[V,\hat{v}_{u}]$ , where $d_{k}$ is the dimension of key embeddings, and $[\cdot,\cdot]$ denotes concatenation along the key/value dimension. Here $\mu_{qk}$ denotes a constant to represent the logit associated with the unconditional evaluation of the proposed action $\hat{a}$ (without regard to the history), which we compute as a running mean of the $QK^{T}$ logits during training. The intuition behind this explicit attention layer is that we want the model to relate the action proposal $\hat{a}$ (encoded as the query $Q=\hat{a}_{\textrm{emb}}^{\prime}$ ) with similar actions in the history (encoded as the keys $K=\{a^{\prime}_{\textrm{emb},i}\}$ ) and refer to their action results (encoded as values $V=\{f^{\prime}_{i}\}$ ). The unconditional logit $\mu_{qk}$ and embedding $\hat{v}_{u}$ exists to handle cases where there are no actions similar to the proposal $\hat{a}$ in the history (or no actions in the history at all). Then the final embedding $\hat{v}$ will go through a score decoder (small MLP with tanh on the output) to obtain a 1-dim logit as the final output score: $s=D_{\textrm{score}}(\hat{v})\in[-1,1]$ . We compare our proposed architecture to a monolithic transformer in Sec. 6.3 and show improved performance with our approach.

During training, we supervise the final predicted score $s$ for the action proposal, using an MSE loss with the ground-truth scores (referred to as “Final Score Loss” in Fig. 1). We also supervise the unconditional embedding $\hat{v}_{u}$ and the action results history embeddings $\{f^{\prime}_{i}\}_{i=1}^{t}$ by passing them through the score decoder $D_{\textrm{score}}$ and supervising them with ground truth rewards for corresponding timestep (referred to as “Unconditional Score Loss” and “History Score Loss” respectively in Fig. 1).

5.3 Network Architecture

We represent actions as dense action fields, where given a point cloud $\{p_{i}\}$ , and an action point $p$ and direction $d$ , the dense action field is the point cloud with 3 extra channels to represent the action, calculated as $\{d_{i}:d*\exp\{-c*||p_{i}-p||_{2}^{2}\}$ where $c$ is a scaling constant. This representation also makes actions state-aware, enabling reasonable understanding about relationships between actions executed in different states. We hypothesize that this dense action representation is easier to process than a sparse action representation, as indicated by our ablation experiments. We then encode this dense action representation $a_{\textrm{emb},i}=E_{\textrm{action}}(a_{i})$ using a PointNet++ [28] point cloud encoder.

We represent the observation encoder $E_{\textrm{obs}}(o_{i-1},o_{i})$ as follows: First, we compute the flow from each point in observation $o_{i-1}$ to the corresponding location of that point in observation $o_{i}$ , which we refer to as the “Observation Flow.” In training we use the ground-truth flow, and at test time we use a state-of-the-art 3D point tracker [29]. We then create a point cloud with 3 extra channels for the 3D flow, which we also encode with a PointNet++ [28] point cloud encoder. We construct the action generator similar to prior work [30], but without history injection; see Appendix B for details.

6 Experiments and Analysis

To evaluate our system’s capability to solve ambiguous robotics tasks, we carried out experiments in 3 different environments: opening general articulated objects, opening ambiguous doors, and lifting objects with uneven mass distributions. We perform experiments both in simulation environments and in a real-world robot setup. At inference time, the robot incrementally builds its history through each interaction with the object.

6.1 Articulated Objects

Baselines: We compare our method to the following baselines: (1) FlowBot3D [31], (2) FlowBotHD [30] with and without “CC” (the heuristic consistency check in FlowBotHD that filters the proposed actions), (3) Generator Only: unconditional diffusion without a verifier, (4) Conditional Generator: conditional diffusion that takes in the same history interaction information as the verifier, we use classifier-free guidance [32] for conditioning (see Appendix C.5 for architecture details, and E.5 for different cfg scales). For all baselines and our own policy, we train the action generator with a training dataset augmented with fully closed object examples [30], and adopt the FlowBotHD switch grasp policy to enable models to adapt from wrong grasp points (see Li et al. [30] for details).

Ablations: We also perform the following ablations to better understand the effect of our design choices: (1) w/o History Score Loss: Train without the “History Score Loss” (see Sec. 5.2 and Figure 1); (2) w/o Uncond Score Loss: Train without the “Unconditional Score Loss” (see Sec. 5.2 and Figure 1); (3) Sparse Action: rather than representing an action as a dense action field (Sec. 5.3), we represent the action as a point cloud with only one action vector at the contact point; (4) Point Cloud as Result: directly use point cloud sequence as action result without explicit extracting observation flow (Sec. 5.3). Please refer to Appendix C.3 for more details. As stated in Sec. 5.3, for the observation encoder $E_{\textrm{obs}}(o_{i-1},o_{i})$ we compute the flow from each point in observation $o_{i-1}$ to the corresponding location of that point in observation $o_{i}$ . The ablations all use the ground-truth flow at test time (obtained from the simulator), to give the ablations the best chance to succeed. For a fair comparison, we show the performance of our method both using ground-truth flow (“HAVE (Ours) + GT obs flow”) as well as our method using estimated flow (“HAVE (Ours) + Estimated flow”) using the 3D tracker DELTA [29].

6.1.1 General Articulated Objects in Simulation

We follow Li et al. [30] and split PartNet-Mobility into train and test instances, and construct a simulation environment with a suction gripper. We initialize each object at a fully closed state, and the task of the policy is to fully open the objects. In Table 1, we show the Failure Rate: the percentage of objects that are opened less than 90% within 30 steps. Given the stochasticity in the generative model, we test each object 5 times. We also report experiments with the split following Eisner et al. [31] where we hold-out 10 categories during training and test on them (Appendix D.1) .

Comparing the failure rate from our policy and the baselines, we can see that HAVE (Ours) reduces the failure rate by about 6x (from 13% to 2%) compared to FlowBot3D [31], about 10x (from 20% to 2%) compared to using a history-conditioned generative model without a verifier, and about 3x (from 6% to 2%) compared to FlowBotHD [30] even with the heurisitc consistency check (CC) policy. This validates the idea that using a generator to mimic a heterogeneous dataset is not sufficient for optimal performance without a verifier.

Ablation Analysis: Our ablation analysis reveals the benefit of each of our design choices. Using a sparse action representation instead of a dense action field (Sparse Action) decreases the performance most. Using point cloud sequences without extracting observation flows (Point Cloud as Result) also underperforms using explicit observation flow. Training without the “History Score Loss” or the “Unconditional Score Loss” leads to a slight performance drop, indicating that supervising the history observation and the unconditional action embedding benefits the model.

6.1.2 Multimodal Doors in Simulation

To further test our method’s performance in an ambiguous setting, we also create a multimodal dataset in simulation with doors from PartNet-Mobility. We create multiple duplicates of each door with the same geometry but different joint positions and rotation directions (corresponding to 4 different opening directions - push left, push right, pull left, and pull right). For doors with handles, we create 2 duplicates with handles (varying whether the door is push or pull), and we create 4 duplicates with the handle removed. In Figure 2, we show an example of a multi-modal door in simulation. In addition to Failure Rate we also compute the Mean steps to Open (the steps taken to open the door to 5%) as a metric of efficiency of disambiguating different modes with exploration. From Figure 2, we can see that our method significantly improves the performance (reduces the failure rate and step number) on ambiguous doors compared to prior approaches and ablations (except for the minor design variant “w/o Unconditional Score Loss”), further validating our approach.

6.1.3 Real World Ambiguous Door

We apply our method in the real world on a custom-made multi-modal door, similar to the multimodal door used in Li et al. [30]. To manipulate this object, we use a Franka Emika Panda Robot with a Schmalz Cobot Pump as an end effector. We give each policy a maximum of 5 steps to succeed; see Appendix D.3 for further details. We can see from Figure 3 that HAVE (Ours) has a greater success rate over the different modes with a lower number of mean steps to open.

6.2 Uneven Object Pick-up

The task is to lift objects with unevenly distributed mass (unknown mass center). We constructed a simulation environment with a Franka gripper equipped with a spatula, along with an ambiguous rod dataset (one rod, but different mass centers) and an unseen object dataset (unseen sizes of rods, and diverse objects), see Appendix D.2 for details. In both cases, positions for the centers of mass are uniformly sampled along the object’s length. We train on the ambiguous rod dataset and evaluate on both datasets to demonstrate the model’s capability and generalization ability.

In Table 2, we show the Failure Rate %: the percentage of objects failed to be lifted within 5 steps, defined as those with a tilt angle exceeding 0.01 radians or an increase in the vertical distance of the center of mass less than 0.01 meters. Each object is tested 5 times in the ambiguous rod dataset and 3 times in the unseen dataset. At each step, the verifier is given 20 action proposals. From Table 2, we can see that on both the ambiguous rod and the unseen dataset, adding HAVE (Ours) as a verifier decreases the failure rate compared to only using either unconditional generator or conditional generator and other three baselines. Unlike in more complex environments like articulated objects, HAVE performs well with both generator types, likely due to a simpler action space. Figure 4 shows an example visualization of the “Conditional Diffusion” model (left) compared to our model’s output (right); we show the theoretical upper and lower bounds of the rod’s center of mass after each attempt (the tilting direction at each step indicates whether the selected point is on the left / right side of the mass center) as bars, and the predicted positions by the model as scatter points.

6.3 Analysis

Architecture.

We compare our architecture with a vanilla transformer baseline where the action tokens ( $\hat{a}_{emb}$ , $\{a_{emb,i}\}_{i=1}^{t}$ ) and observation tokens ( $\{f_{i}\}_{i=1}^{t}$ ) are concatenated correspondingly and processed by self-attention layers (see Appendix C.4). Specifically, we demonstrate each model’s robustness to noisy estimation in the Observation Flow (Sec. 5.3) in all three articulated object environment (Fullset, Multi-modal door, and Held-out). Figure 5 shows that both models perform similarly using the ground-truth flow; however, using estimated flows (from the DELTA 3D tracker [29]) hurts the performance of the vanilla transformer more than our architecture (HAVE). This validates the benefit of our architectural design. Our intuition is that our design forces the model to learn the relationship between query and history actions, making our model more robust to noise.

Number of samples:

We analyze the sample efficiency of our verifier by conducting ablations on how many samples we generate for the verifier to choose from for each step. From Figure 6, we can see that by sampling 5 samples, the performance is already greatly improved, which demonstrates the efficiency of incorporating a verifier. We also plot the time cost measured with 1 RTX4090 GPU on the left to demonstrate the time required to generate and evaluate multiple samples.

7 Conclusion

In this work, we present HAVE, a novel history-aware verifier that reasons over past interactions to guide action selection in ambiguous manipulation tasks. By leveraging a verifier to select among sampled proposals, our method significantly improves efficacy and efficiency compared to baseline approaches. We provide both theoretical analysis and empirical validation across a range of simulated and real-world environments, demonstrating the benefits of our approach in handling fundamental ambiguities in manipulation. We hope that our findings can help inspire more adaptive, history-aware manipulation paradigms through combining generation and verification.

8 Limitations

While our history-aware verifier demonstrated great improvements over baselines in various settings, the current generation-verification pipeline’s performance is still strictly bounded by the expressiveness of the generator: if none of the generator’s proposals are good, the verifier won’t be able to make improvements. Furthermore, our training data generation relies on per-task defined score and privileged knowledge about the ground-truth action, typically obtained from simulation. This level of supervision may not be practically available for a diverse range of objects in real-world scenarios. In future work, we hope to further explore how the verifier can be used to actively improve the quality of the generator’s generation, and how to learn score without privileged supervision.

Acknowledgments

This material is based upon work supported by the National Science Foundation under NSF CAREER Grant No. IIS-2046491 and ONR MURI N00014-24-1-2748. We thank Amrith Setlur, Aviral Kumar, and Gokul Swamy for helpful insights and discussions about generator and verifier gap, Yufei Wang for helpful discussions, and Alexis Hao for help with real robot experiments, and Xilun Zhang for discussion on uneven object environment setups.

Appendix

\parttoc

Appendix A Theoretical Motivation

A.1 Discrete Reward with Independent Generator Verifier Accuracies

Let $a\sim\pi_{G}$ be an action sampled from the generator, $p_{G}$ be the probability of the generated action having reward 1. Let $R_{\text{gt}}(a)\in\{0,1\}$ be the ground-truth reward (i.e., whether the action is good or bad), and $V(a)\in\{0,1\}$ be the verifier output.

We define:

[TABLE]

that is, $p_{G}$ is the probability the generator generates a good action, and $p_{V}$ is the probability the verifier predicts the correct reward given an action, independent of the ground truth action reward (We will show extension without this independence assumption in Theory A’).

We assume that $p_{G}\in(0,1)$ , meaning that the generator does not always generate correct / incorrect action proposals; otherwise, the performance will be deterministic and a verifier would not be needed.

Theorem A.

Given $p_{G}\in(0,1)$ , $N>1$ , then $\mathbb{E}\left[R_{\text{gt}}(a_{\text{w/ver}})\right]>\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]\iff p_{V}>0.5$

*Proof. *

The expected reward of naive sampling from generator: $\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]=p_{G}\times 1+(1-p_{G})\times 0=p_{G}$ .

The expected reward of the selected action using the verifier is:

[TABLE]

$\text{where }Q=P\left(V(a)=1\right)=(1-p_{G})(1-p_{V})+p_{G}p_{V}$ . and

[TABLE]

Substituting into the expression, we get:

[TABLE]

We will first prove $\mathbb{E}[R_{\text{gt}}(a_{\text{w/ver}})]>\mathbb{E}[R_{\text{gt}}(a_{\text{naive}})]\Rightarrow p_{V}>0.5$ , and show that each step is reversible to prove the other direction.

[TABLE]

Rewriting and simplifying:

[TABLE]

Since $p_{G}\in(0,1)$ , we have $Q\in(0,1)$ , then $1-(1-Q)^{N-1}>0$ , so the inequality holds if and only if:

[TABLE]

Substituting the expression for $Q$ :

[TABLE]

Rearranging:

[TABLE]

Since $p_{G}\in(0,1)$ , we can divide both sides by $1-p_{G}$ , yielding:

[TABLE]

Since all of the above steps are reversible, we prove that the verifier improves the expected reward over naive sampling if and only if $p_{V}>0.5$ .

Therefore, when the verifier has better than random accuracy $p_{V}>0.5$ , the overall expected reward is improved. This theory demonstrates the effect of our generator-verifier system.

Numerical Analysis:

We also demonstrate the expected reward as a function of the number of samples $N$ for different values of $p_{G}$ and $p_{V}$ in Fig. 7. We plot the simulated results where we simulate 10k trials, within each trial we sample $N$ samples and simulate the ground truth and verifier reward with $p_{G}$ and $p_{V}$ . We can see even a very biased generator ( $p_{G}=0.2$ ) can be significantly improved by a good verifier ( $p_{V}=0.9$ ) to almost 3x accuracy with only 10 samples.

A.2 Discrete Reward with Dependent Generator Verifier Accuracies

We can also extend the theory to cases where the generator and the verifier is not independent. In the binary reward setting, dependence implies that the accuracy of the verifier differs depending on whether the generated action is good or bad, that is, $p_{V1}=P(V(a)=R_{gt}(a)\mid R_{\text{gt}}(a)=1)$ and $p_{V0}=P(V(a)=R_{gt}(a)\mid R_{\text{gt}}(a)=0)$ . Below we show our theory can be extended to this case.

Let $a\sim\pi_{G}$ be an action sampled from the generator, $p_{G}$ be the probability of the generated action having reward 1. Let $R_{\text{gt}}(a)\in\{0,1\}$ be the ground-truth reward (i.e., whether the action is good or bad), and $V(a)\in\{0,1\}$ be the verifier output.

We define:

[TABLE]

that is, $p_{G}$ is the probability the generator generates a good action, and the verifier’s accuracy is dependent on the generation quality (ground truth reward of the action), $p_{V1}$ is the probability the verifier predicts 1 given the action is good, and $p_{V0}$ is the probability the verifier predicts 0 given the action is bad.

Thm. A′.

Given $p_{G}\in(0,1),N>1$ , then $\mathbb{E}[R_{\text{gt}}(a_{\text{w/ver}})]>\mathbb{E}[R_{\text{gt}}(a_{\text{naive}})]\iff p_{V1}+p_{V0}>1$ .

Proof.

The expected reward of naive sampling from generator: $\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]=p_{G}\times 1+(1-p_{G})\times 0=p_{G}$

The expected reward of the selected action using the verifier is:

[TABLE]

where

[TABLE]

and

[TABLE]

Substituting into the expression, we get:

[TABLE]

We will first prove $\mathbb{E}[R_{\text{gt}}(a_{\text{w/ver}})]>\mathbb{E}[R_{\text{gt}}(a_{\text{naive}})]\Rightarrow p_{V1}+p_{V0}>1$ , and show that each step is reversible to prove the other direction.

[TABLE]

Rewriting and simplifying (omitting steps similar with proof of Theory A):

[TABLE]

Since $p_{G}\in(0,1)$ , we have $Q\in(0,1)$ , then $1-(1-Q)^{N-1}>0$ , so the inequality holds if and only if:

[TABLE]

Substituting the expression for $Q$ :

[TABLE]

Rearranging:

[TABLE]

Since $p_{G}\in(0,1)$ , we can divide both sides by $1-p_{G}$ , yielding:

[TABLE]

Since all of the above steps are reversible, we prove that the verifier improves the expected reward over naive sampling if and only if $p_{V1}+p_{V0}>1$ .

A.3 Continuous Reward

We now extend the theoretical analysis to continuous rewards. Consider a task where the ground truth reward function $R_{gt}:A\to\mathbb{R}$ that assigns real-valued rewards. The generator $G$ produces actions $a\in A$ such that $R_{gt}(a)$ follows a distribution with mean $\mu_{G}$ and variance $\sigma^{2}_{G}$ . The verifier $V:A\to\mathbb{R}$ provides scores for a given action. We make the following assumptions:

•

Assumption 1: The generator’s reward follows a normal distribution $X=R_{gt}(a)\sim\mathcal{N}(\mu_{G},\sigma_{G}^{2})$ .

•

Assumption 2: The verifier’s score satisfies: $Y=V(a)=X(a)+\epsilon$ , where $\epsilon\sim\mathcal{N}(0,\sigma_{V}^{2})$ and is independent of $X$ , therefore $Y\sim\mathcal{N}(\mu_{G},\sigma_{G}^{2}+\sigma_{V}^{2})$ .

We compare the expected reward of 2 methods:

•

Method 1 (Naive Sampling): Sample 1 action $a_{\text{naive}}\sim G$ , the expected reward is $\mathbb{E}\left[R_{\text{gt}}(a_{\text{naive}})\right]=\mu_{G}$ .

•

Method 2 (Verifier Selection): Sampling $N$ actions and selecting $a_{\text{w/ver}}=\arg\max_{i}V(a_{i})$ . We denote the expected reward as $\mathbb{E}[R_{gt}(a_{\text{w/ver}})]=\mu_{G}+\Delta_{ver}$ .

We denote $X_{(N)}$ as the maximum ground truth reward of the $N$ samples: $X_{(N)}=\max\{X_{1},X_{2},...,X_{N}\}$ , where $X_{i}=R_{gt}(a_{i})$ . We also define $Y_{(N)}$ as the maximum verifier’s reward of the $N$ samples: $Y_{(N)}=\max\{Y_{1},Y_{2},...,Y_{N}\}$ , where $Y_{i}=V(a_{i})$ .

Following results derived from Extreme Value Theory on normal variables (Theorem 1.5.3 in Leadbetter et al. [33] and Theorem 5.1 from DasGupta et al. [34]), we first approximate $\mathbb{E}[Z_{(N)}]$ where $Z_{(N)}$ is the maximum reward of $N$ samples from a standard normal distribution $\mathcal{N}(0,1)$ :

[TABLE]

Given Assumptions 1 and 2, $X$ and $Y$ obey a bivariate normal distribution with correlation $\rho$ :

[TABLE]

Therefore the conditional expectation for X given Y is $\mathbb{E}[X|Y=y]=\mu_{X}+\sigma_{X}\,\rho\,\frac{y-\mu_{Y}}{\sigma_{Y}}=\mu_{G}+\sigma_{G}^{2}/(\sigma_{G}^{2}+\sigma_{V}^{2})*(y-\mu_{G})$ , Therefore we have:

[TABLE]

We provide numerical simulation results to visualize and validate the approximation: at each trial, we sample N samples from $\mathcal{N}(\mu_{G},\sigma^{2}_{G})$ , and add a verifier noise to each sample $\epsilon\sim\mathcal{N}(0,\sigma^{2}_{V})$ , and record the difference (improvement) of the reward of the sample with the largest verifier score and the reward of the first sample. Then we plot the simulated average improvement over 10k trials with the theoretical approximation against different $N$ , $\sigma_{G}$ , and $\sigma_{V}$ in Fig. 8. We can see the theoretical approximation stays close to the simulation data and is intuitively interpretable. As $N$ grows, $E[Z_{(N)}]\sim\sqrt{2\ln N}$ , so the gain increases with the number of samples, in proportion to $\sqrt{2\ln N}$ . For fixed verifier noise $\sigma_{V}$ , the prefactor $\sigma_{G}^{2}/\sqrt{\sigma_{G}^{2}+\sigma_{V}^{2}}$ grows nearly linearly in $\sigma_{G}$ once $\sigma_{G}\gg\sigma_{V}$ , meaning a wider ground‐truth reward spread yields almost proportional extra potential benefit from the verifier. Conversely, as the verifier uncertainty $\sigma_{V}$ increases, the same prefactor decays like $1/\sigma_{V}$ meaning that the improvement brought by verifier decays inversely proportional to $\sigma_{V}$ .

We also numerically simulate improvements with different continuous reward distribution and verifier model apart from the bivariate normal definition above:

Gaussian Mixture Reward

We simulate the situation where the reward of the generator’s proposals follows a gaussian mixture model distribution $p(x)=0.5*\mathcal{N}(-0.5,\sigma^{2}_{G})+0.5*\mathcal{N}(0.5,\sigma_{G}^{2})$ (e.g. when the generator can generate multiple action modes with equal possibilities, like push and pull a door). We plot the improvements against $N$ , $\sigma_{G}$ and $\sigma_{V}$ in Fig. 9.

Uniform Reward

We simulate the situation where the reward of the generator’s proposals follows a uniform distribution $U(0,1)$ (e.g. when the generator can generate various actions with equal possibilities), and plot the improvements against $N$ and $\sigma_{V}$ in Fig. 10. We can see that the with about 10 samples the improvement is beginning to become significant.

Verifier Pairwise Accuracy

We also demonstrate the effect of verifier against verifier pairwise accuracy: $P\left(V(a)>V(b)\ |\ R_{gt}(a)>R_{gt}(b)\right)=p_{V}$ . This measure removes the assumption that the verifier is an unbiased estimation of the ground truth reward. We simulate with reward distribution of $U(0,1)$ and plot the improvements against $N$ and $p_{V}$ in Fig. 11. We can see that with $p_{V}>0.6$ there is a significant improvement.

Using the above theoretical and numerical analysis, we see the improvements of the method that uses the verifier together with the generator (compared to only using the generator) across various reward distributions, both discrete and continuous, validating the soundness and efficacy of our generator-verifier approach.

Appendix B Method Details

B.1 Action Generator

Following Li et al. [30], we construct our action generator with a PointNet++ [28] encoder $E_{\Theta}$ and a Diffusion Transformer (DiT) [35] denoiser $D_{\Phi}$ as shown in Fig. 12. We remove the positional encoding in original DiT because point clouds are unordered. Given the current observation point cloud $o\in\mathbb{R}^{N\times 3}$ , and a noisy dense 3D articulation flow (defined in Eisner et al. [31]) $a_{noisy}\in\mathbb{R}^{N\times 3}$ , we first use the PointNet++ encoder to encode the noisy action into a point-wise embedding ( $\mathbb{R}^{N\times d}$ , where b is the embedding dimension). Then we input the per-point features into a Diffusion Transformer (DiT) to predict the noise $\epsilon$ to denoise from the noisy action field.

In our setting, we use $N=1200$ , $d=128$ . And the size of DiT is depth = 5, hidden size = 128, and 4 heads. We train with learning rate = 1e-4, AdamW optimizer with weight decay = 1e-5, batch size = 32 and train for 400 epochs.

From 3DAF to Dense Action Field

The generator generates 3DAF to provide flexibility of selecting grasp points, and we by default choose the point with the largest magnitude as the action point (which means the largest articulation leverage as illustrated in Eisner et al. [31]), and the corresponding flow as the action direction. The action is then transformed into a dense action field for verifier evaluation. The verifier takes in dense action field because of (1) generating random actions for training is more intuitive compared with generating random flows that needs specification of an articulation, (2) action is a more generalizable concept and representation.

For the uneven object pick-up task, we cannot directly apply articulation flow representation due to the lack of hinge mechanism for articulated objects. Instead, our generator represents each surface point by a “pick‑up flow” vector: its direction is normal to the object’s surface (vertical), and its magnitude equals the point’s distance from the object’s center of mass. We then define leverage analogously, selecting the action point as the location with the smallest pick-up flow magnitude.

B.2 Dataset Details

We sample 200 history trajectories for each object in the dataset with various length (range of 1 to 30 for articulated objects and 1 to 10 for uneven object pick-up), and for each trajectory, we include 10 random actions along with ground truth actions and transformed history actions (actions are transformed to the current state, e.g. if the history action is based on closed doors while the current state is an open door, the history action point and direction will be transformed to the corresponding open angle). We simulate each action to collect corresponding ground truth reward labels to supervise training.

Ground truth rewards

The ground truth reward are defined according to task goal: for opening articulated objects, the reward is defined as the relative amount the action opens the door against the ground-truth action at the same state; for uneven object lifting, the reward is the signed distance between the lift point and the mass center.

B.3 Model Details

For action and action result sequential reasoning module, we each use 4 layers of self-attention with 4 heads, and the action and action result embedding dimensions are both 128. We sample 1200 points for each point cloud during training and inference. We train the scoring module for 50 epochs (or early stop at convergence) using AdamW optimizer with lr = 1e-4.

Appendix C Baseline and Ablation Details

In this section, we describe the architecture of our baselines and ablations in details.

C.1 Oracle Experiments

We include two types of oracle experiments for better understanding of the model’s performance: (1) Oracle Sampler: We always append the ground truth action into the action proposals; this experiment aims to test the verifier’s ability with optimal generator; (2) Oracle Verifier: We always choose the closest-to-ground-truth action from the batch when selecting from the action proposals; this experiment aims to test the upper limit of using a verifier on the current generator.

C.2 Sparse Action

In the Sparse Action ablation, instead of representing an action as a dense action field (Sec. 5.3), we represent the action as a sparse action field where only the contact point has nonzero values: given a point cloud $\{p_{i}\}$ , and an action point $p$ and direction $d$ , the sparse action field is the point cloud with 3 extra channels to represent the action, calculated as $\{d_{i}:d*\mathds{1}(p_{i}=p)\}$ . We keep the exact same architecture, but force the first downsampling layer to sample the contact point to prevent complete information loss due to downsampling.

C.3 Point Cloud as Result

In the Point Cloud as Result ablation, instead of explicitly extracting observation flow from the observation pairs before and after an action, we directly encode and pass the observations sequence to a self-attention layer to extract corresponding features as shown in Fig. 13. This gives the model all the information needed to complete the task. The model would need to understand the movements between frames implicitly, unlike our architecture that explicitly computes the flow between observations. The other parts of the architecture are kept the same as HAVE.

C.4 Vanilla Transformer

In the Vanilla Transformer ablation in Sec. 6.3, instead of encoding actions and results independently and using an explicit dot-product attention layer to reason about the action proposal, in this ablation we concatenate the encoded action features with corresponding result features, and use an end-to-end transformer to predict the final score as shown in Fig. 14. We first encode action proposal $\hat{a}$ , history actions and results $\{(o_{i-1},a_{i},o_{i}),i\in[t-1]\}$ in the same way as HAVE: $\hat{a}_{\textrm{emb}}=E_{\textrm{action}}(\hat{a})$ , $\{a_{\textrm{emb},i}=E_{\textrm{action}}(a_{i}),i\in[t-1]\}$ , $\{f_{i}=E_{\textrm{obs}}(o_{i-1},o_{i}),i\in[t-1]\}$ . Then we concatenate the corresponding actions and results $\{h_{i}=concat(a_{\textrm{obs}},f_{i}),i\in[t-1]\}$ . We concatenate the action proposal embedding with an learnable token $\hat{f}$ to keep the same dimension with $\{h_{i}\}$ : $\hat{h}=concat(\hat{a}_{\textrm{emb}},\hat{f})$ . The sequence $\{\hat{h},h_{0},h_{1},...,h_{t}\}$ is then passed through a transformer that produce embedding tokens for each timestep: $\{\hat{h}^{\prime},h_{0}^{\prime},h_{1}^{\prime}...,h_{t}^{\prime}\}$ , we take $\hat{v}=\hat{h}^{\prime}$ as the final score embedding for the action proposal, and pass through an MLP to obtain the final score prediction.

We supervise the model with 2 losses, final score loss that compares the predicted score with the ground truth score for the action proposal, and history score loss in an encoder-decoder manner where we induce another MLP to decode the result features $\{f_{i}\}$ to scores and supervise with the ground truth scores for history actions to encourage learning better representation of action results.

C.5 Conditional Diffusion

To compare HAVE with end-to-end history-aware conditional diffusion without verifier, we construct a conditional diffusion structure merging the unconditional generator and the history-aware verifier structure as shown Fig. 15. We separate the history-aware generator into a history encoder, and a generator head. The history encoder takes in current observation / state, and the history actions and results, and encodes everything into a history embedding representing the guidance information obtained from history interactions. We use the vanilla transformer architecture in Appendix D.3 because instead of outputting a score for a specific action proposal, the history embedding should include information about the possible action itself where the HAVE architecture focused on calculating a score is not directly applicable.

The history embedding will be added with diffusion timestep as a condition into the Diffusion Transformer denoiser through adaLN-Zero conditioning. We train the model with classifier-free guidance style where we randomly drop out histories during training. When there is no history, the history embedding will be replaced by a learnable token.

Appendix D Experiment Results and Details

D.1 Held-out Articulated Categories

Following the protocol of Eisner et al. [31], we split PartNet-Mobility dataset [36] into 11 training categories, and leave 10 categories entirely unseen. In Table 3, we report the failure rate on unseen categories for each of the baselines and for our method. We provide the icon-to-text correspondence in Fig. 16 for easier understanding of the corresponding categories. We observe significant performance improvements with our method compared with baselines without a verifier. We underperform FlowBotHD (w/CC) where it includes a heuristic that filters out actions that doesn’t obey with successful history action movements. This consistency check in FlowBotHD is like a task-specific ground truth verifier, and due to its heuristic design, it is not directly generalizable to other tasks outside of articulated objects (e.g. in uneven objects pick-up, the important phase is to reason about failures where there is no successful history available). Instead, our verifier architecture provides a generalizable insight and achieves comparable results. For ablations, we observe removing unconditional score loss slightly improves the performance. This indicates that whether or not supervise the unconditional score is a minor design choice whose performance may vary across tasks.

D.2 Uneven Object Pick-up

We constructed the training ambiguous rod dataset with 17 rod replicates of the same size and assigned the mass center randomly across the rod length. For the unseen test dataset, we included 40 rods (with sizes and mass centers different from training dataset), 20 complex bookmarks, 20 irregular bookmarks, and 20 knives of various lengths, widths, and heights. Examples of the visual appearances of each categories are shown in Fig. 17.

D.3 Real-World Ambiguous Door

To show HAVE’s ability to transfer to real world scenarios, we visualize the process of two real world trials. We plot the sampling distribution from the generator, the average verifier score for each mode, and the 95% confidence interval to demonstrate how the generator and the verifier work together. In Fig. 18, the door is configured to be opened by pull right (opens from the front right), and the model first tries push left (push is also executed by pull from the back) which does not move the door. For the next step, the sampled batch from the generator has a similar distribution because the observation is similar. Despite being frequently generated by the generator, the failure mode (push left) receives consistently low scores (low mean and very small confidence interval). The verifier ends up choosing the correct action (pull right) among the three remaining actions and opens the door.

In Fig. 19, the door is configured to open by pull left. The first two steps are similar, except that the pull right action opens the door for a small amount. Then given the slightly opened door, the generator is also biased towards the correct mode based on the geometry, and the verifier assigns higher scores for actions that are not failures before, together results in the correct open mode. This demonstrates that with verifier and generator working together, we are effectively utilizing both geometric and history cues towards a successful decision.

D.4 Generation-Verification for Hierarchical Policy Learning

Our idea of using a verifier in ambiguous settings can also be extended to other settings like hierarchical policy learning pipeline. In this section, we utilize high-level sub-goal priors to guide low-level policies to tackle benchmark robotic manipulation tasks in MimicGen [37] (Figure 20).

Sub-goal Generation for Multi-stage Tasks. We consider manipulation tasks in MimicGen [37] using a robot manipulator with 2-finger grippers. Given an expert demo trajectory $\tau=(s_{0},a_{0},s_{1},a_{1},\cdots,s_{T})$ , we define critical sub-goal frames $t_{1},t_{2},\cdots,t_{k}\in\{0,1,\cdots,T-1\}$ as the time steps when the robot gripper switches from open state to closed state or from closed to open. For each time step between two critical sub-goal frames in the expert trajectory, i.e., $t\in[t_{i},t_{i+1})$ , we augment the data point by incorporating the robot gripper point cloud at the sub-goal frame $t_{i+1}$ as the ground-truth sub-goal information for our hierarchical pipeline, i.e.,

[TABLE]

where $P_{t_{i+1}}$ is the robot gripper point cloud at time step $t_{i+1}$ .

Hierarchical System Pipeline. The pipeline mainly consists of two critical parts: (a) High-level sub-goal predictor takes visual observation as input and predicts the next sub-goal in the form of gripper point clouds. (b) Low-level robot policy takes visual observation and sub-goal conditioning as input and generates low-level robot action through diffusion process.

For high-level sub-goal generator, we follow the model backbone in TAX3D [38]. In training time, the input of the model is current gripper point cloud and the full-scene point cloud. The ground-truth target flow is the per-point displacement between the current gripper point cloud and the sub-goal gripper point cloud. For each expert demo trajectory $\tau=(s_{0},a_{0},s_{1},a_{1},\cdots,s_{T})$ , we create $T$ input-label paired data points for training the TAX3D model. During inference time, randomly sampled displacements $\Delta X\sim N(0,I)$ are de-noised conditioned on current gripper features and full-scene features. The final $\Delta X_{0}$ is predicted to displace the gripper into a sub-goal configuration.

For the low-level policy, we use a modified version of 3D Diffusion Policy (DP3) [39], which consists of a point cloud encoder that encodes the point cloud observation into a latent embedding, and a conditional denoising diffusion model that generates noise-free actions for execution. For the point cloud encoder, we perform cross attention among full-scene point cloud, current gripper points and sub-goal gripper points with Rotary Position Embedding [40].

Verifier as Sub-goal Selector. In this setting, high-level sub-goal prediction features implicit multi-modality and ambiguity, due to the fact that the TAX3D model inputs are very similar near critical sub-goal frames while target predictions switch from the previous sub-goal to the next stage. Take Square D2 (Figure 20(a)) as an example, when the gripper is about to grasp the object, the high-level prediction of sub-goal gripper points should switch from the object handle to the target peg. To this end, we use a separately-trained verifier to select the best sub-goal prediction from multiple TAX3D-generated high-level predictions.

More specifically, we use PointNet++ [28] as the model backbone and fit the model for binary classification. In training time, we feed the network with positive demos that consist of full-scene points and ground-truth sub-goal gripper points, and negative demos that consist of full-scene points and randomly perturbed ground-truth sub-goal gripper points. In inference time, we use the model to score TAX3D samples by passing in current full-scene points and predicted gripper points, and then choose the one with the highest score as the high-level sub-goal prediction.

Results. We report the performance of the whole hierarchical pipeline with different high-level conditions in Table 4. TAX3D refers to conditioning on TAX3D predictions with 1 sample, and TAX3D+Verifier refers to sampling 5 TAX3D predictions and using the score predictor to select the best sample as high-level guidance in inference. Generally, the verifier helps tackle ambiguity at critical goal-switching frames and guides the low-level policy to better complete benchmark tasks. This experiment supports our main idea of using a verifier for better action selection and demontrates the insight’s generalization across tasks.

Appendix E Analysis

E.1 Verifier Performance w.r.t History Type

Failure History (Articulated objects)

: On the multi-modal door dataset, we create failure interaction histories (observation flow is set to zeros to represent failure results) with different lengths and failure modes and visualize the normalized scores predicted by the verifier in Fig. 21. We sample 20 actions (different contact point) from each mode (push left, push right, pull left, pull right). Given a specific failure history sequence, we evaluate the sampled actions with the verifier and calculate the normalized score, if the corresponding mode of the action appears in the history sequence, it will be counted as the “failure modes”, and other actions are counted into “other modes”.

We plot the distribution of normalized score (normalizedS to [0,1] range using min-max normalization for better visualization) predicted for failure modes and other modes across different history actions, and different doors in the multi-modal door dataset. We can see that on average the predicted scores for failure modes are lower than other modes. We also demonstrate in Table 5 the percentage of the trials where the selected action (action with the highest score) among the 20 * 4 proposed actions lies in “other modes”. We observe almost 100% of the chosen actions avoid failure actions, demonstrating HAVE’s ability to understand failures and avoid repeating failures.

We also demonstrate detailed score predictions for each mode given one step failure history for 2 randomly selected doors in Fig. 22. We can see that the mode that has led to failure in the history is suppressed by the verifier.

Success History (Articulated Objects)

: To test the verifier’s ability to utilize success histories, we demonstrate the verifier’s evaluation for actions under seriously occluded point clouds given a successful history action. In Fig. 23, we give the verifier a successful history at 20 degrees, and make it reason about occluded point clouds at around $80^{\circ}$ . We first construct action batch with the correct action direction but different action points, and plot the normalized score heatmap. We can see that the verifier basically understands what is a better grasp point (larger leverage) despite the point clouds are geometrically ambiguous. We then keep the action point the same (as the top leverage point in ground truth), and rotate the ground truth action direction around the z-axis (vertical to the ground), and plot their predicted normalized scores. We can see the scores are higher when it is closer to the ground truth direction (closer to $0^{\circ}$ and $360^{\circ}$ ). This experiment demonstrates the verifier’s ability to refer to previous successful histories (even with large open angle difference and thus very different point cloud) and reason about seriously occluded states with fairly precise understanding of good action points and directions.

Failure History (Uneven Objects)

We also demonstrate a similar learning from failure mechanism for uneven object pick-up. During each pick-up attempt, the tilting direction of the object will leak information about which side the center of mass lies on. Therefore, we can calculate a theoretical center of mass range at each step based on the previous histories. Like in Fig. 4, we plot the ground truth center of mass as a horizontal line, the theoretical center of mass range as bars, and the action point chosen at each step as scattered dots. The ideal behavior is that with history information, the model can choose within the theoretical boundaries at each step and converge to the ground truth position with fewer steps. In Fig. 24, we can see that HAVE’s choices lie within the theoretical range at each step, and uses only half of the steps used by the generator. In comparison, the generator-only method repeatedly fails regardless of valuable information from the history failures.

Similarly in Fig. 25, the generator-only method breaks out of the theoretical boundary more often, while ours mostly lies within the boundary with the third step being a slight violation. The visualizations demonstrate HAVE’s ability to actually reason about history actions and extract useful information from previous failures.

E.2 Verifier Performance w.r.t History Length

We experiment in all three articulated objects environments with different maximum history $K$ length during inference time, where we only keep the most recent $K$ step histories. As shown in Fig. 26, we see that the improvement from using 1-step history to 5-step history is significant, further increasing history length improves the performance in fullset and held-out categories, but in a much smaller scale. The intuition behind this is that with a short history length, there might not be enough context for the model to reason with to select the correct mode, but with a long history, the information is often times redundant and harder to process and may induce instability.

E.3 HAVE with Conditional Generators

We compared the performance of HAVE when paired with either an unconditional or a conditional diffusion generator. As shown in Fig. 27, using an unconditional generator with HAVE consistently yields better results in environments with complex action spaces, such as for articulated objects. A potential reason is that the unconditional generator is more expressive and captures a wider range of actions. In contrast, the conditional generator, which is trained to reduce ambiguity, is more likely to experience mode collapse, limiting the verifier’s ability to assist. For environments where the generator is less confident, as with held-out categories, it tends to generate more diverse modes, allowing the verifier to improve performance more significantly compared with other environments. Finally, in the simpler ”Uneven Objects” environment, the conditional generator’s history-aware structure allows it to model the action distribution more effectively, leading to better performance.

E.4 Classifier Guidance

With a diffusion-based generator and a verifier, the most intuitive way is to directly apply the classifier guidance [41] to the generation process. Therefore we also experiment with classifier guidance to compare with our proposed method HAVE. Because we generate 3D Articulation Flow [31] and verify the dense action field (Sec.5.3), we transform the noisy articulation flow prediction $\{f_{i}\}$ from each step to a dense action field $\{d_{i}\}$ to pass through the verifier: We first compute a weighted average of the noisy flow to obtain a global direction $\bar{f}$ , and a weighted grasp center $\bar{p}$ from the point cloud, both using flow magnitudes $||f_{i}||$ as weights. Then we use these point and action direction to create the dense action field following the magnitude decaying equation from Sec.5.3. This process is differentiable and therefore the verifier’s gradients over the initial articulation flow generation can be calculated - we then use the gradients for classifier guidance.

We can see from Fig. 28 the classifier guidance offers a minor performance improvement, and has a big gap with HAVE. One possible reason is that even though classifier guidance provides gradients over intermediate states, these gradients can be misleading: the transformation from noisy flow ${f_{i}}$ to a clean dense action field ${d_{i}}$ may distort the gradient signal, and even without this transformation, the intermediate noisy flows themselves are out-of-distribution for the verifier, which is trained only on clean data. In contrast, HAVE naturally reasons over history trajectories and scores action proposals, offering a more reliable and effective approach for selecting good actions.

E.5 CFG scale for Conditional Diffusion

We tried different classifier-free guidance scale at test time for the conditional diffusion baseline as shown in Fig. 29. In the figure, we plot Normalized Distance (the normalized distance to being fully open, as used in Eisner et al. [31]) against different classifier-free guidance scale. From the curves, we see that the best performance is achieved with classifier-free guidance scale of 0.5 or 1.0 (varies across tasks), but the overall performance is about the same. This indicates that the current conditional diffusion structure is not understanding the histories well, demonstrating that training a history-aware conditional diffusion model is practically harder and less efficient than training a verifier.

E.6 Generalization with Different Verifier Capacity

We test the generalization ability of the verifier and analyze whether overfitting occurs by comparing training and test metrics. As shown in Table 6, the verifier demonstrates strong generalization to unseen instances and reasonable generalization to unseen categories, though performance drops more in the latter case—where overfitting is most noticable.

To investigate how model capacity contributes to this, we varied the transformer size and compared the failure rates on train and unseen categories in Table 7. We use 4 attention heads, 4 attention layers and 256 as feature dimension for the final model. We found that a larger model (8 attention heads, 6 attention layers, and feature dimension as 512) slightly improves training accuracy but hurts generalization, suggesting it may be too large for the task and leads to overfitting. A smaller model (2 attention heads, 2 attention layers, and feature dimension as 256) performs comparably on unseen categories, suggesting that model capacity has limited impact and our chosen size is appropriate for the task.

Bibliography41

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1Setlur et al. [2025] A. Setlur, N. Rajaraman, S. Levine, and A. Kumar. Scaling test-time compute without verification or rl is suboptimal. ar Xiv preprint ar Xiv:2502.12118 , 2025.
2Bohg et al. [2017] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics , 33(6):1273–1291, 2017.
3Huang et al. [2023] P. Huang, X. Zhang, Z. Cao, S. Liu, M. Xu, W. Ding, J. Francis, B. Chen, and D. Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Conference on Robot Learning , pages 734–760. PMLR, 2023.
4Katz and Brock [2008] D. Katz and O. Brock. Manipulating articulated objects with interactive perception. In 2008 IEEE International Conference on Robotics and Automation , pages 272–277, 2008. doi: 10.1109/ROBOT.2008.4543220 . · doi ↗
5Weng et al. [2024] Z. Weng, P. Zhou, H. Yin, A. Kravberg, A. Varava, D. Navarro-Alarcon, and D. Kragic. Interactive perception for deformable object manipulation. IEEE Robotics and Automation Letters , 2024.
6Allevato et al. [2020] A. Allevato, E. S. Short, M. Pryor, and A. Thomaz. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. In Conference on Robot Learning , pages 445–455. PMLR, 2020.
7Xu et al. [2022] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. Tenenbaum, and C. Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning , pages 24631–24645. PMLR, 2022.
8Laskin et al. [2022] M. Laskin, L. Wang, J. Oh, E. Parisotto, S. Spencer, R. Steigerwald, D. Strouse, S. Hansen, A. Filos, E. Brooks, et al. In-context reinforcement learning with algorithm distillation. ar Xiv preprint ar Xiv:2210.14215 , 2022.

	${AVG}_{c}$	${AVG}_{s}$
Baselines
FlowBot3D	39.6	29.4	56.0	66.7	38.6	27.9	28.5	14.1	25.0	0.0	96.7	42.4
FlowBotHD (w/o CC)	30.4	28.1	46.7	6.7	13.8	37.9	6.7	45.9	31.0	0.0	73.3	42.4
FlowBotHD (w/ CC)	14.8	10.9	22.7	0.0	6.2	32.9	3.6	15.3	4.6	0.0	30.0	32.9
Generator Only	44.4	25.0	85.3	66.7	14.8	65.7	30.9	44.7	8.0	0.8	83.3	43.5
Conditional Diffusion	53.4	41.2	78.7	66.7	46.7	60.7	58.8	35.3	26.6	0.0	86.7	74.1
Ours
HAVE (Ours) + GT obs flow	16.9	11.3	44.0	13.3	0.5	13.6	17.6	27.1	3.2	0.0	10.0	40.0
HAVE (Ours) + DELTA	21.5	14.1	62.7	6.7	2.4	20.0	20.6	35.3	4.2	0.0	23.3	40.0
Ours (Oracle)
Oracle Sampler	12.2	9.5	30.7	0.0	2.4	10.7	11.5	17.7	2.4	0.0	3.3	42.9
Oracle Verifier	14.4	9.2	58.7	6.7	1.0	10.7	6.1	23.5	2.4	0.0	3.3	31.2
Ablations
w/o Unconditional Score Loss	16.1	10.6	60.0	6.7	1.0	14.3	7.9	27.1	2.6	0.0	3.3	38.2
w/o History Score Loss	18.1	10.6	52.0	0.0	0.5	12.1	9.1	29.4	2.2	0.0	36.7	38.8
Point Cloud as Result	23.5	13.0	62.7	6.7	3.8	19.3	12.1	33.0	2.6	0.0	56.7	38.2
Sparse Action	22.7	12.5	72.0	33.3	1.9	15.0	9.1	28.2	3.1	0.8	23.3	40.6