Update-Free On-Policy Steering via Verifiers

TL;DR

This paper introduces UF-OPS, a lightweight, update-free on-policy steering method that uses verifiers trained on rollout data to improve robot manipulation success rates without altering the base policy.

Contribution

The paper proposes UF-OPS, a novel method that enhances on-policy policies with verifiers trained during initial evaluation, enabling success likelihood prediction and strategy adaptation at execution time.

Findings

Achieved 49% average success rate improvement over base policy.

Validated effectiveness in both simulation and real-world tasks.

Method is lightweight and does not modify base policy parameters.

Abstract

In recent years, Behavior Cloning (BC) has become one of the most prevalent methods for enabling robots to mimic human demonstrations. However, despite their successes, BC policies are often brittle and struggle with precise manipulation. To overcome these issues, we propose UF-OPS, an Update-Free On-Policy Steering method that enables the robot to predict the success likelihood of its actions and adapt its strategy at execution time. We accomplish this by training verifier functions using policy rollout data obtained during an initial evaluation of the policy. These verifiers are subsequently used to steer the base policy toward actions with a higher likelihood of success. Our method improves the performance of black-box diffusion policy, without changing the base parameters, making it light-weight and flexible. We present results from both simulation and real-world data and achieve an average 49% improvement in success rate over the base policy across 5 real tasks.