OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies

TL;DR

OMNIGUIDE introduces a flexible framework that enhances vision-language-action models for complex tasks by integrating diverse guidance sources as differentiable energy functions, significantly improving performance in simulation and real-world scenarios.

Contribution

It presents a novel, unified approach to incorporate various guidance sources into VLA models, surpassing prior methods in handling complex tasks.

Findings

Improves success and safety rates of generalist policies.

Matches or exceeds performance of guidance-specific prior methods.

Effective in both simulation and real-world environments.

Abstract

Vision-language-action(VLA) models have shown great promise as generalist policies for a large range of relatively simple tasks. However, they demonstrate limited performance on more complex tasks, such as those requiring complex spatial or semantic understanding, manipulation in clutter, or precise manipulation. We propose OMNIGUIDE, a flexible framework that improves VLA performance on such tasks by leveraging arbitrary sources of guidance, such as 3D foundation models, semantic-reasoning VLMs, and human pose models. We show how many kinds of guidance can be naturally expressed as differentiable energy functions with task-specific attractors and repellers located in 3D space, that influence the sampling of VLA actions. In this way, OMNIGUIDE enables guidance sources with complementary task-relevant strengths to improve a VLA model's performance on challenging tasks. Extensive experiments in both simulation and real-world environments, across diverse sources of guidance, demonstrate that OMNIGUIDE enhances the performance of state-of-the-art generalist policies (e.g., $\pi_{0.5}$, GR00T N1.6) significantly across success and safety rates. Critically, our unified framework matches or surpasses the performance of prior methods designed to incorporate specific sources of guidance into VLA policies. Project Page: $\href{https://omniguide.github.io/}{this \; url}$