HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers

TL;DR

HiRT introduces a hierarchical transformer approach that balances computational efficiency and performance in robotic control, enabling real-time interaction and improved success rates in dynamic tasks.

Contribution

This work presents HiRT, a novel hierarchical transformer framework that reduces reliance on high-frequency VLMs, improving dynamic task performance and computational efficiency in robotic control.

Findings

Doubling control frequency in static tasks with maintained success rates.

Increasing success rate from 48% to 75% in real-world dynamic manipulation tasks.

Significant improvements over baseline methods in simulation and real-world experiments.

Abstract

Large Vision-Language-Action (VLA) models, leveraging powerful pre trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, in static tasks, we double the control frequency and achieve comparable success rates. Additionally, on novel real-world dynamic ma nipulation tasks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.