Iterative On-Policy Refinement of Hierarchical Diffusion Policies for Language-Conditioned Manipulation

TL;DR

This paper introduces HD-ExpIt, an iterative framework that refines hierarchical diffusion policies for language-conditioned manipulation by leveraging environment feedback, leading to improved performance on complex manipulation tasks.

Contribution

The paper presents HD-ExpIt, a novel iterative fine-tuning method that enhances hierarchical diffusion policies through self-reinforcing cycles using environment feedback, without relying on fixed offline datasets.

Findings

Achieves state-of-the-art results on CALVIN benchmark.

Significantly improves hierarchical policies trained solely on offline data.

Demonstrates effective grounding of planners in controller capabilities.

Abstract

Hierarchical policies for language-conditioned manipulation decompose tasks into subgoals, where a high-level planner guides a low-level controller. However, these hierarchical agents often fail because the planner generates subgoals without considering the actual limitations of the controller. Existing solutions attempt to bridge this gap via intermediate modules or shared representations, but they remain limited by their reliance on fixed offline datasets. We propose HD-ExpIt, a framework for iterative fine-tuning of hierarchical diffusion policies via environment feedback. HD-ExpIt organizes training into a self-reinforcing cycle: it utilizes diffusion-based planning to autonomously discover successful behaviors, which are then distilled back into the hierarchical policy. This loop enables both components to improve while implicitly grounding the planner in the controller's actual capabilities without requiring explicit proxy models. Empirically, HD-ExpIt significantly improves hierarchical policies trained solely on offline data, achieving state-of-the-art performance on the long-horizon CALVIN benchmark among methods trained from scratch.