Simple Hierarchical Planning with Diffusion
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, Sungjin Ahn
TL;DR
The paper introduces the Hierarchical Diffuser, a fast and effective hierarchical planning method that improves long-horizon task planning and generalization in diffusion-based models, with superior performance on offline RL benchmarks.
Contribution
It proposes a simple hierarchical diffusion-based planning approach with jumpy high-level planning and sub-goal guidance, enhancing efficiency and generalization.
Findings
Outperforms non-hierarchical Diffuser in speed and accuracy
Demonstrates improved generalization on out-of-distribution tasks
Shows effectiveness on standard offline RL benchmarks
Abstract
Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets. However, they often face computational challenges and can falter in generalization, especially in capturing temporal abstractions for long-horizon tasks. To overcome this, we introduce the Hierarchical Diffuser, a simple, fast, yet surprisingly effective planning method combining the advantages of hierarchical and diffusion-based planning. Our model adopts a "jumpy" planning strategy at the higher level, which allows it to have a larger receptive field but at a lower computational cost -- a crucial factor for diffusion-based planning methods, as we have empirically verified. Additionally, the jumpy sub-goals guide our low-level planner, facilitating a fine-tuning stage and further improving our approach's effectiveness. We conducted empirical evaluations on standard offline…
Peer Reviews
Decision·ICLR 2024 poster
- Using a hierarchical structure makes sense in long-horizon planning. - In replanning, a hierarchical structure is more efficient since it only needs to use low-level to - Show the relationship between kernel size and generalization.
- Current SOTA diffusers seem to have a better performance. For example [1] have a 167 score on large maze2d. - The Unet itself has a hierarchical structure. If an environment needs a hierarchical structure, a simple way is to increase the depth of the Unet. The authors might need to provide more results to show that they are better. - The improvement in Mujoco is not enough for me. - The paper said that they have evaluated generalization. However, the OOD task they test on is too simple. In oth
Discretizing planning into sub-goals with diffusion models is shown to be advantageous as more diverse scenarios can be solved. Low-level diffusion planning becomes task-agnostic. Even the sparse diffuser or SD version is better than diffuser because of increased receptive field. The authors validate this by showing that increasing the kernel-size (hence the receptive field) of diffuser leads to better performance but weaker generalization. SD with dense actions leads to better fitting of the
The formulation has limited novelty. While a single diffuser is not sufficient for planning over long-horizons, the work introduces two diffusers: one sparse diffuser for planning sequence of sub-goals between initial state and goal, while a standard diffuser solves for individual sub-goal segments. Given that there are also methods which perform state-only diffusion like Decision-diffuser [2], is it possible to perform relevant ablations to justify why states are not sufficient for having a go
- Originality: This work creatively combines the strengths of hierarchical planning and diffusion models, achieving significant improvements in performance and computational efficiency compared to existing methods. - Quality: The contribution provides a thorough theoretical analysis of the method, backed by well-designed experiments and in-depth analyses. The authors emphasize the rationale behind the design decisions and the resultant benefits of their approach. - Clarity: The paper is well-org
- Considering the existence of the HDMI algorithm, the contribution of this paper are limited. However, compared to HDMI, this paper has conducted extensive experiments on generalization and provided corresponding theoretical support. One suggestion I have is that the authors can emphasize the advantages of Hierarchical Diffuser from the perspective of generalization, and highlight the algorithm's generalization performance in the experimental design. - In the hierarchical framework, the quality
Videos
Taxonomy
TopicsReinforcement Learning in Robotics · Topic Modeling · Machine Learning and Algorithms
MethodsSPEED: Separable Pyramidal Pooling EncodEr-Decoder for Real-Time Monocular Depth Estimation on Low-Resource Settings