Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planning

TL;DR

This paper introduces BISON, a bilevel policy system combining neural imitation learning and symbolic planning to improve long-horizon task solving in embodied AI, demonstrating superior efficiency and scalability.

Contribution

It proposes a novel bilevel policy framework that integrates low-level imitation learning with high-level symbolic planning for long-horizon tasks.

Findings

BISON outperforms existing methods on extended MetaWorld benchmarks.

It generalizes to problems with more objects and longer horizons.

BISON's high-level policies can solve large HL problems quickly, ignoring LL execution.

Abstract

We tackle the challenge of building embodied AI agents that can reliably solve long-horizon planning problems. Imitation learning from demonstrations has shown itself to be effective in training robots to solve a diversity of complex tasks requiring fine motor control and manipulation over low-level (LL), continuous environments. Yet, it remains a difficult endeavour to generate long-horizon plans from imitation learning alone. In contrast, high-level (HL), symbolic abstractions facilitate efficient and interpretable long-horizon planning. We propose to combine the strengths of LL imitation learning for manipulation and control, and HL symbolic abstractions for long-horizon planning. We realise this idea via \emph{bilevel policies} of the form $(\pi^{\mathrm{hl}}, \pi^{\mathrm{ll}})$, consisting of a neural policy $\pi^{\mathrm{ll}}$ learned from LL demonstrations, and an HL symbolic policy $\pi^{\mathrm{hl}}$ that is constructed from symbolic abstractions of the LL demonstrations combined with inductive generalisation. We implement these ideas in the BISON system. Experiments on extended MetaWorld benchmarks demonstrate that BISON generalises to long horizons and problems with greater numbers of objects than those solved by VLA and end-to-end methods, and is more time and memory efficient in training and inference. Notably, when ignoring LL execution, BISON's HL policies can solve HL problems with 10,000 relevant objects in under a minute. Project page: https://dillonzchen.github.io/bison