Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approach

TL;DR

This paper introduces a neuro-symbolic method for robots to detect and recover from execution errors during manipulation tasks without needing failure annotations, improving robustness and efficiency.

Contribution

It presents a novel neuro-symbolic framework combining dense scene graphs with learning and symbolic search for autonomous error recovery in manipulation tasks.

Findings

Effective error detection and localization in simulation

Improved recovery accuracy over baseline methods

Efficient re-planning with heuristic-guided search

Abstract

Automatically detecting and recovering from failures is an important but challenging problem for autonomous robots. Most of the recent work on learning to plan from demonstrations lacks the ability to detect and recover from errors in the absence of an explicit state representation and/or a (sub-) goal check function. We propose an approach (blending learning with symbolic search) for automated error discovery and recovery, without needing annotated data of failures. Central to our approach is a neuro-symbolic state representation, in the form of dense scene graph, structured based on the objects present within the environment. This enables efficient learning of the transition function and a discriminator that not only identifies failures but also localizes them facilitating fast re-planning via computation of heuristic distance function. We also present an anytime version of our algorithm, where instead of recovering to the last correct state, we search for a sub-goal in the original plan minimizing the total distance to the goal given a re-planning budget. Experiments on a physics simulator with a variety of simulated failures show the effectiveness of our approach compared to existing baselines, both in terms of efficiency as well as accuracy of our recovery mechanism.