Hierarchical Planning and Policy Shaping Shared Autonomy for Articulated Robots

TL;DR

This paper introduces a hierarchical shared autonomy framework for articulated robots, integrating human-in-the-loop decision making with deep reinforcement learning and advanced modeling of human states, demonstrated on timber harvesting tasks.

Contribution

It presents a novel framework combining hierarchical planning, policy shaping, and deep RL for shared autonomy, incorporating human internal state modeling using conditional VAEs.

Findings

Effective in handling noisy, non-cooperative human agents

Provides a sliding autonomy level from manual to autonomous

Demonstrated success on timber harvesting robot tasks

Abstract

In this work, we propose a novel shared autonomy framework to operate articulated robots. We provide strategies to design both the task-oriented hierarchical planning and policy shaping algorithms for efficient human-robot interactions in context-aware operation of articulated robots. Our framework for interplay between the human and the autonomy, as the participating agents in the system, is particularly influenced by the ideas from multi-agent systems, game theory, and theory of mind for a sliding level of autonomy. We formulate the sequential hierarchical human-in-the-loop decision making process by extending MDPs and Options framework to shared autonomy, and make use of deep RL techniques to train an uncertainty-aware shared autonomy policy. To fine-tune the formulation to a human, we use history of the system states, human actions, and their error with respect to a surrogate optimal model to encode human's internal state embeddings, beyond the designed values, by using conditional VAEs. We showcase the effectiveness of our formulation for different human skill levels and degrees of cooperativeness by using a case study of a feller-buncher machine in the challenging tasks of timber harvesting. Our framework is successful in providing a sliding level of autonomy from fully autonomous to fully manual, and is particularly successful in handling a noisy non-cooperative human agent in the loop. The proposed framework advances the state-of-the-art in shared autonomy for operating articulated robots, but can also be applied to other domains where autonomous operation is the ultimate goal.