Hierarchical control and learning of a foraging CyberOctopus

TL;DR

This paper introduces a hierarchical control framework inspired by octopus neurophysiology, combining reinforcement learning and neural-network energy shaping to improve multi-arm coordination in a simulated foraging task.

Contribution

It presents a novel hierarchical approach integrating model-free and model-based control schemes, including a fast neural-network energy shaping controller for efficient arm movement.

Findings

Hierarchical control significantly outperforms end-to-end methods.

The neural-network energy shaping controller is 200 times faster than previous methods.

Framework successfully handles complex 3D obstacle-laden foraging scenarios.

Abstract

Inspired by the unique neurophysiology of the octopus, we propose a hierarchical framework that simplifies the coordination of multiple soft arms by decomposing control into high-level decision making, low-level motor activation, and local reflexive behaviors via sensory feedback. When evaluated in the illustrative problem of a model octopus foraging for food, this hierarchical decomposition results in significant improvements relative to end-to-end methods. Performance is achieved through a mixed-modes approach, whereby qualitatively different tasks are addressed via complementary control schemes. Here, model-free reinforcement learning is employed for high-level decision-making, while model-based energy shaping takes care of arm-level motor execution. To render the pairing computationally tenable, a novel neural-network energy shaping (NN-ES) controller is developed, achieving accurate motions with time-to-solutions 200 times faster than previous attempts. Our hierarchical framework is then successfully deployed in increasingly challenging foraging scenarios, including an arena littered with obstacles in 3D space, demonstrating the viability of our approach.