Temporally Layered Architecture for Adaptive, Distributed and Continuous Control

TL;DR

This paper introduces a biologically inspired temporally layered architecture (TLA) for adaptive, distributed control that operates across different time scales, improving exploration, control, and efficiency in continuous control tasks.

Contribution

The paper proposes a novel TLA system with two training algorithms, demonstrating its advantages over existing methods in continuous control environments.

Findings

TLA enhances persistent exploration and adaptive control.

TLA provides explainable temporal behavior and improves compute efficiency.

TLA outperforms several strong baselines in continuous control tasks.

Abstract

We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control. TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale. Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands. Such distributed control design is widespread across biological systems because it increases survivability and accuracy in certain and uncertain environments. We demonstrate that TLA can provide many advantages over existing approaches, including persistent exploration, adaptive control, explainable temporal behavior, compute efficiency and distributed control. We present two different algorithms for training TLA: (a) Closed-loop control, where the fast controller is trained over a pre-trained slow controller, allowing better exploration for the fast controller and closed-loop control where the fast controller decides whether to "act-or-not" at each timestep; and (b) Partially open loop control, where the slow controller is trained over a pre-trained fast controller, allowing for open loop-control where the slow controller picks a temporally extended action or defers the next n-actions to the fast controller. We evaluated our method on a suite of continuous control tasks and demonstrate the advantages of TLA over several strong baselines.