Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent

TL;DR

This paper introduces DEPS, a deep reinforcement learning algorithm that jointly optimizes environment design and control policies for stochastic systems, outperforming existing methods in various control tasks.

Contribution

The paper presents a novel algorithm combining policy gradient and model-based optimization for joint environment and policy design in stochastic systems.

Findings

DEPS achieves higher returns with fewer iterations compared to benchmarks.

Joint optimization of environment and policy yields better performance than separate optimization.

DEPS performs effectively across diverse control environments.

Abstract

We consider the joint design and control of discrete-time stochastic dynamical systems over a finite time horizon. We formulate the problem as a multi-step optimization problem under uncertainty seeking to identify a system design and a control policy that jointly maximize the expected sum of rewards collected over the time horizon considered. The transition function, the reward function and the policy are all parametrized, assumed known and differentiable with respect to their parameters. We then introduce a deep reinforcement learning algorithm combining policy gradient methods with model-based optimization techniques to solve this problem. In essence, our algorithm iteratively approximates the gradient of the expected return via Monte-Carlo sampling and automatic differentiation and takes projected gradient ascent steps in the space of environment and policy parameters. This algorithm is referred to as Direct Environment and Policy Search (DEPS). We assess the performance of our algorithm in three environments concerned with the design and control of a mass-spring-damper system, a small-scale off-grid power system and a drone, respectively. In addition, our algorithm is benchmarked against a state-of-the-art deep reinforcement learning algorithm used to tackle joint design and control problems. We show that DEPS performs at least as well or better in all three environments, consistently yielding solutions with higher returns in fewer iterations. Finally, solutions produced by our algorithm are also compared with solutions produced by an algorithm that does not jointly optimize environment and policy parameters, highlighting the fact that higher returns can be achieved when joint optimization is performed.