FORESEE: Prediction with Expansion-Compression Unscented Transform for Online Policy Optimization

TL;DR

The paper introduces the Expansion-Compression Unscented Transform, a scalable method for state prediction in nonlinear dynamical systems, enabling efficient online policy optimization with applications in robotics.

Contribution

It proposes a novel expansion-compression approach for sigma point propagation, improving scalability and efficiency in online policy optimization under uncertainty.

Findings

Performance comparable to Monte Carlo methods with lower computational cost

Effective in constrained state and control input scenarios

Successfully applied to quadrotor stabilization and leader-follower control tasks

Abstract

Propagating state distributions through a generic, uncertain nonlinear dynamical model is known to be intractable and usually begets numerical or analytical approximations. We introduce a method for state prediction, called the Expansion-Compression Unscented Transform, and use it to solve a class of online policy optimization problems. Our proposed algorithm propagates a finite number of sigma points through a state-dependent distribution, which dictates an increase in the number of sigma points at each time step to represent the resulting distribution; this is what we call the expansion operation. To keep the algorithm scalable, we augment the expansion operation with a compression operation based on moment matching, thereby keeping the number of sigma points constant across predictions over multiple time steps. Its performance is empirically shown to be comparable to Monte Carlo but at a much lower computational cost. Under state and control input constraints, the state prediction is subsequently used in tandem with a proposed variant of constrained gradient-descent for online update of policy parameters in a receding horizon fashion. The framework is implemented as a differentiable computational graph for policy training. We showcase our framework for a quadrotor stabilization task as part of a benchmark comparison in safe-control-gym and for optimizing the parameters of a Control Barrier Function based controller in a leader-follower problem.