Model-Agnostic Zeroth-Order Policy Optimization for Meta-Learning of Ergodic Linear Quadratic Regulators

TL;DR

This paper introduces a model-agnostic zeroth-order meta-learning algorithm for ergodic linear quadratic regulators, enabling efficient optimization over heterogeneous systems without Hessian estimation, supported by convergence analysis and numerical validation.

Contribution

It proposes a novel zeroth-order meta-learning algorithm for LQR systems that avoids Hessian estimation and provides convergence guarantees.

Findings

The algorithm effectively handles heterogeneity in linear systems.

Convergence analysis shows small gradient estimation error.

Numerical example validates the theoretical insights.

Abstract

Meta-learning has been proposed as a promising machine learning topic in recent years, with important applications to image classification, robotics, computer games, and control systems. In this paper, we study the problem of using meta-learning to deal with uncertainty and heterogeneity in ergodic linear quadratic regulators. We integrate the zeroth-order optimization technique with a typical meta-learning method, proposing an algorithm that omits the estimation of policy Hessian, which applies to tasks of learning a set of heterogeneous but similar linear dynamic systems. The induced meta-objective function inherits important properties of the original cost function when the set of linear dynamic systems are meta-learnable, allowing the algorithm to optimize over a learnable landscape without projection onto the feasible set. We provide a convergence result for the exact gradient descent process by analyzing the boundedness and smoothness of the gradient for the meta-objective, which justify the proposed algorithm with gradient estimation error being small. We also provide a numerical example to corroborate this perspective.