Optimal Control via Combined Inference and Numerical Optimization

TL;DR

This paper introduces a novel method that combines second order optimization with inference techniques to improve the efficiency and effectiveness of solving optimal control problems, especially where derivative information is limited.

Contribution

It presents a new approach integrating KL control-based inference with second order methods, enhancing cost function flexibility and computational performance in optimal control.

Findings

Outperforms MPPI and iLQG in sample efficiency.

Handles both convex and non-convex cost functions.

Achieves better control quality in manipulation and obstacle avoidance tasks.

Abstract

Derivative based optimization methods are efficient at solving optimal control problems near local optima. However, their ability to converge halts when derivative information vanishes. The inference approach to optimal control does not have strict requirements on the objective landscape. However, sampling, the primary tool for solving such problems, tends to be much slower in computation time. We propose a new method that combines second order methods with inference. We utilise the Kullback Leibler (KL) control framework to formulate an inference problem that computes the optimal controls from an adaptive distribution approximating the solution of the second order method. Our method allows for combining simple convex and non convex cost functions. This simplifies the process of cost function design and leverages the strengths of both inference and second order optimization. We compare our method to Model Predictive Path Integral (MPPI) and iterative Linear Quadratic Regulator (iLQG), outperforming both in sample efficiency and quality on manipulation and obstacle avoidance tasks.