DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation

TL;DR

DiLQR introduces a differentiable iLQR framework using implicit differentiation, enabling efficient gradient computation and integration into neural networks for control tasks, with significant speedups and improved learning performance.

Contribution

The paper presents an analytical solution for differentiating through iLQR using implicit differentiation, enhancing scalability and performance in differentiable control applications.

Findings

Achieves up to 128x speedup over automatic differentiation.

Demonstrates superior learning performance compared to neural networks.

Successfully integrates into high-dimensional, end-to-end visual control tasks.

Abstract

While differentiable control has emerged as a powerful paradigm combining model-free flexibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains underexplored as a differentiable component. The scalability of differentiating through extended iterations and horizons poses significant challenges, hindering iLQR from being an effective differentiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and differentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differentiation, which ensures a constant backward cost regardless of iteration, while producing an accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analytical method demonstrates superior computational performance, achieving up to 128x speedup and a minimum of 21x speedup compared to automatic differentiation. Our method also demonstrates superior learning performance ($10^6$x) compared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into a larger network with visual inputs to demonstrate the capacity of our method for high-dimensional, fully end-to-end tasks. Codes can be found on the project homepage https://sites.google.com/view/dilqr/.