A Representation Optimization Dichotomy, Lie-Algebraic Policy Optimization

TL;DR

This paper establishes a fundamental dichotomy in the smoothness of Lie-algebraic policy objectives in reinforcement learning, showing how algebraic structure influences optimization complexity and enabling more efficient algorithms.

Contribution

It introduces a representation-optimization dichotomy for Lie-algebra-parameterized policies, linking algebra type to gradient Lipschitz constants and proposing scalable optimization methods.

Findings

Compact algebras have constant smoothness, enabling faster convergence.

Exponential growth in smoothness for non-compact algebras like SE(3).

Projection-based algorithms outperform Fisher inversion in experiments.

Abstract

Structured reinforcement learning and stochastic optimization often involve parameters evolving on matrix Lie groups such as rotations and rigid-body transformations. We establish a representation-optimization dichotomy for Lie-algebra-parameterized Gaussian policy objectives in the Lie Group MDP class: the gradient Lipschitz constant L(R), governing step size, convergence, and sample complexity of first-order methods, depends only on the algebraic type of g, uniformly over all objectives, independent of reward or transition structure. Specifically, L = O(1) for compact g (e.g., so(n), su(n)), and L = Theta(exp(2R)) for g = gl(n), with O(exp(2R)) for all algebras with a hyperbolic element. A key lower bound shows this exponential growth cannot be canceled by interaction between the exponential map and the objective, making the dichotomy intrinsic to the algebra.This yields an algorithmic consequence: for compact algebras, radius-independent smoothness enables O(1/sqrt(T)) convergence using an O(n^2 J) Lie-algebraic projection step instead of O(d_g^3) Fisher inversion. A Kantorovich alignment bound alpha >= 2 kappa / (kappa + 1) provides a computable condition under which this projection approximates natural gradient updates. Experiments on SO(3)^J and SE(3) confirm the theory: constant smoothness for compact algebras, polynomial growth for SE(3), and alignment across condition regimes. The projection step achieves 1.1-1.7x speedup over Cholesky-based Fisher inversion, with increasing gains at larger scales.