Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control

TL;DR

This paper introduces Lyapunov density models, a novel approach combining Lyapunov stability and density estimation to ensure learning-based control agents remain within training data distribution, enhancing reliability.

Contribution

The paper proposes Lyapunov density models that unify control Lyapunov functions with density models to constrain agents to in-distribution states during control tasks.

Findings

Guarantees in-distribution state maintenance over trajectories.

Effective combination of control theory and machine learning concepts.

Improved safety and reliability in learning-based control.

Abstract

Learned models and policies can generalize effectively when evaluated within the distribution of the training data, but can produce unpredictable and erroneous outputs on out-of-distribution inputs. In order to avoid distribution shift when deploying learning-based control algorithms, we seek a mechanism to constrain the agent to states and actions that resemble those that it was trained on. In control theory, Lyapunov stability and control-invariant sets allow us to make guarantees about controllers that stabilize the system around specific states, while in machine learning, density models allow us to estimate the training data distribution. Can we combine these two concepts, producing learning-based control algorithms that constrain the system to in-distribution states using only in-distribution actions? In this work, we propose to do this by combining concepts from Lyapunov stability and density estimation, introducing Lyapunov density models: a generalization of control Lyapunov functions and density models that provides guarantees on an agent's ability to stay in-distribution over its entire trajectory.