A Test-Function Approach to Incremental Stability

TL;DR

This paper introduces a new framework linking reinforcement learning value functions to incremental stability analysis using test functions, offering a novel perspective beyond traditional Lyapunov methods.

Contribution

It establishes an equivalence between incremental stability and the regularity of RL-style value functions, bridging control theory and reinforcement learning.

Findings

RL value functions relate to incremental stability via test functions.

Regularity of value functions indicates system stability.

New theoretical link between control stability and RL value functions.

Abstract

This paper presents a novel framework for analyzing Incremental-Input-to-State Stability ($\delta$ISS) based on the idea of using rewards as "test functions." Whereas control theory traditionally deals with Lyapunov functions that satisfy a time-decrease condition, reinforcement learning (RL) value functions are constructed by exponentially decaying a Lipschitz reward function that may be non-smooth and unbounded on both sides. Thus, these RL-style value functions cannot be directly understood as Lyapunov certificates. We develop a new equivalence between a variant of incremental input-to-state stability of a closed-loop system under given a policy, and the regularity of RL-style value functions under adversarial selection of a H\"older-continuous reward function. This result highlights that the regularity of value functions, and their connection to incremental stability, can be understood in a way that is distinct from the traditional Lyapunov-based approach to certifying stability in control theory.