Stochastic Approximation for Nonlinear Discrete Stochastic Control: Finite-Sample Bounds

TL;DR

This paper develops finite-sample convergence bounds for a nonlinear stochastic control system using stochastic approximation and Lyapunov functions, providing practical guarantees for control policy performance.

Contribution

It introduces a method to derive finite-sample bounds for nonlinear stochastic control systems using Lyapunov functions and smoothing techniques, extending existing convergence results.

Findings

Finite-sample bounds are established for four cases based on Lyapunov function properties.

The bounds depend on exponential or sub-exponential convergence rates.

Numerical experiments validate the theoretical convergence rates.

Abstract

We consider a nonlinear discrete stochastic control system, and our goal is to design a feedback control policy in order to lead the system to a prespecified state. We adopt a stochastic approximation viewpoint of this problem. It is known that by solving the corresponding continuous-time deterministic system, and using the resulting feedback control policy, one ensures almost sure convergence to the prespecified state in the discrete system. In this paper, we adopt such a control mechanism and provide its finite-sample convergence bounds whenever a Lyapunov function is known for the continuous system. In particular, we consider four cases based on whether the Lyapunov function for the continuous system gives exponential or sub-exponential rates and based on whether it is smooth or not. We provide the finite-time bounds in all cases. Our proof relies on constructing a Lyapunov function for the discrete system based on the given Lyapunov function for the continuous system. We do this by appropriately smoothing the given function using the Moreau envelope. We present numerical experiments corresponding to the various cases, which validate the rates we establish.